How is the entity linking F1 calculated?

[yhshu]

Hi there,
I've noticed that in the paper appendix, there's an entity linking evaluation.
I would like to ask how does the linking F1 gets calculated? Is the evaluation in this repo?

Since there're some questions in GrailQA that have no golden entity, and it is definitely possible that the linking result is empty for a question, how does the F1 calculation handles these problems?

Thanks for your reply.

[xiye17]

[no gt entities] and [no detected entities] -> f1 = 1.0
[no gt entities] and [exists detected entities] -> f1 = 0.0
[exists gt entities] and [no detected entities] -> f1 = 0.0
then it's the standard f1 evaluation in other case.

This evaluation process might be a bit different from the original GRAILQA one (they also reported entity linking performance ), but the differences should be subtle enough.

I can share the script for references in any case. Just shoot me an email if needed.

[yhshu]

I got some little higher results.
Would you please provide a script? Thanks.

[xiye17]

sent

[yhshu]

Received, thanks.