-
Notifications
You must be signed in to change notification settings - Fork 1
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Candidate set #17
Comments
Hi, I'm glad you like our tool :) If you mean candidate set in the sense of "all entities that can potentially be linked by a system", i.e., the (sub)set of entities from the KB used by the system: If you want to know how well an entity linker performs on a given benchmark out of the box, then certainly its candidate set should be included in the evaluation. This means, ground truth entities that are not in the candidate set should not simply be ignored). In ELEVANT, we have two error categories related to the candidate set: "Wrong Candidates" and "Multiple Candidates". The error category "Wrong Candidates" shows the percentage of disambiguation errors (not counting NER errors) where the true entity was not in the candidate set. I hope this helps! |
Hi, it's a great work!
I would like to know if it is necessary to ensure that the ground truth is in the candidate set when evaluating Entity Linking, particularly for the REL method.
I look forward to your response.
The text was updated successfully, but these errors were encountered: