Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support evaluating with incomplete gold standards #17

Open
jnothman opened this issue Aug 4, 2015 · 0 comments
Open

Support evaluating with incomplete gold standards #17

jnothman opened this issue Aug 4, 2015 · 0 comments

Comments

@jnothman
Copy link
Member

jnothman commented Aug 4, 2015

A user on the TAC mailing list wished to evaluate a mix of 2014 and pre-2014 EL tasks, such that precision should indicate precision of linking/clustering, ignoring spurious mentions.

While it is not hard to remove spurious mentions using grep, this may be worth facilitating in one of the following ways:

  1. a command to output the subset of a dataset that aligns to the gold standard.
  2. an --ignore-spurious flag to evaluate, significance and confidence.
  3. an is_aligned attribute on Annotations that is True for all gold annotations and set with respect to some gold standard when loading annotations from a system output.

(3.) would appear to be most flexible and in line with current design.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant