Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add AnswerF1Evaluator for answer evaluation #7606

Open
leomaurodesenv opened this issue Apr 27, 2024 · 0 comments
Open

feat: add AnswerF1Evaluator for answer evaluation #7606

leomaurodesenv opened this issue Apr 27, 2024 · 0 comments

Comments

@leomaurodesenv
Copy link

Is your feature request related to a problem? Please describe.
Based in discussion #7395, we can see that it is missing the F1-score evaluator for Extractive QA algorithms in Haystack 2.0.

Describe the solution you'd like
As stated by @julian-risch

Calculating the F1 score is a bit more complicated than the exact match because the F1 score is token based. So the evaluator first needs to tokenize the predicted answer and the ground truth answers. Then it needs to calculate precision and recall based on those tokens and then calculate the harmonic mean of those to get the final F1 score.

Thus, this issue is for requesting the AnswerF1Evaluator class, similar to #7050 . My proposal is following the tradicional "formula" used in SQuaD dataset article, paper link. Here you can see a sample of computing script

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants