Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add AnswerExactMatchEvaluator #7050

Merged
merged 4 commits into from
Feb 23, 2024
Merged

Add AnswerExactMatchEvaluator #7050

merged 4 commits into from
Feb 23, 2024

Conversation

silvanocerza
Copy link
Contributor

@silvanocerza silvanocerza commented Feb 21, 2024

Related Issues

Proposed Changes:

Add AnswerExactMatchEvaluator. This Component calculates the Exact Match metrics given a list of questions, a list of expected answers for each question and the list of predicted answers for each question.

How did you test it?

I added unit tests.

Notes for the reviewer

N/A

Checklist

@silvanocerza silvanocerza self-assigned this Feb 21, 2024
@github-actions github-actions bot added 2.x Related to Haystack v2.0 type:documentation Improvements on the docs labels Feb 21, 2024
@coveralls
Copy link
Collaborator

coveralls commented Feb 21, 2024

Pull Request Test Coverage Report for Build 8006956131

Details

  • 0 of 0 changed or added relevant lines in 0 files are covered.
  • No unchanged relevant lines lost coverage.
  • Overall coverage increased (+0.004%) to 89.806%

Totals Coverage Status
Change from base Build 8006113790: 0.004%
Covered Lines: 5180
Relevant Lines: 5768

💛 - Coveralls

@silvanocerza silvanocerza changed the title ExactMatchEvaluator first draft Add AnswerExactMatchEvaluator Feb 22, 2024
@silvanocerza silvanocerza marked this pull request as ready for review February 22, 2024 14:42
@silvanocerza silvanocerza requested review from a team as code owners February 22, 2024 14:42
@silvanocerza silvanocerza requested review from dfokina, ZanSara and shadeMe and removed request for a team and ZanSara February 22, 2024 14:42
Copy link
Collaborator

@shadeMe shadeMe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some minor changes, but LGTM otherwise! Let's also get @julian-risch to sign off.

@@ -0,0 +1,3 @@
from .answer_exact_match import AnswerExactMatchEvaluator
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now that the metrics are (in)directly associated with specific upstream components, I don't think we need an additional statistical submodule after all.

haystack/components/evaluators/answer_exact_match.py Outdated Show resolved Hide resolved
Copy link
Member

@julian-risch julian-risch left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! 👍 The docstring describes the metric really well and the implementation of the metric is easy to read and correct. 👍

@silvanocerza silvanocerza merged commit b4011af into main Feb 23, 2024
23 checks passed
@silvanocerza silvanocerza deleted the exact-match-evaluator branch February 23, 2024 09:37
@shadeMe shadeMe restored the exact-match-evaluator branch February 23, 2024 11:25
silvanocerza added a commit that referenced this pull request Feb 23, 2024
silvanocerza added a commit that referenced this pull request Feb 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
2.x Related to Haystack v2.0 topic:tests type:documentation Improvements on the docs
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Implement function to calculate Exact Match metric
4 participants