-
Notifications
You must be signed in to change notification settings - Fork 31
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add rerank recall metric to unitxt #662
Conversation
@yoavkatz please review |
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #662 +/- ##
==========================================
- Coverage 91.13% 90.87% -0.27%
==========================================
Files 98 98
Lines 9897 9937 +40
==========================================
+ Hits 9020 9030 +10
- Misses 877 907 +30 ☔ View full report in Codecov by Sentry. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fix the requirements according to unitxt no-requirements policy explained in the comments
You need to add the pytrec_eval to the |
@elronbandel pytrec-eval is added. |
@yoavkatz The I have precommit installed now. |
There is a missing import to pandas. In general, if metrics needs specific packages like pandas or trec_eval, we import inside the relevant metric code, and not at the top of the metrics.py file, so If people don't need the metric, they will not need the include. |
@elronbandel I already fixed the requested change but github still thinks it's pending. |
@yoavkatz can you tell what's holding up this merge? There appears to be an outstanding change request but I already fixed that a while ago. |
I reviewed the code, and it looks fine so I approved. @elronbandel - I think we the main issue is that the coverage is low - although tests were added (but in the prepare file). We should probably give guidance to write the test in test_metrics and not in the prepare file (unless using a wrapper like HuggingFaceMetric). |
Closes #661 in support of adding perplexity reranking to fm-eval.