Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] Missing results causes AssertionError #39

Closed
amallia opened this issue Apr 14, 2023 · 1 comment
Closed

[BUG] Missing results causes AssertionError #39

amallia opened this issue Apr 14, 2023 · 1 comment
Labels
bug Something isn't working

Comments

@amallia
Copy link

amallia commented Apr 14, 2023

Missing results causes AssertionError: Qrels and Run query ids do not match
You should make check_keys optional in evaluate because sometimes queries do not return any results for lexical-based systems.

ranx/ranx/meta/evaluate.py

Line 128 in da0aa52

check_keys(qrels, run)

@amallia amallia added the bug Something isn't working label Apr 14, 2023
@AmenRa
Copy link
Owner

AmenRa commented Apr 17, 2023

Hi, I've added the make_comparable function to Run in v0.3.7.

run.make_comparable(qrels)

The run will be modified based on the qrels passed as an argument.
Query results not appearing in qrels will be removed from run to avoid unpredictable behavior or other bugs.
For convenience, I also added the boolean parameter make_comparable to evaluate and compare (it defaults to False).

Please, consider giving ranx a star if you like it.

Best,

Elias

@AmenRa AmenRa closed this as completed Apr 17, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@amallia @AmenRa