-
Notifications
You must be signed in to change notification settings - Fork 19
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Qrels and Run query ids do not match #12
Comments
I think it's better if users do that on their own so that they are aware of what's happening. |
However, how to integrate this filter during autotune? Even during autotune, since the retrieve results depend on the queries and the training collection, it is unlikely that qrels contains all possible keys from run (although ideally, it will have an intersection with the run). |
You don't need the true relevance value for each query-doc tuple. |
I see. Since it is costly to obtain relevance feedback, I only have it for a general case. So when I try to autotune the retriever for different slices of data, there is no way to guarantee that the keys in qrels and run are identical, only that they intersect. Anyway, thank you. |
Well, can't you extract the intersection before tuning? |
Closing for inactivity. |
I guess this issue happens because in
qrels
, there are not all possible scores for all possible results in therun
.Wouldn't it be interesting to filter the
run
dictionary for only the evaluated cases that occur inqrels
?The text was updated successfully, but these errors were encountered: