-
Notifications
You must be signed in to change notification settings - Fork 433
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Allow trec_eval to take symbols representing standard qrels (instead of full qrel files) #2391
Comments
working on this |
@xpbowler any progress here? |
Sorry! I've been busy with finals these past 2 weeks. Last exam is tomorrow, so I'll get back on this after. |
@lintool What should happen in the case where we have two file names that would in theory have the same associated symbol? Ex. Symbol Would it be okay to simply require symbol names to be longer? |
In your example, As long as the file names are not identical (and they shouldn't be), it's fine. We specify which qrels we're using in the command line to |
For the symbols, let's use the bindings here: https://github.com/castorini/anserini/blob/master/src/main/java/io/anserini/eval/Qrels.java So instead of
We can just do:
|
hey @xpbowler just to prevent duplicate work, I think @DanielKohn1208 is on this! |
Currently, for
trec_eval
, we have to do something like:It would be great if we could do something like:
That is, take a symbol instead of full file.
Simple algorithm: take symbol, find the prefix that matches, e.g., so user can specify
msmarco-passage
instead ofmsmarco-passage.dev-subset
; behind the scenes, we download to~/.cache
.Something like this already works with Pyserini, we need to backport to Pyserini.
The text was updated successfully, but these errors were encountered: