Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Custom k for cut metrics #12

Open
ghost opened this issue Feb 27, 2019 · 3 comments
Open

Custom k for cut metrics #12

ghost opened this issue Feb 27, 2019 · 3 comments

Comments

@ghost
Copy link

ghost commented Feb 27, 2019

Hi,

Is there any way to specify custom k values for map_cut, ndcg_cut etc., instead of default ones ?

It is definitely possible with trec_eval, but I don't see how to do it with the current interface (or do I miss something ?)

Thanks

Thibault

@seanmacavaney
Copy link
Contributor

@tformal I added support for custom thresholds using the same convention as in trec_eval. E.g., {'ndcg_cut.1,15,100'} will give ndcg_cut_1, ndcg_cut_15, and ndcg_cut_100 results.

If this sounds good, I'll go ahead and make a PR.

@ghost
Copy link
Author

ghost commented Feb 25, 2020

@seanmacavaney this sounds perfect !

@seanmacavaney
Copy link
Contributor

Do we also want to support a more natural way of requesting specific thresholds? I.e., {'ndcg_cut_1', 'ndcg_cut_15', 'ndcg_cut_100'} instead of {'ndcg_cut.1,15,100'}? This could be done in a python wrapper that converts the former into the latter.

Inspired by https://twitter.com/lintool/status/1232311794116501504

cvangysel pushed a commit that referenced this issue Mar 2, 2020
* support measure family nicknames #17

* Custom k for cut metrics #12

* support for alternative (nicer) formats for measure params #12. Built wrapper around cpp module which converts alternative formats to trec_eval format.

* plugged memory leak #15

* removed type hints for python<3.5

* removed type hints for python<3.5

* Several fixes
1) Fixed issue with empty qrels on some platforms
2) Exposed the values nicknames expand to and moved logic to wrapper
3) Some cleanupq

* bump version to 0.5

* bump version to 0.5
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant