Thanks for open-sourcing ToolSense — the QA, MCQ, and Realistic Retrieval datasets are a really useful resource. I noticed the repo currently exposes generation entry points only (generate-*, benchmark-*), so a downstream user has the data and the paper's headline metrics but no turnkey way to score a model on them.
I'd like to contribute a small evaluate/ module + CLI (eval-qa / eval-mcq / eval-rrb) that loads the shipped JSONL and reports QA vs. 50% / MCQ vs. 25% / RRB Recall@k, hit-rate, MRR, and nDCG over the analyzed_tools pool, reusing the existing LiteLLM dependency and adding a no-LLM --dry-run mode for CI. It's an inference-only, in-context scorer — explicitly not a reproduction of the trained-ToolGen Rc@50.
I'm happy to implement this and open a PR; just let me know if you'd welcome the contribution and want to assign the issue to me.
Thanks for open-sourcing ToolSense — the QA, MCQ, and Realistic Retrieval datasets are a really useful resource. I noticed the repo currently exposes generation entry points only (
generate-*,benchmark-*), so a downstream user has the data and the paper's headline metrics but no turnkey way to score a model on them.I'd like to contribute a small
evaluate/module + CLI (eval-qa/eval-mcq/eval-rrb) that loads the shipped JSONL and reports QA vs. 50% / MCQ vs. 25% / RRB Recall@k, hit-rate, MRR, and nDCG over theanalyzed_toolspool, reusing the existing LiteLLM dependency and adding a no-LLM--dry-runmode for CI. It's an inference-only, in-context scorer — explicitly not a reproduction of the trained-ToolGen Rc@50.I'm happy to implement this and open a PR; just let me know if you'd welcome the contribution and want to assign the issue to me.