Repository navigation
EvalPort: optional export for EleutherEvalRecipe results #2999
adhabnr-ux
started this conversation in
Ideas
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Hi torchtune team — I maintain EvalPort (https://github.com/adhabnr-ux/evalport), an open JSON-Schema spec for portable LLM eval test cases, graders, and result sets.
recipes/eleuther_eval.py'sEleutherEvalRecipe.evaluate()runs EleutherAI's harnessevaluate()overself.tasksand formats the per-task output withmake_table()— the harness's own result dict already maps cleanly onto EvalPort's ResultSet (one GraderResult per task/metric). A thin adapter that dumpsoutputas an EvalPort ResultSet alongside (or instead of) the printed table would let atune run eleuther_evalrun be compared against other checkpoints/frameworks in one interchange format rather than a recipe-specific text table. Just flagging the fit, not asking you to prioritize it.Schemas: https://github.com/adhabnr-ux/evalport/tree/main/spec/schemas
— Sahi, independent contributor (not affiliated with torchtune)
All reactions