Repository navigation
Research collaboration inquiry: LLM judge calibration for Skill Lift #205
GBX-Max1220
started this conversation in
General
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
cc @chrisknvidia
Hi Christopher and the SkillEvaluator team,
I read the ACES paper and the Tier 3 evaluation documentation. One point in Section 7 caught my attention: live trajectory inter-judge agreement, human calibration, and uncertainty propagation are still unmeasured.
I've also looked at #62 and PR #124, so I'm not proposing to duplicate the existing uncertainty reporting or scoring fixes.
I'm interested in studying how judge disagreement affects paired Skill Lift estimates, especially whether the same agent trajectories can lead to different conclusions about whether a skill helps.
A small pilot could use public paired trajectories, independent human ratings, and multiple judge evaluations, with the underlying agent runs held fixed.
Before building anything, I wanted to ask whether this is a direction your team would find useful, and whether there's a publicly shareable paired-run subset or a recommended way to generate one from the reference skills.
If there's a fit, I'd be interested in contributing the experimental design, analysis, and writing as a research collaborator.
Best,
Baixin Guo (Max)
All reactions