Repository navigation
Benchmarks or contrast sets for distinguishing refuted from not-established conditions? #4600
Unanswered
murillodutt
asked this question in
General
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
I'm looking for existing evaluation tasks or contrast-set methods, rather than reporting a harness bug. Our custom workflow classifier runs outside lm-evaluation-harness.
One field asks whether work has already been admitted to a current, bound supervision cycle:
unknownis an application-level answer, not model refusal, a missing output, or a confidence threshold. It does not itself trigger supervision. Whether supervision would be useful is a separate question.Synthetic illustrations only, untested and not independently adjudicated:
In 200 saved development cases, reference counts were 18 yes / 131 no / 51 unknown. The classifier returned no for 48 reference-unknown cases and 6 reference-yes cases; another 2 reference-yes cases became unknown. Agreement was 144/200 (72%), versus 65.5% for always-no. These cases already informed development: these are reference disagreements, not independent validation or proof that the references are correct.
What existing benchmarks, tasks, or contrast-set construction methods would you use to test this boundary? In particular:
I read #4233 on separating format compliance from correctness. This question is about a different semantic distinction; a well-formed yes/no/unknown output is not enough. Pointers to existing tasks before designing a new diagnostic set, and criticism of the definition or illustrations, would be especially useful.
Murillo Dutt
All reactions