Repository navigation
Evaluating yes/no/unknown: refuted conditions vs insufficient evidence, separately from abstention #1497
Unanswered
murillodutt
asked this question in
Q&A
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
I'm seeking evaluation-design advice from people using Laya for typed decisions, not reporting a confirmed Laya bug or presenting a Laya benchmark.
Our application-level question is whether work has already been admitted to a current, bound supervision cycle:
Here unknown is a semantic answer, not low confidence, model refusal, a missing output, or a threshold-based abstention. It does not itself trigger supervision. Whether supervision would be useful is a separate judgment.
Synthetic illustrations only, untested and not independently adjudicated:
In an existing development set outside a Laya evaluation, 48 of 51 reference-unknown cases were classified as no. These cases already informed development, so this is a reference-disagreement pattern, not independent validation or proof that our references are correct.
For a prospective three-option evaluation, what controls would you recommend to distinguish:
Would matched groups that keep the proposition fixed while varying supporting evidence, authoritative refutation, and incomplete records be a useful diagnostic? How would you independently adjudicate them and prevent lexical shortcuts, especially absence from a complete authoritative record versus absence from a partial record?
Most importantly, what observable evidence would discriminate these competing explanations, rather than merely suggest another prompt rewrite? Criticism of the definition and synthetic illustrations is welcome, as are existing contrast sets or validated interventions.
For transparency, I have also asked the broader benchmark question in EleutherAI: EleutherAI/lm-evaluation-harness#4600 . This question focuses on controls relevant to typed-decision models and their SDKs; no claim is made that the disagreement counts above were produced by Laya.
Murillo Dutt
All reactions