Repository navigation
Evaluating multi-question decisions: per-question gains versus whole-case reliability #1500
Unanswered
murillodutt
asked this question in
Q&A
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
For Laya users evaluating several typed questions on the same input, how do you compare checkpoints when per-question accuracy and whole-case reliability move in opposite directions?
Hypothetical example, not a reported benchmark: an update improves several frequent questions but concentrates errors in one important output, or spreads formerly clustered errors across more cases. Mean cell accuracy can improve while fewer cases have every required answer correct. Multiplying marginal accuracies would assume independence that the shared input does not justify.
The comparison I am considering would keep the question schema, label definitions and held-out cases fixed, then report:
For uncertainty, I would preserve each case's output vector when resampling, and group related variants together rather than treating every question as an independent observation. With unequal numbers of applicable questions, I would also separate cell-weighted results from case-weighted results.
Is there an existing Laya evaluation/reporting pattern or worked example for this? In particular, what would you use as the primary comparison and uncertainty estimate when rare critical errors are too sparse for a reassuring aggregate interval? I am interested in a practical protocol that does not select the winning metric after seeing the results. This is an evaluation-design question, not a claim of a Laya defect or measured improvement.
All reactions