How large does an eval set need to be to be useful? #24
Answered
by
martex-dev
martex-dev
asked this question in
Q&A
|
How large does an eval set need to be to be useful? Notes and answer below. |
Answered by
martex-dev
Aug 25, 2026
Replies: 1 comment
|
Smaller than most people expect for catching regressions, larger than most expect for ranking two close variants. Twenty carefully chosen examples covering the failure modes you actually care about will reliably catch a regression when a model version changes. Distinguishing two prompts that differ by a few percent needs enough examples that the difference exceeds sampling noise, which is usually in the low hundreds. |
0 replies
Answer selected by
martex-dev
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Smaller than most people expect for catching regressions, larger than most expect for ranking two close variants. Twenty carefully chosen examples covering the failure modes you actually care about will reliably catch a regression when a model version changes. Distinguishing two prompts that differ by a few percent needs enough examples that the difference exceeds sampling noise, which is usually in the low hundreds.