The one number you wish you had measured six months earlier #141
akash-coded
started this conversation in
General
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
One reply each. The number, why you did not have it, and what it would have changed.
Mine is the ungrounded baseline — what the model scores on the eval set with no retrieved
context at all.
It costs one run. Without it, every
answer_correctyou report is uninterpretable, because youcannot tell the system's contribution from the model's prior knowledge. It is the right by
accident floor from The Four
Verdicts, and I have never once
seen it in a client deck.
Suggested shape
Answers that are a category of number rather than a specific one are fine — "anything with an
interval" is a legitimate entry and probably the most common one.
All reactions