Replay real cohort questions as an eval slice — for distribution, not volume

[akash-coded]

Every cohort asks roughly the same twenty questions in Discussions, and those questions are
better than our synthetic eval set in one specific way: they are the questions people
actually have.

The idea: turn the Q&A category into an eval slice. Each answered thread becomes a
(question, expected-evidence) pair pointing at the notebook section that answers it. Then
measure whether a retriever over the repo's own documentation can find the right section.

Why it might be worth it

• It is a real corpus (our docs) with real questions (student threads) and real gold labels
  (the section the answer linked to)
• It would immediately show which docs are unfindable, which is a documentation metric nobody
  has
• It is a genuine second corpus for the toolkit, which currently only ever sees one

Why it might not be

• The corpus is tiny — maybe 400 chunks of docs — so N=100 is a full scan again
• Question volume is low; it would take several cohorts to get to a usable set size
• The gold label is "the section a human linked to", which is a weaker label than the fact
  graph gives us

What would make me confident: somebody counting how many distinct answered Q&A threads
exist after one cohort. If it is 15, this is not worth building. If it is 60, it might be.

[akash-coded]

Rafael · cohort peer · came from search infra

Strongly in favour, and I think the value is different from the one in the proposal.

The proposal frames it as more eval data. The bigger win is distribution. Our synthetic eval set is generated from a fact graph, so its questions are uniformly well-formed: complete sentences, correct entity names, unambiguous intent. Real cohort questions are none of those. They have typos, missing context, two questions in one, and pronouns with no antecedent.

A retriever tuned on well-formed questions and deployed against real ones degrades in ways the synthetic set cannot show you. The replay slice is valuable precisely because it is messy, and that should be the headline argument rather than the sample count.

[akash-coded]

Noor · cohort peer · works in a regulated bank

Adding the constraint that will decide whether this is buildable: if you replay real questions you are handling real user data, and in my industry that needs consent and a retention policy before the first question is stored.

Not an objection — a design requirement. Anonymise at capture, not later; store the question text and nothing that identifies who asked it; and put a TTL on the slice. It is much easier to build this correctly than to retrofit it.

[akash-coded]

Both right, and Noor's point promotes this from an idea to something with a design. Capturing the shape:

• Value is distributional realism, not volume — that is the sentence that goes in the issue.
• Capture anonymises at write time; question text only; TTL enforced.
• Labelling is the cost nobody has priced. Real questions have no gold evidence, so somebody has to label them, and that is the work. Probably a small hand-labelled slice rather than the whole stream.
• Open question worth its own thread: if the replay slice disagrees with the synthetic slice, which one gates a release? I do not think the answer is obvious and I would rather argue it before we have the data than after.

[akash-coded]

Filed as #57.

The issue leads with Rafael's reframing rather than the original one: the value is distributional realism, not sample count. That is the sentence that justifies building it, and the version in the opening post would have got it scoped as "more eval data" and deprioritised.

Noor's constraints are acceptance criteria, not notes — anonymise at capture, question text only, TTL enforced rather than documented. Retrofitting any of those is much harder than building them in.

The labelling cost is called out explicitly as the real cost and left unscoped on purpose, because I do not have an honest estimate for it yet.

The open question — if the replay slice disagrees with the synthetic slice, which one gates a release? — is in the issue but should be argued in its own Design Review thread. Deciding it after the data exists means deciding it in the presence of a number somebody wants.