How do I talk about a synthetic-corpus project without it sounding like a toy?

[akash-coded]

Casebook thread. A reconstructed conversation, written for teaching. The people in it
are illustrative; every number is real and reproducible from the notebook cited below.
Replies are welcome — if you hit this yourself, add your numbers.

I want to put this project on my CV but I am worried the first question will be "isn't the
data fake?" and that I will not have a good answer.

How do I frame it honestly without undercutting myself?

[akash-coded]

Lead with the framing rather than defending it, and the question mostly does not get asked.

The move is to make the synthetic corpus the reason the project is rigorous, which is
true:

"The corpus is generated from a fact graph, which means the gold evidence is true by
construction — there's no annotation-error floor under any number I report. That's what let
me measure things like the reranker's ceiling exactly rather than approximately. What it
costs is external validity, and the repo says so explicitly — there's a real-versus-stand-in
table on the front page."

Three things that does: turns the apparent weakness into a design decision, demonstrates you
know what it costs, and shows you documented the limitation before anyone asked.

Then move to what actually transfers, quickly, because that is where the interesting
conversation is:

"The absolute numbers don't transfer and I wouldn't quote them. What transfers is the method
— measure before you change, one change at a time, every delta with an interval against a
measured noise band. Two of my four improvements turned out to be inside that band and
they're in the decision record as rejected."

That last sentence is the one that lands. Almost no candidate volunteers a negative result, and
an interviewer who hears it stops assessing whether you can build RAG and starts assessing how
senior you are.

Have these ready for the follow-ups:

• "Why not use a real dataset?" — the record schema matches MultiHop-RAG and there is a loader
  for it; the constraint was that the whole curriculum has to run offline in seconds, which is
  ADR-0002.
• "So you didn't work with real data?" — separate the claim. You built and measured a system;
  you did not do a client engagement. If you have done client work, that is a different bullet.
  Do not blur them.
• "What would you do differently?" — have a real answer. "Double the eval set; the noise band
  at 207 questions is ±0.06 and several of my deltas sit inside it" is a good one because it
  shows you understand the limit of your own evidence.

What not to do: do not oversell it as production experience, and do not apologise for it
either. It is a rigorous piece of engineering with a stated scope. Say that, and move to the
decision record — which is the artefact almost nobody else in the pile will have.

---

_{Seeded by faculty as a worked example of the format. Start your own thread rather than replying here unless you have something to add.}

[akash-coded]

Marcus · cohort peer

Blunt version: do not lead with the corpus. Lead with the measurement.

Nobody asks "is the data fake" about a project whose headline is "I built an evaluation harness that blocks a merge on a metric regression." They ask it about a project whose headline is "I built a RAG system." The first is a claim about your engineering; the second invites a question about your data.

[akash-coded]

Priya · cohort peer

True but incomplete — the question still comes, just later, and answering it badly at that point is worse because it looks like you were avoiding it.

[akash-coded]

Marking this as the answer since it is the framing question rather than a technical one, and getting it wrong costs people interviews.

The reframe: synthetic is a methodological choice, and you should say so first.

The defensive answer is "it is synthetic, but…". The correct answer is that you chose it, on purpose, for a reason that is hard to argue with:

"The corpus is generated from a fact graph, which means the gold labels are true by construction — there is no annotation-error floor underneath any number I report. On a hand-labelled set, a 2-point recall difference can be annotator disagreement. On this one it cannot be. That was the point: I wanted to teach measurement, and measurement is unteachable if the ground truth is itself uncertain."

That is not a defence. It is a reason a careful person would make the same choice.

Then volunteer the limitation before they ask, which is what actually builds credibility:

"What it does not give me is distributional realism. Real corpora have duplicates, contradictions, OCR noise and documents that are wrong. My retriever never had to survive any of that, and I would expect the absolute numbers to move on a real corpus. What transfers is the harness — the metrics, the eval-set construction, the CI gate — not the constants."

Why this works. You have demonstrated the thing panels are actually screening for: knowing which parts of your own result generalise. A candidate who defends their numbers as universally valid is a risk. A candidate who says "the method transfers, the constants do not" is describing how they will behave on the client's data, which is the only thing the interview is really about.

Two failure modes to avoid.

• Over-apologising. "It's only synthetic, so it's not really representative" concedes more than is true and invites them to discount the whole project.
• Over-claiming. Do not say it is "like production data". It is not, they know it is not, and one sentence of over-claim costs you the credibility of everything else you said.

If you have thirty extra minutes before the interview, the strongest possible version of this is to run the harness against a real corpus — MultiHop-RAG plugs into the same record schema, and notebook 02 documents the mapping. Then the answer becomes "I ran it on both; here is what moved and what did not", and the question stops being a challenge and becomes the most interesting thing you said.