One command that turns a client corpus into an eval set

[akash-coded]

The problem

The first real blocker on every engagement is the same: the client has no labelled data, and
until they do, nothing anyone changes can be shown to have worked.
Week 2 of the first-30-days playbook
is entirely this, and it is a week because the work is manual.

notebooks/02 teaches the method. Nothing automates it.

The proposal

python -m nanorag.evalgen ./their_docs --out eval.jsonl --n 200 --nulls 0.15

Extract entities and relations, generate multi-hop questions whose evidence is known by
construction, include null questions at a stated base rate, and emit records conforming to the
eval schema.

How we would know it worked

Not "it produces questions". Do the questions discriminate? Run two configurations known to
differ — say hybrid against dense-only — and check the generated set separates them with an
interval. A generated eval set on which every configuration scores identically is worse than no
eval set, because it looks like evidence.

What it costs

The generated questions inherit the generator's biases, exactly as this repo's synthetic corpus
does. That is a real limitation and it should be printed in the output header, not buried in
docs.