The notebook gives different numbers than the README. Which is right?

[akash-coded]

Casebook thread. A reconstructed conversation, written for teaching. The people in it
are illustrative; every number is real and reproducible from the notebook cited below.
Replies are welcome — if you hit this yourself, add your numbers.

The question in one line

I ran notebook 01 and got different evidence recall than the README quotes.

What I have already tried

Re-ran the cell. Same result.

Where

Notebook 01 §1.3 vs README

[akash-coded]

Almost always a stale kernel holding an older nanorag module. Restart the kernel and run
all cells from the top. bootstrap() pins the seed but it cannot un-import a module Python
already loaded.

If it persists after a clean restart, work through these in order — this is a small version of
the diagnostic discipline the whole curriculum teaches:

1. Are you comparing the same thing? The README quotes the TUNED configuration over the
   full 243-question set. Several notebook cells run on a 60–90 question sweep subsample to
   stay fast, and say so in the cell. A different population gives a different number and
   neither is wrong.
2. Same slice? Some cells report dev only; the frozen slice is reported separately.
3. Did you change something upstream? A different chunking strategy or dim changes
   everything downstream. quickstart() prints its configuration — compare that line.
4. Genuinely different environment? Different NumPy versions can produce tiny floating-point
   differences in the SVD. If your numbers differ in the third decimal, that is this and it does
   not matter. If they differ in the first, it is one of 1–3.

If you have ruled all four out, that is a bug and worth an issue — with your
quickstart() output line, your Python and NumPy versions, and both numbers. There is a
seeded example of exactly that shape in the tracker.

The habit worth taking from this: "the numbers disagree" is not one question, it is four,
and asking them in order takes about two minutes. That is the same move as the encoder-swap
checklist in notebook 04 §4.7, at a smaller scale.

---

_{Seeded by faculty as a worked example of the format. Start your own thread rather than replying here unless you have something to add.}

[akash-coded]

Priya · cohort peer

Adding the checklist I wish I had had, because I lost an afternoon to this exact thing and it was none of the obvious causes.

Ordered by how often it turned out to be the answer:

1. Different k or N. The README quotes the default config; a notebook cell three sections earlier may have rebound cfg. Print the config next to the number, always.
2. Different chunking strategy. structural and fixed give materially different evidence recall on the same corpus. If you ran the bake-off cell, you may still be on the last strategy it tried.
3. Different slice. dev and frozen are different question sets. The README quotes frozen.
4. A stale kernel. You changed a parameter, re-ran the eval cell but not the indexing cell.

The general habit that makes all four go away: never report a number without the config that produced it. pipeline.RagConfig has a __repr__ for this reason — print it in the same cell as the metric.

[akash-coded]

Dana · reviewer · owns the eval harness

Endorsing point 4 specifically, because it is the one that survives being careful.

Notebook state is not the order of the cells on screen; it is the order you ran them. Nothing in the notebook tells you they differ. The check that costs nothing:

import json, hashlib
from nanorag.pipeline import RagConfig

fingerprint = hashlib.sha256(
    json.dumps(cfg.__dict__, sort_keys=True, default=str).encode()
).hexdigest()[:12]
print(f"config {fingerprint}  n_chunks={len(chunks)}  n_questions={len(eval_set)}")

Print that line above every metric you intend to quote anywhere. If two runs disagree, the fingerprints tell you in one second whether you are looking at a real difference or a stale kernel — which is the difference between a five-minute check and an afternoon.