v0.1.0 — the first release whose numbers you can check #148
akash-coded
announced in
Announcements
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
v0.1.0 · the harness is tagged.
Ten notebooks, a from-scratch toolkit, an eval gate that blocks a merge on a metric regression,
12 auto-graded labs, and a documentation site at
akash-coded.github.io/nanorag.
What is worth your attention
Three measurements that contradict the material, kept rather than tuned away. RRF losing to
weighted fusion, comparison starvation not reproducing, and no retrieval-score threshold separating
answerable from unanswerable. Each carries how to re-test it and the number that would
falsify it — see the README.
A fourth was added after a discussion thread: the learned reranker made retrieval worse at
every k, because a reranker over the same signals as the first stage is an expensive identity
function.
Eight ADRs, each with a falsifier. "What would change this" is a required section. A decision
without one is a preference.
What this release does not claim
Every number here was measured on a synthetic corpus this repository generated itself. That
choice is deliberate — gold labels are true by construction, so there is no annotation-error floor
under any result — and it means every finding has an untested precondition.
The extension project exists to find out which
of them survive contact with real data.
E0–E6, 17 issues, and a board field with three values:reproduced,contradicted,inconclusive.Contradicted is the most valuable outcome. It would say precisely what a synthetic corpus
cannot teach, which is a thing almost nobody writes down.
Where to start
docs/00-orientation/start-here.mdIf you find a number that does not reproduce
Open an
[errata]thread in Q&A. That is the highest-value contribution anyone can make here, andit has already happened once — #85 caught this repository quoting the
Landis–Koch κ bands as though they were a standard.
All reactions