Follow-up from @northdpole's reviews on #925 and #937 (Module C - The Librarian).
The gap: the eval harness (scripts/evaluate_librarian.py) already fails loudly on an explicit-slice regression - it exits non-zero if the "explicit" slice drops below 100% on the golden dataset. The problem is nothing in CI actually runs it. make test only runs the unit tests, so today that 100% gate exists on paper but isn't enforced. A change could quietly break the explicit slice and still merge green.
What we'd like to do: wire the harness into CI so it guards merges automatically.
- Add a small subprocess test (or a
Makefile target) that runs evaluate_librarian.py against the committed golden dataset (application/tests/librarian/fixtures/golden_dataset.json). If the explicit slice isn't at 100%, the check fails and blocks the merge. This runs fully offline - no API key needed.
- Later, extend the same setup to cover the C.1 retriever's recall@k. That part needs live embeddings (a populated cache DB + an embedding API key), so it can either be an optional CI job or a documented manual pre-merge check.
The offline explicit gate is the priority since it's deterministic and needs no credentials.
Scope: Module C. Not blocking W1–W3 - this just makes sure the harness is doing its job before the heavier ML slices (W4+) land.
Follow-up from @northdpole's reviews on #925 and #937 (Module C - The Librarian).
The gap: the eval harness (
scripts/evaluate_librarian.py) already fails loudly on an explicit-slice regression - it exits non-zero if the "explicit" slice drops below 100% on the golden dataset. The problem is nothing in CI actually runs it.make testonly runs the unit tests, so today that 100% gate exists on paper but isn't enforced. A change could quietly break the explicit slice and still merge green.What we'd like to do: wire the harness into CI so it guards merges automatically.
Makefiletarget) that runsevaluate_librarian.pyagainst the committed golden dataset (application/tests/librarian/fixtures/golden_dataset.json). If the explicit slice isn't at 100%, the check fails and blocks the merge. This runs fully offline - no API key needed.The offline explicit gate is the priority since it's deterministic and needs no credentials.
Scope: Module C. Not blocking W1–W3 - this just makes sure the harness is doing its job before the heavier ML slices (W4+) land.