Skip to content

CI: wire the Librarian eval harness in as a merge gate (explicit 100% + optional recall@k) #956

Description

@PRAteek-singHWY

Follow-up from @northdpole's reviews on #925 and #937 (Module C - The Librarian).

The gap: the eval harness (scripts/evaluate_librarian.py) already fails loudly on an explicit-slice regression - it exits non-zero if the "explicit" slice drops below 100% on the golden dataset. The problem is nothing in CI actually runs it. make test only runs the unit tests, so today that 100% gate exists on paper but isn't enforced. A change could quietly break the explicit slice and still merge green.

What we'd like to do: wire the harness into CI so it guards merges automatically.

  • Add a small subprocess test (or a Makefile target) that runs evaluate_librarian.py against the committed golden dataset (application/tests/librarian/fixtures/golden_dataset.json). If the explicit slice isn't at 100%, the check fails and blocks the merge. This runs fully offline - no API key needed.
  • Later, extend the same setup to cover the C.1 retriever's recall@k. That part needs live embeddings (a populated cache DB + an embedding API key), so it can either be an optional CI job or a documented manual pre-merge check.

The offline explicit gate is the priority since it's deterministic and needs no credentials.

Scope: Module C. Not blocking W1–W3 - this just makes sure the harness is doing its job before the heavier ML slices (W4+) land.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions