Contributing and Testing

Contributing and Testing

Canonical setup instructions: CONTRIBUTING.md. This page covers the conventions — what a good test looks like here, and why.

Setup, briefly

pip install -e ".[dev]"
docker compose up -d --wait      # or: make db-up
pytest -v                        # or: make test
ruff check .                     # or: make lint

.[dev] gets you the engine, pytest, ruff and what the MCP tests need. Optional extras layer in as needed → Installation-and-Setup.

The tests need a real database

There is no mock DB. Database-touching tests run against a real PostgreSQL + pgvector container.

The reasoning is not purity. A mocked store cannot exercise the row-level-security policy, the in-place schema migration, the HNSW index behaviour, or the reconnect path — which is to say it cannot exercise the parts most likely to be wrong. A test suite that passes against a mock and fails against Postgres is worse than no suite, because it produces confidence rather than information.

The safety interlock

The suite DROP TABLEs.

So conftest.py reads RECALL_TEST_DSN — never RECALL_DSN — and refuses to run if the test DSN equals RECALL_DSN, or points at a non-local host without an explicit override variable. Exporting your real DSN and running pytest cannot touch your data.

Each test gets its own uuid-named table via fixtures and cleans up on teardown, so a normal run against the dev container puts nothing you own at risk.

Schema tests exercise the ordered SQL migrator and its checksum ledger, not only the deprecated compatibility wrapper. If you touch migrations, grants or readiness, run the schema-specific tests and regenerate the migration checksum file only when the migration bytes intentionally changed before release. Applied migration bytes are immutable by design.

Tests that self-skip

Tests needing an optional extra (the real-model rerank and entailment cases, the cloud-embedder test) skip themselves when the extra or its API key is absent. That is why CI passes without installing every extra.

⚠️ If you are touching one of those paths, install the relevant extra locally — otherwise your change is skipped rather than exercised, and the green run tells you nothing about it.

What a good test looks like here

The house rule: a test must fail for the right reason. Concretely, it must be impossible for a naive or vacuous implementation to satisfy it. Four shipped examples, each illustrating a distinct trap:

Test the restriction under the conditions where it applies. The RLS tests connect as a role that cannot bypass RLS. As a superuser they would pass while testing nothing — a green result proving only that the query returned rows. The privilege level of the test connection is part of the test.

Prove the precondition before asserting the absence. The cross-tenant test asserts the other tenant's row exists before checking it is invisible. Without that step, a silently failed write makes the isolation test pass — you cannot see what was never there.

Assert the invariant, not the outcome. The supersession-cache test counts real table scans. Asserting only "the right answer came back" would stay green if a fix quietly turned the cache into rescan on every search: correct results, catastrophic behaviour. The count is what makes the regression visible.

Assert on the real path, not the instrumented one. The metrics test asserts the counters move on the actual retrieval path. Instrumentation that is never wired up reports zero forever — and zero reads as "nothing is going wrong".

The generalisation: ask what a lazy implementation could do to make this test pass, and if the answer is anything other than "implement it correctly", the test is not finished.

Regression tests quote their input

Several defects were found only by running the library against a real corpus on a real server, and each carries a regression test quoting the exact input that caused it:

• a single NUL byte in one file aborting an index run over hundreds of files;
• every declared supersession edge failing on reference formatting rather than on a missing target;
• tests that encoded the developer's own environment and failed on a correctly-configured host.

That last category is worth internalising: a test that passes only on your machine is not a test, and the failure it produces elsewhere costs someone else the debugging time you saved.

CI

CI runs ruff, the suite against PostgreSQL, dependency audit over a checked-in lockfile, and claim artifact checks over published result files as gates, not reports. A red CI blocks.

Keep the lockfile in sync. The audit job verifies it before scanning; a drifted lockfile means the audit is scanning stale versions, so it is a hard failure rather than a warning. If you change a dependency, regenerate the lock and commit it in the same change.

Before opening a PR

From CONTRIBUTING.md, with the two that carry the most weight highlighted:

• ruff check . and pytest -v pass locally;
• the lockfile check passes if you touched dependencies;
• new behaviour has a test that would fail without the change — and that satisfies the "fails for the right reason" bar above, not merely a final-count assertion;
• if a published claim changes because of your PR — a number moves, a caveat needs updating — update it in the same PR. A stale published number is the failure mode this project exists to catch; reintroducing it in the project's own documentation would be self-refuting;
• commit messages describe why, not just what.

If your change touches retrieval, trust verdicts or calibration, re-run the evaluation harness and look at whether the numbers moved before claiming they didn't:

make eval

For LOCOMO, MTRAG, ladder rerank, promotion and enterprise retrieval changes, make eval is not the whole evidence surface. Use the study-specific scripts linked from Evidence-Map and update the retained artifacts or caveats in the same change.

Register and tone

From the contributing guide, and it is a real constraint rather than a style note: direct, technical, no marketing — a claim you can't measure is a claim you shouldn't make. The README is the reference for the register; the withdrawn-claims section is the reference for what honesty costs and why it is paid.

Where to start reading

• Architecture — module ownership and dependency direction.
• The-Trust-Layer — the conceptual core, and where changes are riskiest.
• recall/types.py — the whole data contract in under a hundred lines. The fastest orientation in the repo.
• Open issues — the known limits are filed, not hidden; several are linked from the README's "What this does not do".

---

See also: Installation-and-Setup · Evidence-Map · CODE_OF_CONDUCT.md · SECURITY.md