You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
A lenient test fake certifies broken code: #9739 introduced the state-head helper whose read-after-write regression shipped to prod (23 Sentry events, 503s on POST /v3/memories) because the ledger tests' transaction fake allowed orderings real Firestore forbids. #9780 (merged as PR #9783) fixed it and added a strict fixture — _StrictLedgerTransaction + companion db/document/collection types (backend/tests/unit/test_memory_ledger.py:132) — that enforces read-before-write, making the bug class untestable-around. This matters doubly because the watchdog auto-merge gate (#9822 item 5) uses "tests green" as its bar: that gate is exactly as strong as the strictest fake in the path.
Current state of the class, verified:
Firestore transactions — the live gap. Two independent strict fixtures now exist: the ledger one (models set + reference get only; no update/delete/queries) and an earlier candidate-lifecycle fake (backend/tests/unit/test_candidate_lifecycle.py:33) that also enforces read-before-write and does support update. Both are private to their test files. Every other transaction test in the suite builds its own ad hoc fake with whatever semantics its author remembered — the fix(memory): preserve trusted state head across ledger writes #9739 conditions exactly.
Rust Redis — already closed, correctly.services/redis.rs is tested against a scripted local RESP peer that drops an EVAL response, reconnects, and asserts the non-replayable command executed exactly once, plus generation-replacement and single-flight tests. Server-side recording is the right mechanism (a client-side fake cannot know whether an ambiguous request executed). Nothing to do.
So the actionable gap is one boundary, and the durable part is the default.
Fix: one shared strict Firestore transaction fixture + a strictness default, one small PR
Consolidate the two fixtures into one helper under the existing tests.unit helper convention: reconcile the ledger fixture (read-before-write on set/reference-get) with the candidate-lifecycle fake (update support) into a single strict transaction fixture; both test files consume it. Document its honest coverage boundary in its docstring: models set/update/reference-get ordering; does not model delete, queries, or retry semantics — extend on the next incident, don't speculate.
Strict by default, lenient by named opt-out. The shared fixture fails loudly on constraint violations; a test needing lenient behavior passes an explicit parameter naming what it relaxes — greppable and reviewable.
One guidance bullet in backend/AGENTS.md → Testing: "A fake for a service boundary must enforce that boundary's ordering/constraint semantics — use the shared strict transaction fixture for Firestore transactions; a lenient fake is how fix(memory): preserve trusted state head across ledger writes #9739 reached prod. When a boundary has no strict fixture, prefer a server-side recorder (see the Rust RESP peer in services/redis.rs tests) over a client-side fake."
Watchdog linkage: the auto-merge eligibility audit (Failure-class protocol and recurrence ratchets #9822 item 5) can reference this: transaction-touching backend subsystems are auto-merge-eligible only under the shared strict fixture.
Registry: FC-6 "lenient service fake", closed-by = shared strict fixture + strict-by-default convention; not-covered = boundaries without incident evidence and fake styles (MagicMock, module stubs) that no mechanical check can classify — recurrence there is caught by the sweep, not a ratchet.
Explicitly no ratchet in this issue: "ad hoc fake of a boundary" spans custom classes, MagicMock, fakeredis, monkeypatched globals, and Rust TCP peers — not mechanically specifiable without unacceptable false positives. If two migrations establish a stable banned construct, a narrow ratchet can follow.
Both existing suites (memory ledger, candidate lifecycle) pass on the shared fixture with no assertion changes.
Any third suite converted opportunistically that starts failing under strictness is triaged: real latent bug (filed) or legitimate leniency (named opt-out).
FC-6 registry entry with the fields above.
Cut list
No Rust changes — Redis and SSE are verified-covered; extraction of the RESP peer waits for a second consumer.
No bulk migration of every transaction test — the two source suites plus opportunistic conversions.
No mechanical ratchet (see above).
No new fakes for boundaries without incident evidence (GCS, Pinecone, etc.).
Problem
A lenient test fake certifies broken code: #9739 introduced the state-head helper whose read-after-write regression shipped to prod (23 Sentry events, 503s on
POST /v3/memories) because the ledger tests' transaction fake allowed orderings real Firestore forbids. #9780 (merged as PR #9783) fixed it and added a strict fixture —_StrictLedgerTransaction+ companion db/document/collection types (backend/tests/unit/test_memory_ledger.py:132) — that enforces read-before-write, making the bug class untestable-around. This matters doubly because the watchdog auto-merge gate (#9822 item 5) uses "tests green" as its bar: that gate is exactly as strong as the strictest fake in the path.Current state of the class, verified:
set+ referencegetonly; noupdate/delete/queries) and an earlier candidate-lifecycle fake (backend/tests/unit/test_candidate_lifecycle.py:33) that also enforces read-before-write and does supportupdate. Both are private to their test files. Every other transaction test in the suite builds its own ad hoc fake with whatever semantics its author remembered — the fix(memory): preserve trusted state head across ledger writes #9739 conditions exactly.services/redis.rsis tested against a scripted local RESP peer that drops anEVALresponse, reconnects, and asserts the non-replayable command executed exactly once, plus generation-replacement and single-flight tests. Server-side recording is the right mechanism (a client-side fake cannot know whether an ambiguous request executed). Nothing to do.So the actionable gap is one boundary, and the durable part is the default.
Fix: one shared strict Firestore transaction fixture + a strictness default, one small PR
tests.unithelper convention: reconcile the ledger fixture (read-before-write onset/reference-get) with the candidate-lifecycle fake (updatesupport) into a single strict transaction fixture; both test files consume it. Document its honest coverage boundary in its docstring: modelsset/update/reference-getordering; does not modeldelete, queries, or retry semantics — extend on the next incident, don't speculate.backend/AGENTS.md→ Testing: "A fake for a service boundary must enforce that boundary's ordering/constraint semantics — use the shared strict transaction fixture for Firestore transactions; a lenient fake is how fix(memory): preserve trusted state head across ledger writes #9739 reached prod. When a boundary has no strict fixture, prefer a server-side recorder (see the Rust RESP peer inservices/redis.rstests) over a client-side fake."closed-by= shared strict fixture + strict-by-default convention;not-covered= boundaries without incident evidence and fake styles (MagicMock, module stubs) that no mechanical check can classify — recurrence there is caught by the sweep, not a ratchet.Explicitly no ratchet in this issue: "ad hoc fake of a boundary" spans custom classes,
MagicMock,fakeredis, monkeypatched globals, and Rust TCP peers — not mechanically specifiable without unacceptable false positives. If two migrations establish a stable banned construct, a narrow ratchet can follow.Proof
Cut list
Sizing
One small PR: one shared fixture module (consolidating two existing ones), two test-file migrations, one AGENTS.md bullet, one registry entry.