Fix /v3/memories 503: Firestore read-after-write in legacy ledger transaction

[Git-on-my-level]

Summary

POST /v3/memories is returning retryable 503 Service temporarily unavailable responses for beta/dev Desktop traffic because the legacy memory-ledger transaction performs a Firestore read after it has staged writes.

This is a follow-up to #9662's broader transaction-contract work, but is a distinct, reproducible ordering regression introduced while preserving canonical state-head fields in #9739.

User impact

• Desktop Sentry group: OMI-DESKTOP-2F3.
• 23 reported events across 7 users between 2026-07-14 20:36:58Z and 2026-07-15 00:48:52Z.
• Full event breadcrumbs show the failed request is POST https://api.omiapi.com/v3/memories (the beta/dev API), not the production sync-backfill endpoint.
• The effect is a failed new-memory write. The route returns before post-processing/vector upsert; do not claim successful persistence or data corruption from these failures.

Confirmed runtime evidence

At an exact Sentry failure time, dev Cloud Run backend returned 503 for /v3/memories in ~91 ms. The request neighborhood contains:

ERROR:routers.memories:Firestore create_memory failed
ReadAfterWriteError: Attempted read after write in a transaction.

The same route failure persisted across successive dev backend revisions; it is not explained by a single traffic shift or by the unrelated production backend-sync-backfill OOMs.

Root cause

#9739 added _state_head_write_payload() to preserve trusted canonical state-head fields when a legacy ledger write advances its independent head.

In backend/database/memory_ledger.py, both ledger transaction variants can execute in this order:

1. Read memory_state/head, the existing memory projection, and the commit document.
2. Call projection_writer(transaction), which stages a set() of the memory document.
3. Stage the commit write.
4. Evaluate _state_head_write_payload(...) for the final state-head write.
5. If trusted state-head fields are absent, _state_head_write_payload(...) reads memory_state/apply_control using the same transaction.

Firestore forbids step 5 after steps 2–3. It raises ReadAfterWriteError, which routers/memories.py:create_memory() catches and translates to HTTP 503.

The existing unit tests exercise the state-preservation semantics but their fake transaction does not enforce Firestore's read-before-write ordering, so they did not catch this.

Prescribed fix

Keep #9739's state-preservation behavior, but perform all possible transaction reads before the first write:

1. In both _append_commit_transaction and _append_commit_with_builder_transaction, build the final state-head payload (including any apply_control fallback read) before calling projection_writer(transaction) or any transaction.set(...).
2. Reuse that already-built payload for the final state_ref write. Do not remove canonical-head preservation or silently fall back to an untrusted synthetic head.
3. Add a regression test with a strict transaction fake (or Firestore emulator) that raises if get() occurs after set():
   • initial state lacks trusted canonical head fields;
   • an apply_control fallback is needed;
   • a legacy create_memory ledger write succeeds;
   • the resulting state preserves the validated canonical fields.
4. Cover both the direct append and builder append paths if both retain the helper.

Acceptance criteria

• The strict ordering regression test fails on the pre-fix code and passes with the fix.
• Legacy POST /v3/memories no longer produces ReadAfterWriteError / 503 for this path in dev.
• Canonical trusted state-head fields still survive legacy-ledger writes, preserving fix(memory): preserve trusted state head across ledger writes #9739's invariant.
• No raw user IDs, memory content, request bodies, or secrets are emitted in logs/tests.
• After dev deployment, verify a controlled write plus route-level 5xx/error-log rate; promote separately under normal release controls.

Related

• fix(memory): preserve trusted state head across ledger writes #9739 — introduced the canonical-head preservation path whose read must occur before writes.
• Prevent Firestore query/index and transaction-contract regressions in backend #9662 — closed broader Firestore transaction/index prevention work; this is a concrete read-after-write regression requiring a targeted fix.