ledger: /verify reports ok:false — content_mismatch at seq 5 on a 3-day-old chain

[JSONbored]

Observed on a live self-host instance

GET /v1/public/decision-ledger/verify
{"ok":false,"checked":4,"tipSeq":1639,"totalCount":1639,"prunedRecords":0,
 "break":{"kind":"content_mismatch","atSeq":5,
          "recordId":"record:JSONbored/loopover#8921@a2446db8e808529d19803ada464cc9d605a89638"}}

Verification stops at the first break, so 4 of 1,639 rows are checked and the endpoint publicly reports
ok: false. Anyone running the documented verification against this deployment sees a chain break.

Why this is probably not tampering

The chain is only three days old — seq 1 @ 2026-07-26T13:40:09Z, seq 1639 @ 2026-07-29T14:01:03Z — and
prunedRecords: 0, so the legitimate pruned-preimage explanation does not apply. The instance has been
tracking beta images across that window (currently orb-v3.6.0-beta.6).

The likely cause is that the decision-record preimage changed between the release that wrote seq 5 and the
release verifying it now. Recomputing the digest over the current shape yields a different value than the one
committed at write time, and the content check correctly reports a mismatch.

Why that matters more than one bad row

If the record preimage is not stable across versions, then every schema change silently invalidates the
content check for every row written before it — and the published /verify endpoint starts reporting
ok: false on an untampered ledger. The chain-structure guarantees (sequence, prevHash, rowHash) still hold;
it is specifically the content re-check that breaks.

That inverts what the surface is for. A verifier cannot distinguish "someone edited a record" from "the
software changed shape", which is exactly the discrimination the ledger exists to provide.

What needs deciding

• Is the record preimage intended to be version-stable? If yes, this is a bug in whatever changed it, and
  the fix is a versioned preimage (a schemaVersion in the digest input, with the verifier recomputing under
  the row's own version) rather than a re-hash of history.
• If it is NOT intended to be stable, /verify needs to say so — the same way prunedRecords already
  carves out legitimately-unre-checkable rows — so a shape change does not read as tampering.

Reproduce

Any deployment whose ledger spans a release that altered the decision-record shape. Confirmed above on a
3-day-old chain.

Links

src/review/decision-record.ts, src/review/ledger-anchor.ts, the /v1/public/decision-ledger/verify
handler in src/api/routes.ts. Related: #9271, #9267.

[linear-code]

JSO-6873

[JSONbored]

Diagnosed against the live database — my original hypothesis was wrong

I guessed canonicalization drift across beta releases. It is not that. The record content round-trips
perfectly; the ledger committed a digest for content that is not what the record now holds.

Recomputing with the runtime's own contentDigest over the stored record_json:

seq 4 | ledger 5987e8d984cd | record's own 5987e8d984cd | recomputed 5987e8d984cd | agree | roundtrips true
seq 5 | ledger e66db5eb697d | record's own e3b1d83a45fa | recomputed e3b1d83a45fa | DISAGREE | roundtrips true

So the record is self-consistent (own == recomputed) and canonicalization is fine (roundtrips true). Only
the chain row is out of step.

What actually happened

seq | record_id                                              | digests_agree | record.created_at        | ledger.created_at
  5 | record:JSONbored/loopover#8921@a2446db8e808529d1980... | f             | 2026-07-26T13:47:36.088Z | 2026-07-26T13:44:14.737Z
  7 | record:JSONbored/loopover#8921@a2446db8e808529d1980... | t             | 2026-07-26T13:47:36.088Z | 2026-07-26T13:47:36.091Z

Two ledger rows, the same record id, and the record's own created_at is the LATER one. The PR was
decided at 13:44 (seq 5 chained that digest), re-decided at 13:47, and the record row was replaced in
place rather than revisioned — so seq 5 now points at content it never committed to.

persistDecisionRecord's revision scheme exists precisely to prevent this: it counts priors and inserts
...:rev2, with a PK-collision retry, and its own comment says "a supersession's OWN new row, so
re-decisions are visible history rather than silent replacement." It did not engage — both rows carry
the base id, unsuffixed.

Scope: not a one-off

83 / 1654 ledger rows commit a digest that disagrees with their record

5% of the chain. Any full verification stops at the first one (seq 5 here, after checking 4 rows), so
/v1/public/decision-ledger/verify publicly reports ok: false on an untampered ledger.

What I ruled out

• Not canonicalization drift — canonicalJson(parsed) === stored record_json exactly.
• Not the pg dialect — decision_records is not in REPLACE_CONFLICT_KEYS, and the rewriter only touches
  INSERT OR REPLACE / INSERT OR IGNORE; this is a plain INSERT.
• No UPDATE of decision_records anywhere in src/.

So the mechanism by which the row is replaced is not yet identified, and that is the remaining work. I am
deliberately not guessing at a fix: this is ledger integrity, and the difference between "the revision
counter races" and "something else overwrites the row" leads to different fixes.

Suggested next step

Reproduce a re-decision at the same head SHA against Postgres and watch whether priorCount sees the
existing row. The count-then-insert is not atomic; two decisions ~3 minutes apart should not race, which
makes a plain race unlikely and points at priorCount returning 0 when it should return 1.

Related: #9742 (verdict immutability per head SHA) is the same territory and may be the intended fix.

[JSONbored]

Diagnosed against the live chain — it is a historical write bug, already fixed, and NOT tampering

Measured on edge-nl-01 (now orb-v3.6.0-beta.8), recomputing every one of the first 500 ledger rows with this repo's own contentDigest:

schemaVersion	verified	mismatched
v2	70	21
v3	182	62
v5 (current)	165	0

Every record written by the current shape verifies. The 83 mismatches are all pre-v5.

What it is not

Three hypotheses ruled out with data, not reasoning:

1. "The preimage shape changed across releases." Then all v2/v3 rows would fail. They don't — 252 of 335 verify fine.
2. Supersession rewrites (:rev2 ids, "latest-finalize-wins"). Rows with revision ids are 79/79 clean — zero mismatches.
3. Non-canonical storage. For the first failing row (seq 5), the stored record_json is already byte-identical to canonicalJson(JSON.parse(...)), and re-canonicalising is a fixed point. Passing and failing v3 records have identical key sets.

What it is

For seq 5: the stored preimage is canonical and self-consistent, yet contentDigest(stored) ≠ record_digest. So at write time the digest was computed over an object that differs from the one that was stored. That is a write-path bug in the v2/v3-era code — and it is unrepairable after the fact: you cannot recover a digest that was wrong when it was committed.

The current write path is correct (165/165 at v5), so this is a bounded, historical population that will age out of the retention window.

The decision this needs — and it is a real trade-off

The content re-check on pre-v5 rows currently produces false positives, which is worse than useless: ok:false on an untampered chain, and (until the recent fix to keep scanning) it hid every structural break after the first one. A verifier cannot distinguish "someone edited a record" from "written by buggy old code" — exactly the discrimination this endpoint exists to provide.

The obvious fix is to treat pre-current-schema rows as content-unverifiable-by-construction — counted in their own field like prunedRecords, excluded from the ok verdict, with chain-structure checks (seq / prevHash / rowHash) still fully applied.

But that genuinely reduces tamper detection for those rows: an edit to a pre-v5 record_json would stop being detectable by content, leaving only the structural guarantees. The counter-argument is that for those rows the content check already provides no signal — it fails regardless — so making it explicit is strictly more informative than a blanket ok:false.

That is a security-posture call, not a mechanical fix, so I have not made it unilaterally. Two viable shapes:

• legacyPreimage count — exclude pre-v5 rows from the verdict, report the count and the schema-version boundary publicly so the reduction is stated rather than hidden.
• Freeze the boundary at a known-good seq — everything at or after the first v5 row is verdict-bearing; everything before is reported as legacy. Narrower, and it does not silently extend to a future schema bump.

I would take the second: it does not hand a future v6 the same exemption by default, which is how this class of thing quietly becomes permanent.

[JSONbored]

Resolved by #9857 and #9863 (both merged).

Both of the "What needs deciding" questions have answers now, and neither is the one this issue guessed:

Is the record preimage version-stable? Yes — DECISION_RECORD_SCHEMA_VERSION is part of the record and the digest round-trips exactly. #9857 diagnosed the live database directly and found the record self-consistent (record's own digest == recomputed); what diverged was the chain row, which had committed a digest for content the record no longer held. That is #9123's record-overwriting UPDATE, and the write path was already fixed by the :rev<N> revision scheme. The damage is historical and bounded: 83 rows in seq 5–257, with 1,061 subsequent rows clean. Canonicalization drift was not the cause.

Does /verify need to say so? It does now, and that is what #9863 built — the exact carve-out this issue asked for, mirroring how prunedRecords already handles legitimately-unre-checkable rows. LOOPOVER_LEDGER_CONTENT_WAIVER (<fromSeq>-<toSeq>:<reason>) declares a bounded exclusion from the content re-check only, counted separately as waivedContentMismatches so an excused row never renders as a clean one.

#9857 also fixed the amplifying half of the bug: verifyDecisionLedger returned at the first content mismatch, so one permanently-damaged row at seq 5 meant rows 6–1654 were never examined at all.

Verified against origin/main: parseLedgerContentWaiver and the LedgerContentWaiver type are in src/review/decision-record.ts, with the four disclosure properties (seq-bounded, both ends required, mandatory reason, content-only) enforced in code and pinned by tests, and the malformed-value case surfaced by src/selfhost/preflight.ts rather than left silently inert.

Closing as completed.