v0.26.0 — a second implementation, and a log that can forget

Forty-nine commits since v0.25.0, and the two arcs that dominate them are a second
implementation of the matching rules that disagrees with the engine on purpose, and a
write-ahead log that can finally forget. Between them they found six correctness
defects — three of which no test, fuzzer or race detector in this repository had
caught — and turned restart cost from a property of how long the venue has been up
into a byte budget an operator sets.

The measurement that corrects a published claim is in here too: distinct symbols do
not scale linearly across cores. They scale 2.24×.

A reference matcher, and the three defects it found

internal/refmatch is a limit order book written to be read — two sorted slices
scanned linearly, cancel by linear search, depth by a fold. No index, no pool, no
id-to-node map. It is slow on purpose and must never be optimised, because a model
with an index has the engine's bug class. It imports the standard library and nothing
else, and a test parses the files to keep it that way. pkg/types is the tempting
exception and it is refused: an invariant both sides inherit from the same seven lines
is an invariant nothing is checking.

Both sides produce one comparable Observation per command, compared whole with
reflect.DeepEqual rather than field by field — because a field-list comparator is
where a future field silently stops being checked. Twenty-one deliberate engine
mutations are all caught, and every shrunk reproduction is 1 to 4 commands. One seed
catches 8 of 18; the sweep catches 18 of 18, which is the argument for the sweep's
size.

It found three real defects. Each had more than one defensible answer, so each was
pinned as a failing test first and decided in
DIFFERENTIAL-FINDINGS.md before being repaired:

• A rejected fill-or-kill moved LastTradePrice. match recorded the price from
  the final print, the fill-or-kill branch reversed every one of those prints, and
  nothing put the price back. The rule is now stated in recordLast's doc comment:
  LastTradePrice is the price of the last trade this venue published.
• Under ProRata, a taker meeting its own resting liquidity was skipped entirely.
  matchProRata never called takerSTP, so all five self-trade-prevention modes left
  the taker's remainder resting across the spread — bid 100 / ask 99 on a
  continuous book. A venue configured ALLOW did not get the self-trade it asked for
  and one configured CANCEL_BOTH cancelled neither order: pro-rata was silently
  overriding the venue's STP configuration, and declining unrelated accounts'
  liquidity on the way.
• A REJECTED command's event batch may now carry further events. A rejection
  drops only the events describing state the engine actually undid, so a REJECTED
  may be followed by CANCELED, REPLACED, ACCEPTED or TRIGGERED. Consumers of
  EventSink must apply them — see Breaking, below.

The harness's own guards were then sabotaged twelve ways, and the two that did not
behave as specified are written down rather than quietly fixed
(REFERENCE-MATCHER.md §10.3).

Three iceberg defects, one of which bricked checkpoints

• A refused iceberg made the venue's own snapshot unloadable. ProcessIceberg
  registered the iceberg before settling it and did not undo that when the settle
  refused, so TakeSnapshot wrote an entry for an order that is not on the book and
  LoadSnapshot refuses such a snapshot outright. One refused iceberg and the venue
  could never load a checkpoint again — not after further trading, not after a
  restart. It also landed on this release's own upgrade path: the runbook tells an
  operator to checkpoint after accepting a semantics mismatch, and measured end to
  end, that produced a venue that would not restart. The repair is the invariant, not
  the symptom — the registry holds icebergs whose displayed slice is resting.
• A recovery from the journal alone lost every iceberg's hidden reserve. A nine-lot
  iceberg shown three was journalled as quantity 3, display 3, and a replay rebuilt
  it with hidden = 0. A venue recovering from its log alone came back with every
  client's reserve gone — the path a venue takes when its snapshot is missing,
  refused as corrupt, or below the retention floor. Recovery with a snapshot was
  never affected.
• A failing fill-or-kill corrupted an iceberg it consumed. Negative FilledQty,
  the entire hidden reserve displayed in the open, a shown size larger than the order's
  own quantity, and its refill registration dropped. On the two-command reproduction, 9
  total / 3 displayed became FilledQty -6, RemainingQty 9, reserve 0, best ask
  100:9. An iceberg exists to hide size; one rejected order from an unrelated account
  leaked all of it, permanently.

Also fixed: admission measured an iceberg's displayed slice rather than its total,
so a client could switch off the fat-finger cap by choosing an order type — the audit
behind it found five admission checks measuring the wrong quantity
(ICEBERG-ADMISSION.md).

Restart cost becomes a number you choose

Two changes, in order. A restart no longer parses the part of the log its snapshot
already covers — every record is still read and CRC-verified, but only records past
the snapshot's sequence are decoded and retained. On a 500,000-record covered prefix
that is 1.66 s → 64 ms, and allocation goes flat in the prefix. The saving was
~26×, more than the design predicted, because the decode turned out to be ~97% of the
marginal cost of a covered record; the BENCHMARKS.md row that mispredicted it is
corrected.

Then the log rotates into segments, and a prefix of them can be deleted once a
verified snapshot covers it. Restart cost is now O(retained log), and the retained size
is a byte budget you set. The decision the slice turns on is that a segment declares
its base sequence in an 18-byte header rather than having one inferred from its
position — a renamed, copied or restored segment cannot quietly put records into the
wrong sequence space. Design, and every place the code disagreed with its own spec:
LOG-ROTATION.md §12.

That budget is not set by default, and a venue that leaves -wal-retain unset
still gets slower to restart every day it stays up. That is the honest state of it.

A journal that refuses rather than lying

Three of the changes above alter what matching does with the same input, so a log
recorded before them and recovered after them produces state that never existed on the
venue that wrote it. Until this release nothing on disk said so: recovery replayed the
log, started, and the book was wrong in a way nothing downstream flagged.

matching.SemanticsVersion stamps an equivalence class of builds — two builds
share a version if and only if, for every command sequence and configuration, they
produce the same trades, events, verdicts and book. It is 2 on this release.
A release version used as the stamp would refuse journals that replay identically on
every upgrade, and the response to a check that cries wolf is a permanent override.

The override names the versions it accepts rather than being a boolean, which is the
most important detail in it: -wal-semantics-mismatch-ok goes into a unit file during
one incident and stays for the life of the deployment.
-wal-accept-semantics 1 stops working the moment the number moves again.

What this deliberately does not cover is engine configuration: two builds at the
same semantics version with different ProRata, SelfTradePrevention, MaxOrders or
PriceBand replay the same log into different books and nothing notices. That gap is
arguably larger than the one this closes, and it is named in
SEMANTICS-VERSION.md §6 rather than implied to be handled.

The venue counts what it refuses

Sixteen metric families already said what the venue did; nothing said what it
dropped, and the durability path was untimed.

obgw_refused_total{reason} counts every refusal at the single funnel all fifty
refusal sites pass through, and increments before the encode, so it can never claim
a rejection the client did not get — measured equal end to end at 51,568 against 51,568
CmdRejects received. obgw_shed_unreported_total counts the shed with nobody left to
tell. obgw_wal_append_latency_ns and obgw_wal_sync_latency_ns are timed on opposite
sides of the group commit, and are 225× apart under -sync-every-command (17 µs against
3.9 ms) — a sync latency that is really an append latency reads healthy while fsync is
the slow thing. That also corrects the recovery point objective, which is
20 ms + p99 fsync and not the 20 ms ticker alone.

Cost: 82 ns and zero allocations per timing. Thirteen threshold rows in
RUNBOOKS.md, each with a normal value, a trouble value and an
action, because a metric nobody has a threshold for is a metric nobody looks at.

Adversarial review caught two of these before they shipped: a paging threshold of 1 s
standing against a histogram whose top bucket was 250 ms, so that tier could never fire
and an operator would have read healthy through an arbitrarily slow disk; and failed
logins invisible to a counter whose name implied it covered refusals.

Measured, and corrected

• Sharding scales 2.24×, not linearly. Two documents said distinct symbols "scale
  linearly across cores" and nobody had measured it. BenchmarkShards_Scaling, pinned
  to the four performance cores of an M4: 876 K ops/s at one book, 1.64 M at two
  (1.88×), 1.97 M at four (2.24×) — and then nothing, 2.00 M at six and 1.99 M at
  eight, inside the ±4% spread. Each shard is a pair of goroutines, so past the core
  count the machine goes into the handoff. Books beyond that buy queue headroom, not
  throughput. Venue capacity is not symbols × single-book throughput. Corrected in all
  three places that claimed it.
• "Snapshots bound restart time" was wrong in seven places. A snapshot bounds the
  replay and the parse, not the read. What bounds the read is retention.
• A four-hour soak across three books — 14,400,199 messages, goroutines and
  descriptors flat across 240 samples, no orphans (SOAK.md §1e).
• Twelve documents in docs/ were reachable from nothing in the README. A document
  nothing links to is a document nobody reads. All thirty-four are indexed now.

Breaking

Pre-1.0, so this is a minor bump; the changelog header states the policy.

• matching.CommandLog gained a sixteenth method. Any implementation of the
  interface outside this repository must add it.
• matching.SemanticsVersion is 1 → 2. See upgrading, below.
• EventSink consumers must apply the events that follow a REJECTED. A rejected
  command used to publish exactly one event. It no longer does.

Upgrading

A pre-stamp log declares nothing, and nothing is not "compatible" — it is exactly the
log that does not have the three matching changes above. So an unstamped or mismatched
segment is refused when its records would be replayed, and accepted when they would
not.

Checkpoint before you upgrade and there is no ceremony at all: a mismatched segment
the snapshot already covers is read, CRC-verified, skipped and reported, never refused.
obgw does not checkpoint on a clean shutdown, so the ordinary upgrade path meets the
refusal once — the two-line procedure is "Upgrading across a semantics change" in
RUNBOOKS.md. The override is -wal-accept-semantics 0
(wal.RecoverOptions.AcceptSemantics), and it relaxes the semantics gate and nothing
else: ErrCorrupt, ErrLogGap and the retention floor are untouched.

With thanks

OrderBook.Add refused a duplicate id when the book was full, and advanced the book
sequence for an add that changed nothing — a documented contract whose outcome depended
on unrelated capacity state. Reported and fixed by
@Taz33m in
#8 /
#9. internal/semcheck stayed
green across the merge, which is the enforcement declining to fire on a change it
should not — the false-positive half of the semantics design, exercised by an outside
contribution rather than by a sabotage run.

---

Still an experiment, and it has still never run a live market. What that costs you is
in PRODUCTION-READINESS.md.

Full changelog: CHANGELOG.md
· v0.25.0...v0.26.0