v0.22.0 — the multi-symbol release
The multi-symbol release, and the third in a row where writing the spec first found
a defect that predated the feature. PRODUCTION-READINESS.md called a multi-symbol
venue "a routing layer you write." Checking the code instead of the sentence:
ShardsConfig had no way to supply a CommandLog, and durability, recovery and
replication all hang off it — so a sharded venue could not survive a restart at
all. That is not a routing layer, it is most of a venue.
The design decision everything else follows from is a refusal: there is no order of
events across symbols, because a venue-wide sequence needs a serialisation point
every command passes through, which is the bottleneck sharding exists to remove.
Each symbol is its own timeline; what that costs is listed in the spec rather than
discovered later. Ids are the one thing shared across books, and they are
partitioned rather than centralised precisely so that sharing costs no
coordination — a shared counter would have made a shard's ids depend on how its
traffic interleaved with every other shard's, so replaying one log alone would no
longer reproduce its own ids.
cmd/obgw still serves one instrument, and is now the only thing between this
design and a multi-symbol venue anyone can run.
Added
-
Multi-symbol identity (MULTI-SYMBOL.md, deliverables
all six). Order and trade ids partition into a 15-bit shard index and a 48-bit
per-shard counter, so anint64names an order at a venue with many books. A
shared counter would have been simpler and wrong: it makes a shard's ids depend on
how its traffic interleaved with every other shard's, so replaying one log alone
would no longer reproduce its own ids — trading the property this project is most
confident about for one it merely wants. Shard 0 composes to the sequence itself,
so every single-symbol deployment keeps the ids, snapshots, logs and golden vectors
it already had.matching.Manifestis the price: a durable, CRC-checkedsymbol -> indexmapping,
refused rather than repaired. Losing it is worse than losing a snapshot, which
costs only a replay. -
ShardsConfig.NewLog— the finding that turned a one-line gap into a spec.
ShardsConfighad no way to supply aCommandLog, and durability, recovery and
replication all hang off it, so a sharded venue could not survive a restart at
all.PRODUCTION-READINESS.mdcalled multi-symbol "a routing layer you write";
it was most of a venue. One log per shard, so recovery and the replication drills
are the existing single-symbol code paths run N times. -
Venue-wide
ClOrdIDadmission (Registry.IsLiveClOrdID). The naming index is
keyed by account and client id with no symbol, so at two instruments a repeat
overwrites the first and the account's next cancel retargets. -
examples/multisymbol— a two-book reference venue with a feed and a log per
shard, serving both books over sockets. It exists for the reason
examples/replicationdoes: the multi-symbol seams are each plausible on the page,
and this repository's record with seams claimed but never consumed is documented
and bad. Replication drill D8 covers a two-symbol venue, with a negative
control — a follower on the wrong shard index rebuilds the same orders under
different numbers, and the digest catches it.cmd/obgwis deliberately not converted: it still serves one instrument, and
it is now the only thing between this design and a multi-symbol venue anyone can
run. Converting it is one runner, one feed, one gate, one recovery path, sixteen
call sites and fifteen test files — its own arc, not a rider on a protocol change.
Changed
-
BREAKING: wire protocol v3 → v4.
MDSubscribegainsSymbol. It named an
incarnation and a sequence but no instrument, so a market-data connection could
only ever mean "the one book this venue serves". A subscription now selects
exactly one symbol and every message on that connection belongs to it, so no
other market-data payload changed — the regenerated golden vectors differ only
in their version byte. A subscription for an unserved instrument is refused with
MDRejectUnknownSymbolrather than quietly given the wrong book, which a
subscriber cannot detect for itself. -
BREAKING:
Shardsrefuses a second symbol without aManifest. It previously
gave every shard index 0 and served colliding ids silently — a failure whose only
symptom is two orders nobody can tell apart, discovered much later. Anyone running
Shardsmulti-symbol today was doing this unknowingly. -
Engine.Bustvalidates the shard field, so busting another symbol's trade is
ErrUnknownTradeinstead of annulling whichever local print shared the low bits.
Fixed
-
Replication drill D6 blamed the wrong follower, about one run in twelve. The
drill drove traffic untilShed() != 0and assumed the follower cut was the
wedged one. A follower that actually applies commands is slower than a wedged
socket — which merely fills a kernel buffer and costs the primary nothing until it
is full — so driven flat out, the healthy follower's own ship buffer overflowed
first and it was the one shed. The drill then reported "shedding the wedge broke
the healthy follower", the opposite of what had happened.Primary.ShedPeersnow attributes each cut to a peer address, which is the part
worth having beyond the test: a bare drop counter cannot tell an operator whether
a client stopped reading or a follower is merely running behind, and those need
opposite responses. D6 waits for the wedge specifically, asserts no other follower
was cut, and paces the tape against the healthy follower so there is one candidate
rather than two. 0 failures in 40 runs, against roughly 8% before; still fails
against a fanout that blocks instead of shedding.
Upgrading: this release contains two breaking changes. Any client built against wire v3 must be rebuilt (MDSubscribe gains Symbol), and anyone driving matching.Shards with more than one symbol must now supply a Manifest — previously every shard silently used index 0 and minted colliding ids.
Full changelog: v0.21.0...v0.22.0