Skip to content

v0.22.0 — the multi-symbol release

Choose a tag to compare

@intrepidkarthi intrepidkarthi released this 10 Aug 15:24
· 80 commits to main since this release

The multi-symbol release, and the third in a row where writing the spec first found
a defect that predated the feature. PRODUCTION-READINESS.md called a multi-symbol
venue "a routing layer you write." Checking the code instead of the sentence:
ShardsConfig had no way to supply a CommandLog, and durability, recovery and
replication all hang off it — so a sharded venue could not survive a restart at
all.
That is not a routing layer, it is most of a venue.

The design decision everything else follows from is a refusal: there is no order of
events across symbols, because a venue-wide sequence needs a serialisation point
every command passes through, which is the bottleneck sharding exists to remove.
Each symbol is its own timeline; what that costs is listed in the spec rather than
discovered later. Ids are the one thing shared across books, and they are
partitioned rather than centralised precisely so that sharing costs no
coordination — a shared counter would have made a shard's ids depend on how its
traffic interleaved with every other shard's, so replaying one log alone would no
longer reproduce its own ids.

cmd/obgw still serves one instrument, and is now the only thing between this
design and a multi-symbol venue anyone can run.

Added

  • Multi-symbol identity (MULTI-SYMBOL.md, deliverables
    all six). Order and trade ids partition into a 15-bit shard index and a 48-bit
    per-shard counter, so an int64 names an order at a venue with many books. A
    shared counter would have been simpler and wrong: it makes a shard's ids depend on
    how its traffic interleaved with every other shard's, so replaying one log alone
    would no longer reproduce its own ids — trading the property this project is most
    confident about for one it merely wants. Shard 0 composes to the sequence itself,
    so every single-symbol deployment keeps the ids, snapshots, logs and golden vectors
    it already had.

    matching.Manifest is the price: a durable, CRC-checked symbol -> index mapping,
    refused rather than repaired. Losing it is worse than losing a snapshot, which
    costs only a replay.

  • ShardsConfig.NewLog — the finding that turned a one-line gap into a spec.
    ShardsConfig had no way to supply a CommandLog, and durability, recovery and
    replication all hang off it, so a sharded venue could not survive a restart at
    all.
    PRODUCTION-READINESS.md called multi-symbol "a routing layer you write";
    it was most of a venue. One log per shard, so recovery and the replication drills
    are the existing single-symbol code paths run N times.

  • Venue-wide ClOrdID admission (Registry.IsLiveClOrdID). The naming index is
    keyed by account and client id with no symbol, so at two instruments a repeat
    overwrites the first and the account's next cancel retargets.

  • examples/multisymbol — a two-book reference venue with a feed and a log per
    shard, serving both books over sockets. It exists for the reason
    examples/replication does: the multi-symbol seams are each plausible on the page,
    and this repository's record with seams claimed but never consumed is documented
    and bad. Replication drill D8 covers a two-symbol venue, with a negative
    control — a follower on the wrong shard index rebuilds the same orders under
    different numbers, and the digest catches it.

    cmd/obgw is deliberately not converted: it still serves one instrument, and
    it is now the only thing between this design and a multi-symbol venue anyone can
    run. Converting it is one runner, one feed, one gate, one recovery path, sixteen
    call sites and fifteen test files — its own arc, not a rider on a protocol change.

Changed

  • BREAKING: wire protocol v3 → v4. MDSubscribe gains Symbol. It named an
    incarnation and a sequence but no instrument, so a market-data connection could
    only ever mean "the one book this venue serves". A subscription now selects
    exactly one symbol and every message on that connection belongs to it, so no
    other market-data payload changed
    — the regenerated golden vectors differ only
    in their version byte. A subscription for an unserved instrument is refused with
    MDRejectUnknownSymbol rather than quietly given the wrong book, which a
    subscriber cannot detect for itself.

  • BREAKING: Shards refuses a second symbol without a Manifest. It previously
    gave every shard index 0 and served colliding ids silently — a failure whose only
    symptom is two orders nobody can tell apart, discovered much later. Anyone running
    Shards multi-symbol today was doing this unknowingly.

  • Engine.Bust validates the shard field, so busting another symbol's trade is
    ErrUnknownTrade instead of annulling whichever local print shared the low bits.

Fixed

  • Replication drill D6 blamed the wrong follower, about one run in twelve. The
    drill drove traffic until Shed() != 0 and assumed the follower cut was the
    wedged one. A follower that actually applies commands is slower than a wedged
    socket — which merely fills a kernel buffer and costs the primary nothing until it
    is full — so driven flat out, the healthy follower's own ship buffer overflowed
    first and it was the one shed. The drill then reported "shedding the wedge broke
    the healthy follower", the opposite of what had happened.

    Primary.ShedPeers now attributes each cut to a peer address, which is the part
    worth having beyond the test: a bare drop counter cannot tell an operator whether
    a client stopped reading or a follower is merely running behind, and those need
    opposite responses. D6 waits for the wedge specifically, asserts no other follower
    was cut, and paces the tape against the healthy follower so there is one candidate
    rather than two. 0 failures in 40 runs, against roughly 8% before; still fails
    against a fanout that blocks instead of shedding.


Upgrading: this release contains two breaking changes. Any client built against wire v3 must be rebuilt (MDSubscribe gains Symbol), and anyone driving matching.Shards with more than one symbol must now supply a Manifest — previously every shard silently used index 0 and minted colliding ids.

Full changelog: v0.21.0...v0.22.0