Live transport: crossbook connects to venues, and verifies against them - #1
Merged
Conversation
Lands the transport the README listed as in progress: crossbook now connects to a venue itself rather than only reconstructing books from events it is handed. Everything difficult is pure and testable without a network - RFC 6455 framing, the handshake, URL parsing, the capture format - and only the socket and TLS are platform code, quarantined in a separate optional target. The header-only library keeps its zero dependencies; consuming crossbook does not pull any of this in. - ws_frame.hpp: RFC 6455 codec. Rejects reserved bits, non-minimal lengths, fragmented and oversized control frames, and masked server frames. Failures latch: there is no resynchronisation point in a length-prefixed stream. - handshake.hpp: SHA-1 and base64 from scratch so Sec-WebSocket-Accept is verified rather than assumed. Checked against the RFC 6455 and FIPS 180-4 vectors. - transport: Schannel on Windows (ships with the OS, so the tools build on a stock machine), OpenSSL elsewhere with explicit hostname verification. - capture.hpp: length-prefixed capture format, so a live measurement can be replayed byte-identically by someone else. - crossbook_capture: connects to Kraken or Binance with no API key. 52 new test cases and a fuzz target for the frame reader, which is the one parser here exposed to unframed bytes off a socket.
Found by pointing crossbook_verify at Kraken rather than by a test. The first live run reported 98.66% - 4 of 298 updates mismatched - and the book held 20 bid levels for a subscription that asked for 10. The depth-limited contract has a gap that is easy to miss. Kraken reports cancellations, so a reader that handles those looks correct. It never reports that a level fell out of the top ten because a BETTER level arrived: from the venue's side there is nothing to say. Those orphaned levels sit below the checksummed depth doing no harm, until enough removals near the touch promote one back into view - and then the checksum fails, on an update that was itself perfectly fine, minutes after the actual divergence. - BasicL2Book::trim keeps the levels nearest the touch and drops the rest. Allocation-free: removals batch through a stack buffer and the loop repeats if one pass cannot name them all. - Feed takes a depth and trims BEFORE verifying, since the stale level the checksum is about to catch is exactly the one trimming removes. - FeedStats::levels_trimmed, so a depth that was never configured is visible rather than silent. Verified on both book implementations, including prices spread wide enough to push levels into the array book's overflow map.
Makes the README's central claim executable and checkable by someone who is not me. crossbook_verify connects to Kraken with no API key, rebuilds the book, recomputes the exchange's CRC32 over local state on every update, and reports the match rate with every divergence enumerated. Exit status is non-zero on any mismatch, decode failure, or resync - a verifier that reports problems and then exits successfully is one nobody will wire into anything. Measured over three minutes on BTC/USD: 2759 of 2759 checksums matched. The part that matters more than the number: tests/fixtures/kraken_btcusd_l2.cbcap is 72 KB of verbatim Kraken bytes, committed, and replays to 301 of 301 matches and one fixed state hash on every platform. CI runs it offline on Linux, macOS and Windows, so the number is a regression test rather than an anecdote. This is only possible because crypto market data can be redistributed; the equities equivalent cannot, which is why public ITCH projects ship without runnable data. Also here: - Instrument scales and subscription depth are read off the venue's own spelling in the snapshot, so a capture is self-describing and a replay needs no arguments beyond the file. A wrong scale fails every checksum, and a hard-coded table rots silently when a venue changes precision. - Split handshake and read timeouts. One value cannot serve both: an opening handshake through a busy edge takes tens of seconds - measured against Kraken, sometimes over twenty, and curl agrees - while a steady-state read wants about a second so the poll loop stays responsive. The single-timeout version failed to connect during exactly the episodes it needed to ride out. - Connect retry with exponential backoff, because reconnecting in a tight loop is what earns a throttle in the first place. - The live CI job is manual-only. A green build must never depend on an exchange being reachable.
Three POSIX-only failures, none of which MSVC can see. - socklen_t is unsigned on POSIX and a signed int on Winsock, so every setsockopt and connect length was a sign conversion under -Werror. Named the difference as a `SockLen` alias rather than sprinkling casts that happen to be right on one platform. - GCC 13 rejected `clear()` then `resize()` on the send buffer under -Wstringop-overflow, unable to prove the frame header had room. The header now goes into a fixed std::array whose bound is in the type. Better code for the same reason the warning fired. - Two includes MSVC supplies transitively and libstdc++ does not.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes the one item the README listed as not built: crossbook now opens the socket itself, and the correctness claim it makes is executable rather than descriptive.
What this adds
Transport — RFC 6455 framing, the opening handshake, URL parsing and the capture format are all pure functions of bytes, unit tested and fuzzed without a network. Only the socket and TLS are platform code, quarantined in an optional
crossbook_nettarget. The library stays header-only with zero dependencies;-DCROSSBOOK_BUILD_TOOLS=OFFdrops the transport entirely and consumingcrossbook::crossbooknever pulls in a TLS stack.TLS uses Schannel on Windows (ships with the OS, so the tools build on a stock machine with nothing installed) and OpenSSL elsewhere, with explicit hostname verification.
crossbook_verify— connects to Kraken with no API key, rebuilds the book, recomputes the exchange's CRC32 on every update, and reports the match rate with every divergence enumerated. Non-zero exit on any mismatch, decode failure or resync.crossbook_capture— records raw frames verbatim with arrival timestamps. It deliberately does not decode: recording and interpreting are separate jobs, which is what makes a capture evidence rather than output.The result
Three minutes live on BTC/USD: 2759 of 2759 checksums matched.
More useful than the number —
tests/fixtures/kraken_btcusd_l2.cbcapis 72 KB of verbatim Kraken bytes, committed, replaying to 301 of 301 matches and one fixed state hash. CI runs it offline on Linux, macOS and Windows, so the README figure is a regression test rather than an anecdote.Live verification found a real bug
The first live run reported 98.66% — 4 of 298 mismatched — with 20 bid levels held for a depth-10 subscription.
Kraken reports cancellations, so a reader handling those looks correct. It never reports that a level fell out of the top ten because a better one arrived; from the venue's side there is nothing to say. Those orphaned levels sit below the checksummed depth until removals near the touch promote one back into view, and then the checksum fails on an update that was itself fine, minutes after the real divergence.
Fixed by
BasicL2Book::trim, with a negative-control test asserting the same capture still fails when trimming is disabled — so the passing test cannot start passing for the wrong reason.No amount of testing the book against itself would have surfaced this.
Two design notes worth flagging in review
curlagreed — while a steady-state read wants about a second so the poll loop stays responsive. The single-timeout version failed to connect during exactly the episodes it needed to ride out.Overlap with the decode layer
Rebased on
50146de.book.hppandfeed.hppare touched by the trimming fix — that is the one place this branch reaches into the decode layer, and it is the bug above.Verification
/W4 /WXclean on MSVCfuzz_ws_frametarget, in CI at 90 s — the frame reader is the only parser here fed bytes nothing we control has framed