gateway: scan the inflated init content without putting it in front of the engine (ibx#317) - #370
Open
userFRM wants to merge 1 commit into
Open
gateway: scan the inflated init content without putting it in front of the engine (ibx#317)#370userFRM wants to merge 1 commit into
userFRM wants to merge 1 commit into
Conversation
…f the engine The logon ACK arrives DEFLATE-compressed inside `8=FIXCOMP` envelopes, and the routing tags the connect path needs are in the plaintext. It decompressed every segment and appended the result to `init_data` so the tag scan could see it — but `init_data` is also the buffer handed to the engine, which is given the burst still compressed and decompresses the same segments itself. So the hot loop read each message in the init burst twice from a single delivery: once out of the segment it inflated, once from the plaintext copy appended behind it. Every account record, resting order and bulletin in the burst was handled twice. The inflation is now taken into a copy. `init_scan_buffer` takes the burst by reference and returns the buffer to scan, so the engine's copy cannot be the one that grows — reintroducing this would take changing the signature, not adding a line. Nothing else changes: the tag scan sees exactly what it saw, the engine is handed exactly what arrived. Closes deepentropy#317.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The logon ACK arrives DEFLATE-compressed inside
8=FIXCOMPenvelopes, and the routing tags the connect path needs are in the plaintext.Gateway::connectdecompressed every segment and appended the result toinit_dataso the tag scan below could see it.init_datais also the buffer handed to the engine — and the engine is given the burst still compressed, so it decompresses the same segments itself.The hot loop therefore read each message in the init burst twice from a single delivery: once out of the segment it inflated, once from the plaintext copy appended behind it. Every account record, resting order and bulletin in the burst was handled twice. The duplicate was locally manufactured, not something the gateway sent.
What this changes
The inflation is taken into a copy.
init_scan_buffertakes the burst by reference and returns the buffer to scan, so the buffer the engine is handed cannot be the one that grows — reintroducing this takes changing the signature, not adding a line.Nothing else changes: the tag scan sees exactly what it saw, and the engine is handed exactly what arrived.
This is deliberately not a deduplication layer. Deduplicating in the hot loop would suppress messages that are legitimately byte-identical — a
35=Gbinary bar carries no sequence number or sending time, so an unchanged bar resent deliberately is indistinguishable from a repeat — and it would leave the duplicate being generated.Tests
the_inflated_init_content_is_scanned_but_not_handed_to_the_engine— the scan buffer gains exactly one inflated copy of a segment's content while the buffer the engine receives is unchanged. It fails by name when the inflation is dropped, and when the scan buffer stops carrying what arrived.Closes #317.
Test plan
the_inflated_init_content_is_scanned_but_not_handed_to_the_engineby name.&[u8]signature makes the original defect a compile error to reintroduce, not a review item: taking the burst by reference means the engine's copy cannot be the one that grows.cargo check --offlineclean on--lib,--lib --features python,--bins,--examples, and each integration target individually.tests/ib_paper_compatcompared against a clean checkout of the base commit — identical sorted diagnostic sets.cargo test --offline --lib— only the two knownconfig::expiry_testsfailures, which fail on the base commit for missing legacy tzdata (fixed separately in config: resolve the legacy timezone names IB states its times in (ibx#335) #336).