Skip to content

gateway: scan the inflated init content without putting it in front of the engine (ibx#317) - #370

Open
userFRM wants to merge 1 commit into
deepentropy:mainfrom
userFRM:fix/init-burst-double-seed
Open

gateway: scan the inflated init content without putting it in front of the engine (ibx#317)#370
userFRM wants to merge 1 commit into
deepentropy:mainfrom
userFRM:fix/init-burst-double-seed

Conversation

@userFRM

@userFRM userFRM commented Jul 30, 2026

Copy link
Copy Markdown

Problem

The logon ACK arrives DEFLATE-compressed inside 8=FIXCOMP envelopes, and the routing tags the connect path needs are in the plaintext. Gateway::connect decompressed every segment and appended the result to init_data so the tag scan below could see it.

init_data is also the buffer handed to the engine — and the engine is given the burst still compressed, so it decompresses the same segments itself.

The hot loop therefore read each message in the init burst twice from a single delivery: once out of the segment it inflated, once from the plaintext copy appended behind it. Every account record, resting order and bulletin in the burst was handled twice. The duplicate was locally manufactured, not something the gateway sent.

What this changes

The inflation is taken into a copy. init_scan_buffer takes the burst by reference and returns the buffer to scan, so the buffer the engine is handed cannot be the one that grows — reintroducing this takes changing the signature, not adding a line.

Nothing else changes: the tag scan sees exactly what it saw, and the engine is handed exactly what arrived.

This is deliberately not a deduplication layer. Deduplicating in the hot loop would suppress messages that are legitimately byte-identical — a 35=G binary bar carries no sequence number or sending time, so an unchanged bar resent deliberately is indistinguishable from a repeat — and it would leave the duplicate being generated.

Tests

the_inflated_init_content_is_scanned_but_not_handed_to_the_engine — the scan buffer gains exactly one inflated copy of a segment's content while the buffer the engine receives is unchanged. It fails by name when the inflation is dropped, and when the scan buffer stops carrying what arrived.

Closes #317.

Test plan

  • Mutation: dropping the inflation fails the_inflated_init_content_is_scanned_but_not_handed_to_the_engine by name.
  • Mutation: having the scan buffer stop carrying what arrived fails the same test.
  • The &[u8] signature makes the original defect a compile error to reintroduce, not a review item: taking the burst by reference means the engine's copy cannot be the one that grows.
  • cargo check --offline clean on --lib, --lib --features python, --bins, --examples, and each integration target individually.
  • tests/ib_paper_compat compared against a clean checkout of the base commit — identical sorted diagnostic sets.
  • cargo test --offline --lib — only the two known config::expiry_tests failures, which fail on the base commit for missing legacy tzdata (fixed separately in config: resolve the legacy timezone names IB states its times in (ibx#335) #336).

…f the engine

The logon ACK arrives DEFLATE-compressed inside `8=FIXCOMP` envelopes, and the routing tags the connect path needs are in the plaintext. It decompressed every segment and appended the result to `init_data` so the tag scan could see it — but `init_data` is also the buffer handed to the engine, which is given the burst still compressed and decompresses the same segments itself.

So the hot loop read each message in the init burst twice from a single delivery: once out of the segment it inflated, once from the plaintext copy appended behind it. Every account record, resting order and bulletin in the burst was handled twice.

The inflation is now taken into a copy. `init_scan_buffer` takes the burst by reference and returns the buffer to scan, so the engine's copy cannot be the one that grows — reintroducing this would take changing the signature, not adding a line.

Nothing else changes: the tag scan sees exactly what it saw, the engine is handed exactly what arrived.

Closes deepentropy#317.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ccp: the session-start execution replay is delivered twice and only fills are deduplicated, so every other callback fires twice

1 participant