ccp: let a duplicate execution suppress the fill, not the whole report (ibx#330) - #331
ccp: let a duplicate execution suppress the fill, not the whole report (ibx#330)#331userFRM wants to merge 3 commits into
Conversation
The fill block recorded the ExecID and then booked the fill only if the order was already in `context`. There was no other branch, so a fill for an order the session does not track was dropped in full: no `Fill`, no execution detail, no position update, and nothing in the log to say a fill had been seen at all. The ExecID had already been consumed by then, which made the loss permanent. The replay the gateway pushes after a reconnect — the thing that would have recovered it — is rejected as a duplicate. An order is missing from `context` in ordinary ways. It reached a terminal status and was removed, which is the cancel/fill race: the cancel ack is processed first and the fill arrives to find nothing. Or it was placed from another client, or in an earlier session; the recovery insert does not pick those up, since it only fires for a New/New report carrying a contract id and a quantity. What is left is a position the account holds and the engine does not, disagreeing with the gateway's own position feed, with no error raised either way. The execution report already carries what booking needs, so the fill is now placed from the report when the order is unknown. Both the contract and the side must be on it: a guessed side moves the position the wrong way, which is worse than reporting that the fill could not be placed, so a report without tag 54 is refused rather than defaulted. Registration goes through the fallible path, because a full instrument table is a condition to report and not one to abort an inbound message on. A replayed execution is excluded. On a fresh process the gateway resends prior executions carrying `97=Y` and their original ExecIDs, for orders that no session tracks — booking those would build a position out of history on top of the one the position feed already reports. Within a process the ExecID window covers the reconnect burst, so the marker is only consulted on the untracked path. The ExecID is recorded once the fill can be booked rather than before. An execution that could not be placed stays replayable. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Strengthened the test to assert the status notification, not just the removal. The property this change turns on is that a duplicate still lets the terminal status reach the application. Treating the duplicate as though it had booked a fill would swallow that, because the status notification is suppressed whenever a fill was reported instead — and the original test could not tell the difference, since it only checked that the order had been removed. It now pins exactly one The wider "and nothing else" surface — the rich-order cache write, terminal removal for |
a259793 to
8fa8ea4
Compare
|
Closed four mutations the tests were letting through, one of which was the whole point of the change. The position was never asserted. Making the duplicate arm move the position again passed the entire suite — that is precisely the harm deduplication exists to prevent, and nothing caught it. The test now pins the position, the filled quantity on both the status update and the completed-order record, and the event channel, which every other test here passes The empty-ExecID path had no coverage anywhere in the repo. Dropping the A late duplicate could put a finished order back on the open list. The enrichment block had no guard against a stale status, so a duplicate of an earlier partial arriving after the terminal rewrote the cache to Filed #344 for the one finding that is not this change's to fix: the ExecID window is 1024 entries, so a replay batch deeper than that double-books its head. The same probe behaves identically before this commit, so it is the bound rather than anything here — but #317's double delivery makes it easier to reach than it looks. |
…ast print (ibx#315) `order_status` was given the size and price of the print that triggered it where the callback's contract asks for the order as a whole. Argument 3, `filled`, received this print's quantity; argument 5, `avgFillPrice`, received this print's price. Argument 8 is `lastFillPrice` and received the same price, which is the one place it belonged. IB's contract is that `filled` is cumulative and `avgFillPrice` is volume-weighted across every print, with `lastFillPrice` describing the print. A 300-share order filling in three 100-share prints therefore reported `filled=100` three times, at three different prices each presented as the average. Average-cost and P&L calculations reading that field are wrong for any order that fills in more than one print, and `filled` never reaches the order quantity, so completion detection on it never fires. The gateway sends both numbers on every execution report — CumQty (14) and AvgPx (6). They were already parsed, but only into the execution record, never into the status callback. They now travel on the fill itself. Reading them from the order cache at dispatch time would have been smaller, but a second print arriving between the push and the drain would report its cumulative against the earlier fill; carrying them with the event keeps each one self-consistent. A report that omits either field falls back to the print, which is what the callback reported before. Every consumer of the drained fill is corrected, not only the one the report named. The Python compat client dispatches the same fill through its own `order_status` and had the identical pair; the open-order snapshot both clients poll stored the print as the order's filled quantity, so a partial fill would have had the push callback and `reqOpenOrders` disagreeing about the same order. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…t (ibx#330) Deduplication returned out of the entire handler. Everything after the fill block went with it: the order and contract cache update, and the terminal block that pushes the completed order and removes it from `open_orders`. So a replayed execution carrying a terminal status never retired its order. It stayed open indefinitely — `req_open_orders` kept returning an order that was filled, and no completed-order record was ever produced for it. That is reachable in ordinary operation rather than in theory: the gateway replays recent executions at session start, and each of those messages currently arrives twice within the same session, so the second copy of a final fill lands here with the order still open. Stale open-order state is exactly what cancel-all loops and re-entry guards read. A duplicate now skips only the booking — the position update, the `Fill` event, the execution record — and the rest of the report is processed as it would be for any other frame. The status has already been applied by the time the fill block is reached, so the state machine is the part that must not be short-circuited. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
8fa8ea4 to
c5c4600
Compare
The gateway marks a report that restates history — 97=Y is PossResend, 43=Y is PossDupFlag — and neither tag was read anywhere in the handler. The only thing standing between a replayed execution and a second booking was the ExecID window, which a fresh process does not have: it has never seen the ID. At session start the gateway replays recent executions, so a restart holding open partially-filled orders emitted a fill, and moved a position, for something that happened before it started. A marker demotes trust rather than vetoing. Suppressing every marked report would lose a real fill: a CCP reconnect keeps this state, and the gateway replays recent executions on the new session — so a fill that executed during the outage arrives marked, with an ExecID this session has never seen, and is the first news of it. What a marked report is adjudicated on is whether it carries more cumulative quantity than the order already has. Restating quantity the order holds is history; carrying more is news. That needs a baseline, so the recovery record seeds `filled` from the cumulative quantity it states rather than assuming zero — otherwise a fresh process believes nothing has filled and every replayed execution behind that record looks like new quantity. The dedup window was also skipped outright when tag 17 was absent, which is the shape a replay takes and precisely when it matters. Such an execution is keyed on the fields that identify it, including LeavesQty: that strictly decreases across an order's successive partials, so two genuine slices of the same size at the same price inside one timestamp tick — the ordinary shape of algo and iceberg execution — are told apart, while a replayed copy restates the same value and still dedups. Neither change touches the early return on a duplicate, which leaves a terminal report's bookkeeping undone; that is deepentropy#331. Closes deepentropy#320. Closes deepentropy#260.
The gateway marks a report that restates history — 97=Y is PossResend, 43=Y is PossDupFlag — and neither tag was read anywhere in the handler. The only thing standing between a replayed execution and a second booking was the ExecID window, which a fresh process does not have: it has never seen the ID. At session start the gateway replays recent executions, so a restart holding open partially-filled orders emitted a fill, and moved a position, for something that happened before it started. A marker demotes trust rather than vetoing. Suppressing every marked report would lose a real fill: a CCP reconnect keeps this state, and the gateway replays recent executions on the new session — so a fill that executed during the outage arrives marked, with an ExecID this session has never seen, and is the first news of it. What a marked report is adjudicated on is whether it carries more cumulative quantity than the order already has. Restating quantity the order holds is history; carrying more is news, and the execution's own last quantity is the increment — every increment is one whole execution, so a report whose cumulative exceeds the running total is provably not yet counted. A gap between the two means the replay omitted an execution its own cumulative figure counts, which no per-report rule can repair without attributing those shares to the wrong fill at the wrong price, so it is logged rather than reconciled silently. Adjudicating needs a baseline, so the recovery record seeds `filled` from the cumulative quantity it states rather than assuming zero — otherwise a fresh process believes nothing has filled and every replayed execution behind that record looks like new quantity. The seed is clamped to the order's own size, because a float-to-int cast saturates and an absurd value would otherwise overflow the running total on the next live fill. The dedup window was also skipped outright when tag 17 was absent, which is the shape a replay takes and precisely when it matters. Such an execution is keyed on the fields that identify it, including LeavesQty: that strictly decreases across an order's successive partials, so two genuine slices of the same size at the same price inside one timestamp tick — the ordinary shape of algo and iceberg execution — are told apart, while a replayed copy restates the same value and still dedups. Neither change touches the early return on a duplicate, which leaves a terminal report's bookkeeping undone; that is deepentropy#331. Closes deepentropy#320. Closes deepentropy#260.
…suppresses the fill, not the report
Summary
returnfrom the entire handler, so everything after the fill block went with it — the order and contract cache update, and the terminal block that pushes the completed order and callscontext.remove_order.open_ordersindefinitely:req_open_orderskept returning an order that was filled, and no completed-order record was produced.Closes #330.
Reachable in ordinary operation
The gateway replays recent executions at session start, and #317 shows each of those messages is currently delivered twice within the same session — so the second copy of a final fill lands here with the order still open.
Stale open-order state is what cancel-all loops and re-entry guards read. An order that shows as working forever is the input that makes those act.
Change
A duplicate skips only the booking — the position update, the
Fillevent, the execution record. The rest of the report is processed as it would be for any other frame. The status has already been applied by the time the fill block is reached, so the state machine is the part that must not be short-circuited.Test
A partial fill is booked, then the same ExecID is replayed carrying the terminal status. The fill is not counted twice, and the order still reaches its terminal state and is reported completed. Restoring the early
returnfails it.Test plan
cargo test --offline --lib— 817 passed. The 2 failures areconfig::expiry_tests::{named_zone_converts_with_dst, instant_round_trips_to_wire}, which fail on the base commit too: the host has no legacy timezone files (fixed separately in config: resolve the legacy timezone names IB states its times in (ibx#335) #336).cargo check --offline --features python— clean.🤖 Generated with Claude Code