req_pnl: pnl() callback never fires for intraday-opened positions

[deepentropy]

Summary

After the v0.4.5 fix for #142, req_pnl() no longer leaks notional values into daily_pnl, but introduces a silent regression: the pnl() callback never fires for positions opened intraday.

Repro

Probe script: connect to paper account (flat at start), req_account_updates(True) + req_pnl(), place a 1-share marketable BUY, observe callbacks for 180s, place SELL to flatten.

Probe results (paper, US pre-market 2026-05-08, account/IDs redacted):

[ 11.633s] req_account_updates(True) + req_pnl(...)
[ 11.655s] update_account_value DailyPnL=0.00       (one push, never updates again)
[ 11.984s] pnl()  daily=+0.00  unrealized=+0.00  realized=+~$X
[ 41.967s] BUY 1 SPY @ 735.00  filled
            -- 180s observation window --
            -- ZERO pnl() callbacks fired during this window --
[222.199s] SELL @ 735.07  filled (flat)

Key data points:

• pnl() fires once at startup (computed from prior-session state via the midnight-seed reconstruction)
• During 180s post-fill on a fresh intraday position, pnl() never fires
• update_account_value("DailyPnL", ...) also pushes only once with stale 0.00, despite NetLiquidation updating multiple times

Diagnosis

src/client_core.rs:898-956 (poll_pnl) iterates only over midnight seeds:

let seeds = shared.portfolio.midnight_seeds();
if seeds.is_empty() { return None; }
for seed in &seeds { ... }

Positions opened intraday have no entry in the midnight-seed map, so they're excluded from the computation entirely. For a flat-at-midnight account, the loop produces stale totals that never change → cached last_pnl suppresses any callback.

Bug class progression

Version	Symptom for intraday-opened position
v0.4.4	daily_pnl accumulates trade notional per fill (the original report)
v0.4.5+	pnl() callback never fires at all

The v0.4.5 fix swapped a loud bug for a quiet one. Both stem from the same root cause: client-side reconstruction from midnight seeds cannot represent positions opened today.

Open architectural question

Whether the upstream client achieves this via:

1. A different subscription that streams server-computed daily P&L live, or
2. The same midnight-seed message but with intraday updates after fills

is unknown without protocol capture. A capture request has been filed at deepentropy/ib-agent.

A purely-local fix (extend poll_pnl to handle no-seed positions by synthesizing money_traded = qty_now × avg_cost, which collapses to unrealized P&L for fresh positions) is mathematically correct but patches the architecture rather than aligning with upstream behavior. Holding the PR pending capture data.

Probe script

scripts/.tmp/probe_pnl_142.py (not committed; uses .venv\Scripts\python.exe directly).

Related

• req_pnl() callback returns cumulative notional values instead of daily P&L #142 — original report (notional leak)
• deepentropy/ib-agent#146 — capture request

[deepentropy]

Architectural answer from deepentropy/ib-agent#146

The capture confirms the upstream client treats reqPnL as a server-stream passthrough, not a client-side reconstruction:

• subscribe → API-socket frame, payload (reqId, account, modelCode)
• stream → API-socket frame, payload (reqId, dailyPnL, unrealizedPnL, realizedPnL) — pushed event-driven by the gateway whenever the underlying P&L computation changes (quote-tick or fill)
• cancel → API-socket frame, payload (reqId)

Two stream frames were observed 1.008s apart with values changing between them, no client action in between. Quote-driven push by the gateway, no polling.

Caveats

• Capture was Saturday (US markets closed); per-fill cadence not characterized but not needed to settle the architecture question.
• The v0.4.5 fix remains correct as a stopgap for the original notional-leak symptom (req_pnl() callback returns cumulative notional values instead of daily P&L #142); replacement is preferable but not urgent.

---

Correction (edit): the "passthrough" framing below was wrong

I originally wrote that the fix would route via the API-socket subscribe/stream/cancel triplet "the same pattern used by other EClient methods." On closer review of the codebase that claim is false — ibx is FIX/CCP-only at runtime. Every EClient method (req_mkt_data, req_account_summary, req_positions, req_pnl, …) routes through the FIX-side ccp / farm / hmds connections in src/engine/hot_loop/. There is no API-socket connection in the engine. The capture's subscribe → stream → cancel triplet is upstream-client behavior on a port ibx does not currently speak.

The architectural conclusion in deepentropy/ib-agent#146 (gateway server-streams the live daily P&L) holds; only the implementation path needs to be reframed.

Three options, honestly

A. Add a TWS-API-socket layer to ibx. New protocol module (binary length-prefixed framing for the PNL message family), new connection in hot_loop alongside the FIX sockets, separate login handshake on the API-socket port. Hundreds of LoC. Architecturally clean — gives true wire-level iso-equivalence with the upstream client and sets a precedent if other request types ever migrate off the FIX side.

B. Find a CCP/FIX-side equivalent of the live P&L push. The gateway's internal P&L calculator consumes the midnight-seed cash-flow input and emits the live update on the API socket. Whether the CCP side has any equivalent push is unverified — the upstream client and ibx talk to different sides of the same calculator. Confirming or refuting would need another protocol capture; likely a dead end.

C. Pragmatic client-side fix in poll_pnl. Stay on FIX/CCP. Extend src/client_core.rs:898-956 to iterate over all current positions (not just seeds) and synthesize a default money_traded = qty_now × avg_cost when a position has no seed entry. The existing formula then collapses to unrealized P&L for fresh positions — correct, no notional leak. Closes this issue's silent-stale failure mode without an architecture change. Matches upstream pnl() callback semantics for the user-visible case but not the upstream wire behavior.

Reading of "iso with upstream API endpoints"

If "iso" is interpreted at the EWrapper-callback semantics level, Option C is the right call — Python user code observes the same pnl() values as it would against the upstream client, and ibx keeps its low-latency FIX-side advantage.

If "iso" is interpreted at the wire-protocol level, Option A is required, but it's a substantial undertaking.

Decision pending.

---

Original implication (superseded — see correction above)

The current ibx routing — CCP/FIX subscription + MidnightSeed + poll_pnl reconstruction — is on the wrong side of the gateway boundary. The midnight-seed message is what the gateway internally consumes to compute the daily-P&L value it then pushes to the upstream client; ibx is reimplementing gateway-internal math instead of subscribing to the already-computed result.

The correct fix is a passthrough that:

• routes req_pnl / req_pnl_single / cancel_pnl / cancel_pnl_single via the API-socket subscribe/stream/cancel triplet (the same pattern used by other EClient methods)
• drops MidnightSeed, poll_pnl, set_midnight_seeds, and the last_pnl change-detection cache as dead code
• emits the pnl() callback directly on every inbound stream frame

This structurally eliminates the failure mode described in this issue: no per-conId midnight-seed iteration means an intraday-opened position cannot be missed.

Next step: PR to follow once the existing API-socket call sites are surveyed and the new path lands behind a single subscribe→callback flow.

The "same pattern used by other EClient methods" is the part that's wrong — there is no such API-socket pattern in ibx today. The first-half observation about being on the wrong side of the gateway boundary remains accurate; what changes is what "the right side" requires (a new layer, not following an existing one).