feat: P1 gate storm harness — load + N leader switchovers + divergence report#44
Merged
Merged
Conversation
…e report The P1.5 acceptance instrument (match#32): one command produces the divergence/overflow numbers used to accept P1.1-P1.4. - tools/durability/storm.py: orchestrates the existing pieces (repro-bug9 observer, oms_load, reconcile) through sustained OMS-REST load + >=50 forced leader switchovers -> runs/<ts>/p1-gate-report.json. Detaches the market gateway during the run (match#37 OOM until P3.3) and restores it; REQUIRES the truthful status API (admin-gateway#13) and cross-checks its leader claim against the observer's newLeader events. Exit 0 = harness completed; workstream gates live in the report (--enforce-gates once P1.1/P1.2 land). - tools/durability/diag_counters.py: harvests submitted/terminal/overflowRej/ droppedMkt/droppedOms from EGRESS-DIAG lines in the node logs, stitching PM log rotation and summing per-incarnation values (counters reset on every node restart; a decrease marks a new incarnation). - tools/repro-bug9/reconcile.py: adds the terminal-vs-OPEN classification (omsOpenClusterTerminal = the stuck-holds direction / omsTerminalClusterOpen) from observer statusRaw + trade-tape fully-filled evidence, and switches per-order REST gets to bulk per-user fetches (a storm run submits far too many orders for one GET each). - Makefile: p1-gate / p1-gate-smoke targets; run-load-test.sh: LOADGEN_CORES taskset pinning (unpinned load-gen reports an artifactual ~9k/s ceiling). Part of match#32 (P1.5). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
emrebulutlar
added a commit
that referenced
this pull request
Jul 3, 2026
A 5-minute auto-snapshot firing while the storm has a node down or rejoining is exactly the snapshot/housekeeping-with-a-member-down hazard (match#35) and would strand a member mid-measurement. Suspend it at run start and restore the prior schedule in the finally, alongside the market-gateway restore. Follow-up to #44 (match#32 P1.5). Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
emrebulutlar
added a commit
that referenced
this pull request
Jul 3, 2026
The first full 50-switchover run (20260703-224136) exposed a race: a status read immediately after restart-node returns the CACHED pre-kill world (2s poller), so wait_healthy_core passed in 0.0s, the pre-kill leader was recorded (all 50 'disagreements'), and kills outpaced the OMS's reconnect — it rejected 309k of 312k orders and the divergence numbers were measured over near-zero in-flight traffic. The observer cross-check flagged it (statusObserverAgreement=false), which is how this was caught. wait_switchover_settled(killed) now requires, in order: the restart is OBSERVED (killed node leaves HEALTHY or the leader moves off it), TWO consecutive healthy-core polls, and an accepted canary order (dedicated user 819999, far-from-market LIMIT, submit+cancel) proving OMS -> cluster ingress is live end-to-end before the next kill. Also: load-collapse guard in the report (no silent caps) — acceptance rate below 50% marks loadCollapsed and fails harnessOk instead of presenting a near-empty measurement as a valid baseline. Follow-up to #44 (match#32 P1.5). Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
emrebulutlar
added a commit
that referenced
this pull request
Jul 3, 2026
…51) Run 20260703-233910 aborted at switchover 1 because the canary (BUY at price=1) was PRICE_COLLAR-rejected for the whole 90s settle window. The canary now SELLs 0.05 at mid*1.05: inside the 10% collar, resting (never crosses), and clear of the FixedPoint negative-notional band (~$922-$1844, match#30) that spuriously rejects ~3% of typical orders today. README: run-sizing guidance for the OMS open-order slot leak found tonight (500/user in-memory counter never decremented for bug9-lost terminals; consecutive runs walled at 15001 -> 2356 -> 617 accepted). Restart OMS before full runs; size users x 500 > orders x ~0.2 until P1.2 lands. Follow-up to #44/#50 (match#32 P1.5). Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #32 (P1.5) — the acceptance instrument for P1.1-P1.4:
make p1-gate(orp1-gate-smokefor a ~5-minute 3-switchover variant) producesruns/<ts>/p1-gate-report.jsonwith the gate numbers first-class: terminal-vs-OPEN divergence counts/pcts, overflow-rejects, engine counters, invariants.healthfields — and cross-checks the status leader claim against the observer'snewLeaderevents, so the harness never solely trusts the API it is stressing.omsOpenClusterTerminal= the dangerous stuck-holds direction) and switches per-order REST gets to bulk per-user fetches.LOADGEN_CORESpinning for run-load-test.sh.Wall-clock: ~15-30s per switchover cycle, so the full 50-switchover gate is a 25-45 minute run. P1.2's acceptance (0% terminal, <0.5% self-healing OPEN) reads straight off the report's
divergenceblock.🤖 Generated with Claude Code