Sweep the jobqueue at campaign scale and record what it proves - #16
Merged
Conversation
benchmarks/campaign.py spends seeds three ways: 100,000 seeds of chaos against the intact jobqueue (green in six minutes at ~280 seeds/s on ten processes), 10,000 seeds against each ablation counting every failing seed rather than the first (densities recorded; the missing idempotency key fails roughly every other seed, the rest on every seed), and 100 re-runs of 20 sampled failing seeds proving identical trace hashes throughout. Both sweeps checkpoint and resume. Scale found one thing the 200-seed test budget cannot: two seeds where the keyless duplicate is still queued at settle, surfacing as a convergence violation — same missing safeguard, different symptom, replayable like everything else. A modest nightly workflow keeps the numbers honest.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
benchmarks/campaign.py(green / ablations / stability subcommands, chunked with checkpoint-and-resume) plus the recorded results inbenchmarks/README.md:Scale surfaced one genuinely new fact: on 2 of 10,000
no-idempotency-keyseeds the keyless duplicate is still queued when the cluster settles, so the violation lands asconvergencerather thanexactly-once— invisible at the 200-seed test budget, replayable from its seed with a stable hash, and now documented (the campaign's expected-invariant check is what caught it).Also: a small nightly campaign workflow (2,000 green seeds + 500-per-ablation + a stability sample on the ubuntu runner) so the published numbers keep being re-earned; README's performance section now cites the parallel campaign figure; changelog bullet added. Checkpoint files are gitignored.
262 fast + 10 slow pass, mypy strict clean.