fix(bench): cap a rung's backlog and retry per-scenario connects - #177
Merged
Conversation
…te cap An unsustained rung's backlog is resident on the broker for as long as the producers run. At 64 KiB the 25 000/s rung offers 1.6 GB/s to a container in an 8 GB VM; on 2026-09-08 Redis stopped answering under it, the next cell's connect timed out, and the pass died at cell 19 of 168. The rung had already shown the consumers cannot keep up; letting the producers run out the window only buried the broker. Stop a rung's producers once lag x payload_bytes passes the byte cap that already keeps a drain corpus resident (--drain-max-bytes, default 2 GiB), and record the rung as not sustained with the window it actually ran; the ladder skips the rungs above as it already does after any unsustained rung. A sustained rung never approaches the cap, so nothing that passes changes. The cap lives on HarnessConfig so the runners read it without a CLI in hand, and the CLI's --drain-max-bytes overrides it in both run loops. Tested against the in-process backend with a sleeping handler: a 5 s rung stops in under 2 s with the backlog at the cap and the verdict not sustained.
…g up Every stress binary's connect closure gave up on the first failure with an expect, so a broker still recovering from a heavy cell ended the whole pass instead of the one cell (Redis after the 64 KiB rung, 2026-09-08: "connect Redis: standalone ping failed: timed out" at cell 19). Route them through a harness helper that retries with doubling delays capped at ten seconds, up to ten attempts, and panics only when the broker stays down, which is the fail-fast the harness wants for a broker that is really gone.
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
Deploying with
|
| Status | Name | Latest Commit | Preview URL | Updated (UTC) |
|---|---|---|---|---|
| ✅ Deployment successful! View logs |
shove-docs | 255f118 | Commit Preview URL Branch Preview URL |
Sep 09 2026, 07:14 AM |
…rain honour it Review of the backlog cap found the stop half-wired: the row still claimed the nominal window, so the document's validation refused the whole results file for a window shorter than the rung's; the verdict never saw the cap, so a rung stopped at a lag ratio under the threshold could read as sustained and let the ladder climb; the residual drain still ran the sixty-second grace and failed the cell when slow consumers could not clear the backlog; --drain-max-bytes without --drain-messages was refused, so a load-only run could not set the cap it now reads; and the SQS connect never touched LocalStack, so its retry had nothing to retry on. Carry the cap through LoadOutcome into the row as backlog_capped (absent, hence false, on rows written before it existed), force the verdict to not sustained, let validation accept the shorter window and re-derive the verdict the same way, bound the post-cap drain to five seconds and leave the rest to the next cell's purge, accept the flag whenever a ladder is selected, and ping inside the SQS connect callback. The backlog test now pushes its row and validates it; a new test pins the flag's acceptance.
Kafka's connect builds the producer without contacting the cluster, the same shape review found on SQS: a broker that is down passes the retry and fails the scenario later. Ask the cluster inside the callback.
The first pass of the cap read --drain-max-bytes, and at the matrix's 3 GiB it fired too late: Redis and RabbitMQ both survived a 3.2 GB drain corpus published to an idle broker and both died under a 3.2 GB backlog piled onto a broker that was also serving the consumers and still reclaiming the previous cell (2026-09-08/09, the 64 KiB rung at 25 000/s on each). The two caps bound different things and cannot share a number. Make the backlog cap --load-backlog-max-bytes, default 1 GiB, with a compile-time check that it stays below the drain cap; give --drain-max-bytes back its single meaning and its original refusal without --drain-messages. A rung a gigabyte behind is unsustained by any definition, so nothing that passes changes.
zannis
changed the title
This was referenced Sep 9, 2026
zannis
added a commit
that referenced
this pull request
Sep 9, 2026
The harness now stops an offered-load rung whose backlog passes the byte cap, records the window it actually ran and flags the row backlog_capped (#177, #180). chartgen validates the same window invariant the harness did and refused the first six-backend document to carry such a row: "measured window is shorter than the rung's nominal window", which blocked publishing the RabbitMQ re-run. Read the flag into LoadAccount (absent, hence false, on older rows), let the window check pass when it is set, and fold it into the re-derived verdict so a capped row's not-sustained claim matches, as the harness does. A short window without the flag is still malformed.
This was referenced Sep 9, 2026
zannis
added a commit
that referenced
this pull request
Sep 9, 2026
* fix(chartgen): accept a rung the harness's backlog cap stopped The harness now stops an offered-load rung whose backlog passes the byte cap, records the window it actually ran and flags the row backlog_capped (#177, #180). chartgen validates the same window invariant the harness did and refused the first six-backend document to carry such a row: "measured window is shorter than the rung's nominal window", which blocked publishing the RabbitMQ re-run. Read the flag into LoadAccount (absent, hence false, on older rows), let the window check pass when it is set, and fold it into the re-derived verdict so a capped row's not-sustained claim matches, as the harness does. A short window without the flag is still malformed. * test(chartgen): prove the cap flips the verdict, not just the window check The first capped fixture already failed the lag threshold, so the verdict override had no coverage of its own. A capped row whose consumers kept up must render as not sustained and be refused if it claims sustained.
zannis
added a commit
that referenced
this pull request
Sep 9, 2026
Re-measure RabbitMQ on the pinned matrix with the harness fixes from #172, #177 and #180 in place: the memory watermark that let the 64 KiB corpus stay resident, the FIFO flow without --concurrent, and the backlog cap on the ladder. 140 rows, one failure. Against the published run the 64 KiB drains are 2 to 14x higher (they were memory-alarm throttled), FIFO at 64 B recovers from 2.4k to 18.4k msg/s, and the 4-consumer dips are gone. The 64 B and 1 KiB drains read 20 to 50 % lower than the published run, most on consumer_group; an isolated re-measurement of two of those cells reproduced the consumer_group figures, so they are recorded as measured rather than dismissed as noise, and the cause is still open. The one failure is the 64 KiB consume_parallel drain at eight consumers, whose corpus was fully consumed before the last worker attached. Charts follow once chartgen accepts a backlog-capped row (#182); the three 64 KiB rungs the cap stopped are in this document.
zannis
added a commit
that referenced
this pull request
Sep 9, 2026
) Re-measure RabbitMQ on the pinned matrix with the harness fixes from #172, #177 and #180 in place: the memory watermark that let the 64 KiB corpus stay resident, the FIFO flow without --concurrent, and the backlog cap on the ladder. 140 rows, one failure. Against the published run the 64 KiB drains are 2 to 14x higher (they were memory-alarm throttled), FIFO at 64 B recovers from 2.4k to 18.4k msg/s, and the 4-consumer dips are gone. The 64 B and 1 KiB drains read 20 to 50 % lower than the published run, most on consumer_group; an isolated re-measurement of two of those cells reproduced the consumer_group figures, so they are recorded as measured rather than dismissed as noise, and the cause is still open. The one failure is the 64 KiB consume_parallel drain at eight consumers, whose corpus was fully consumed before the last worker attached. Charts follow once chartgen accepts a backlog-capped row (#182); the three 64 KiB rungs the cap stopped are in this document.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Both item-4 re-runs died at the 64 KiB ladder rung at 25,000/s: Redis at cell 17, RabbitMQ at cell 32. The rung offers 1.6 GB/s to a container in an 8 GB VM; the broker stopped answering, the next cell's connect hit its
expect, and the pass ended. Both brokers had survived a 3.2 GB drain corpus minutes earlier; what killed them was the same bytes piling up as backlog while they also served consumers and reclaimed the previous cell. Three harness changes, no matrix change:lag × payload_bytespasses--load-backlog-max-bytes(default 1 GiB, checked at compile time to stay below the 2 GiB drain cap) the rung's producers stop, the rung is recorded as not sustained with the window it actually ran andbacklog_cappedset on its row, and the ladder skips the rungs above as it already does after any unsustained rung. A sustained rung never approaches the cap, so no passing cell changes. Validation accepts the shorter window and re-derives the verdict the same way; the post-cap drain waits five seconds and leaves the rest to the next cell's purge.connect_with_retriesfor every stress binary's per-scenario connect. Doubling delays capped at ten seconds, up to ten attempts, a panic only when the broker stays down. Kafka's and SQS's callbacks ping, because theirconnectbuilds clients without contacting the broker.--drain-max-byteskeeps its single meaning and its refusal without--drain-messages.History in this PR
The first cut read the cap from
--drain-max-bytes. Review found it half-wired (a capped row failed the document's validation, the verdict ignored the cap, the residual drain could still error, the flag was refused for load-only runs, the SQS and Kafka retries could never fire); all fixed. The re-runs then showed the 3 GiB matrix value fires too late for either broker, hence the separate 1 GiB default.Tests
a_rung_whose_backlog_reaches_the_byte_cap_is_stopped_early: an in-process rung with a sleeping handler and a 500-message cap stops in under 2 s of a 5 s window, records peak lag at the cap, is not sustained, and its row passesvalidate_run.connect_with_retries_survives_a_broker_that_answers_on_the_third_try.the_backlog_cap_has_its_own_flag_and_a_lower_default_than_the_drain_cap.cargo fmt -- --checkand both clippy gates pass.benches/README.mddocuments the cap next to the SQS deviation.