fix(queue): bound the maintenance lane so a retry storm cannot re-saturate it#1283
Merged
Conversation
…urate it
The shared gittensory-jobs consumer had no max_concurrency (Cloudflare may fan
out many concurrent consumers each pulling a heavy sweep/backfill batch,
amplifying GitHub-rate pressure) and no retry_delay (a failed job re-delivers
immediately, re-hammering the already-overloaded path). Now that webhooks have
their own lane, bound this maintenance lane:
- max_batch_size 5 (was 10): a batch can't bundle as many heavy jobs at once.
- max_concurrency 3: bounded fan-out — drains metagraphed's worst sweep within
the 2-min cron interval without flooding the shared installation rate bucket.
- retry_delay 30: back off a failed job 30s instead of re-hammering instantly.
Webhooks are unaffected (their own lane). A follow-up may tighten max_concurrency
to 2 once the sweep fans into tiny per-PR jobs.
Contributor
|
Superagent didn't find any vulnerabilities or security issues in this PR. |
This was referenced Jun 24, 2026
This was referenced Jul 18, 2026
galuis116
pushed a commit
to galuis116/gittensory
that referenced
this pull request
Jul 19, 2026
JSONbored#4892 asked to make the hosted loopover-jobs queue's Cloudflare Queues consumer settings (max_batch_size/max_concurrency/max_retries/retry_delay) runtime-configurable and re-derive better defaults. Neither half is actually achievable right now: these are consumer-binding attributes Cloudflare's control plane reads at `wrangler deploy` time, before any Worker code runs -- there's no env-var override mechanism for them, and re-deriving new defaults needs real production job-volume data this worktree has no access to. What's real and worth doing: the existing comment justified the current values against one specific repo's historical incident (JSONbored#1283, "metagraphed's worst sweep"). Rewrites it as a durable, formula-based constraint (max_batch_size × max_concurrency bounds concurrent in-flight GitHub calls against one installation's rate-limit headroom) so a future re-tune starts from a stated principle instead of one repo's sizing. No literal value changed -- verified via `wrangler deploy --dry-run`. Advances JSONbored#4892
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The shared
gittensory-jobsconsumer had nomax_concurrency(Cloudflare may fan out many concurrent consumers, each pulling a heavy sweep/backfill batch — amplifying GitHub-rate pressure) and noretry_delay(a failed job re-delivers immediately, re-hammering the already-overloaded path). This is the magnitude half of the overload (Flaw B). Now that webhooks have their own lane (#1276), bound this maintenance lane:max_batch_size5 (was 10): a batch can't bundle as many heavy jobs at once.max_concurrency3: bounded fan-out — drains metagraphed's worst sweep (50 PRs) within the 2-min cron interval without flooding the shared installation rate bucket.retry_delay30: back off a failed job 30s instead of re-hammering instantly.Webhooks are unaffected (their own lane). A follow-up may tighten
max_concurrencyto 2 once the sweep fans into tiny per-PR jobs.Scope
wrangler.jsonconly — consumer config on the existinggittensory-jobsqueue. No new binding, no DB migration, nosrcchange (so no coverage obligation).Validation
max_concurrency,retry_delay,max_batch_size) via the Cloudflare docs — all within limitsnpx wrangler deploy --dry-runsucceeds (config parses + builds);cf-typegenproduces no binding-type change;tsccleanmainimmediately before opening · reversible config-only changeSafety
site//CNAME/lovable; no changelog edit