Cap exponential backoff in consume() at 5s

[justin-reid]

TL;DR

Problem. Coordinators::RedisCoordinator#consume doubles its XREADGROUP BLOCK <ms> interval on every empty poll with no upper bound. Once doubling has gone on long enough (around 15 empty iterations from the 10 ms start), a single BLOCK call can last 5–10 minutes, during which the worker cannot re-check complete? / abort?. Any transient moment when complete? is momentarily false after the queue has functionally drained — pipelined XACKs not yet visible, a worker that died holding a claim and not yet reaped, a slow ack write on a high-latency Redis path — is enough to push the worker into one of those multi-minute blocks, where it then sits until the BLOCK expires or the platform kills the job. The unbounded BLOCK is the amplifier: without it, the next iteration would be a short poll, the worker would re-check on its own, and the same transient inconsistency would heal in milliseconds.

Fix. Cap the doubled value at MAX_BACKOFF = 5_000 ms. Extract the doubling into a private next_backoff method so it's unit-testable without a live Redis. Bounds the worst-case unresponsiveness to a single MAX_BACKOFF window while keeping the same exponential ramp-up for the common case.

Details

Coordinators::RedisCoordinator#consume doubles exponential_backoff on every empty iteration with exponential_backoff <<= 1 and no upper bound. That value is passed to redis.xreadgroup(... block: <ms>), i.e. Redis XREADGROUP ... BLOCK <ms>, which the client implements as poll() on the socket.

Starting from INITIAL_BACKOFF = 10 ms, after 15 consecutive empty iterations a single BLOCK call lasts ~5m27s; after 16, ~10m. The break if combined_results.complete? and break if combined_results.abort? checks only run after the BLOCK returns, so once a worker has entered a multi-minute BLOCK it cannot escape until the BLOCK expires or the platform kills the job.

Under the right conditions this can surface as a post-test teardown hang: Minitest reports N/N 100%, but bin/rails test then sits in ppoll() on the coordinator's Redis socket for several minutes until the build times out. We've observed this path in production. It doesn't happen on every run — it requires the transient inconsistency to land between two specific iterations — but the worst case it leads to is unbounded.

The fix caps the doubled value at MAX_BACKOFF = 5_000 ms (reached after ~9 iterations, ~10 s of cumulative empty polls). It bounds the worst case where a worker is unresponsive to complete? / abort? after the queue drains to a single MAX_BACKOFF window, while keeping the same exponential ramp-up behaviour for the common case.

The doubling logic is extracted into a private next_backoff method so it can be unit-tested without a live Redis. New tests cover the doubling, the clamp, and the bounded iteration count.

Testing

• Added test/minitest/distributed/coordinators/redis_coordinator_test.rb covering: doubling below the cap, clamping at the cap, and the bounded-iteration count.

[justin-reid]

Heads-up on the red typecheck job: that failure is pre-existing on main, not introduced by this PR. The workflow runs on: push and main's last non-dependabot push pre-dates the Rubocop/Sorbet versions bundler now resolves, so the typecheck just hadn't run against current tooling until this branch.

The errors fall into four buckets, all unrelated to the backoff change:

1. ~4360 sig-less-method errors across sorbet/rbi/gems/*.rbi (autogenerated files defaulting to # typed: strict).
2. sorbet/rbi/minitest.rbi is # typed: strict but missing sigs for ~17 methods, and two existing sigs are wrong (assert_includes rejects String; assert_raises rejects blocks).
3. T::Array[Module] on redis_coordinator.rb:289 (custom_middlewares) hits Sorbet 5046 because Module needs type args.
4. prism-1.9.0's gem-shipped RBI references Prism::LexCompat::Result, a constant the gem removed in the same release.

Happy to clean these up in a separate PR so this one stays scoped to the actual fix.

_{🤖 AE · ✅ approved by @justin-reid}

[jose-shopify]

left a non-blocking nit, code looks good