Bug (circuit_breaker): single-prober election is per-process — concurrent threads all probe on recovery (the thundering herd it exists to prevent)

[Iamrodos]

Hi 👋 — another find from our circuit-breaker alpha evaluation. This one's a small tweak we'd love to see while it's still alpha: it would open the utility up to use cases that are currently blocked — concurrent, in-process (threaded) fan-out to a shared circuit. Details below.

Expected Behaviour

On recovery, the breaker elects a single prober — exactly one request tests the recovering backend, which (per the docs) "stops a thundering herd of every environment hammering a recovering backend at once." And the failure counter trips at exactly failure_threshold.

Current Behaviour

Under concurrent threads sharing one circuit, neither holds. The single-prober election keys on a per-process id, so all threads in a process share the owner: after one thread wins the half-open election, its sibling threads pass the half_open_owner == <id> check too and also probe. We saw two probes for a single recovery window — the exact thundering herd the election exists to prevent. (Separately, the in-memory failure counter's increment isn't atomic, so it can overshoot failure_threshold across threads.)

We hit this evaluating the breaker around an SMTP send behind a worker-thread pool. We recognise the utility is modelled per-execution-environment (single-threaded under the Lambda model), so this may be considered by-design — but the protection silently fails under the natural threaded use, and nothing in the docs signals the assumption.

Possible Solution

Make the probe-owner id per-thread: the existing DynamoDB conditional election then elects a single thread across all threads and processes — cross-process coordination is unchanged (it's the DDB write; per-thread granularity just lets the same write disambiguate threads). Lock the in-memory counter increments, and guard the persistence's per-call state.

No boto3 change needed — the persistence already uses the low-level client, so the heaviest part of Idempotency's own thread-safety fix (#1899) is already done here. Thread safety has been added across the toolkit over time (Idempotency #1899, Logger thread-safe keys #5141, with CI for it #6889), so this keeps the breaker in step. If instead the single-threaded assumption is intended to stay, at minimum the docs should say so — a "not thread-safe within a process" note like Logger's append_keys warning.

Tests we'd add (fitting the existing thread-safe CI patterns, #6889):

• N threads sharing one circuit → exactly one probe per recovery window, regardless of thread count.
• N threads driving failures → the circuit trips at exactly failure_threshold, with no overshoot from racing increments.
• The same guarantees hold across processes as well as threads — one election spanning both.

Happy to open a PR with the per-thread election, the locks, and these tests.

Steps to Reproduce

1. Share one circuit across N worker threads in a single process.
2. Make the guarded downstream fail so the circuit trips OPEN.
3. Keep the threads calling; after recovery_timeout, more than one thread runs the half-open probe concurrently — multiple HALF_OPEN → OPEN reopens for one recovery window (and the failure counter can trip a few over failure_threshold).

Environment

• Powertools for AWS Lambda (Python): 3.31.0 — utility circuit_breaker_alpha
• Runtime: Python 3.14 (not Lambda-specific — reproduces in any multi-threaded process)
• Packaging: PyPi

[leandrodamascena]

Thanks for this detailed analysis @Iamrodos, really solid write-up. I reproduced both problems locally and you're right on both.

The probe one is the serious bug. _ENVIRONMENT_ID is generated once per process at import, so every thread in the process carries the same id. The DynamoDB conditional write does elect a single owner correctly, but then the record.half_open_owner == _ENVIRONMENT_ID check passes for every sibling thread, since they all share that id. So they all probe. I ran it with 8 threads and got 8 probes, which is exactly the thundering herd the election is supposed to stop. And it fails silently, nothing in the logs or docs hints at it.

The counter is real too, though slightly different from how it reads in the issue. The non-atomic get() + 1 mostly causes lost updates, which makes the circuit trip late rather than overshoot, and separately several threads can each see failures >= threshold in the same window and each call save_open, so you get redundant writes for one transition. Both come from the same missing lock.

Two things I'd add to the fix while you're in there: _LOCAL_SUCCESSES in the probe path has the identical race, and _LAST_OBSERVED_STATE is also read/written without a lock. Worth covering all of them in one pass rather than just the failure counter.

On whether this is by-design: the utility was modelled per-execution-environment, so single-threaded was the working assumption. But it's alpha, and this is the cheap moment to make it thread-safe before the contract sets. Idempotency and Logger already went thread-safe, so I'd rather keep the breaker in step than ship a "not thread-safe within a process" note. Your per-thread id idea is the right call, the same DynamoDB write then disambiguates threads and processes with no protocol change.

Please go ahead with the PR. The tests you listed are the right ones, and there's the existing thread-safe CI pattern (#6889) to follow so they stay deterministic. Just fold in the successes counter and the observed-state dict alongside the election fix.

[Iamrodos]

Please go ahead with the PR. The tests you listed are the right ones, and there's the existing thread-safe CI pattern (#6889) to follow so they stay deterministic. Just fold in the successes counter and the observed-state dict alongside the election fix.

All done.