fix(selection): make recovery probes breaker-ineligible (probe/selection wedge defence-in-depth)#4
Open
snowkide wants to merge 1 commit into
Conversation
…ion wedge defence-in-depth)
Defence-in-depth for the same probe/selection wedge as the half-open permit
leak: a selection-recovery probe rides u.Forward through the upstream circuit
breaker. When the breaker is open, TryAcquirePermit denies the probe, Forward
returns ErrFailsafeCircuitBreakerOpen, and the prober records that denial as a
probe FAILURE in the selection tracker. The upstream's error rate stays high, the
selection policy keeps it excluded, the next probe is denied too — the upstream
can never gather a real health signal and stays excluded until restart.
Fix: mark a probe's context as breaker-ineligible so it neither acquires a permit
nor records an outcome into the breaker. The probe reaches the upstream
regardless of breaker state and its REAL result (success or genuine failure)
flows through the selection tracker, so the policy re-admits the upstream once it
is actually healthy. The breaker still recovers independently via its HalfOpen
trials on real traffic — probes don't pollute the breaker window.
Implementation is a context flag, NOT a request mutation: the prober shares the
NormalizedRequest with live client traffic (mirror() forwards the same object),
so flipping a field on it would corrupt the in-flight client request. The flag
lives on the probe's own detached ctx instead.
- common: WithSelectionProbe / IsSelectionProbe (new selection_probe.go)
- upstream: upstreamBreakerEligible now consults ctx; eligibility decided once
so permit acquisition and outcome recording stay consistent
- internal/policy/prober: tag the probe ctx in mirror()
Tests: upstreamBreakerEligible returns false for a probe ctx, true for ordinary
traffic, false for hedges; context marker roundtrip. policy + upstream + failsafe
packages still pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
xray — see through AI slop with deterministic architecture PR diff reviews |
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Context
Split out from #3 (the half-open permit-leak fix) so the two changes can land and be reviewed independently. #3 is the root-cause fix for the probe/selection wedge; this PR is defence-in-depth on top of it — they touch disjoint files and do not depend on each other.
Problem
Selection-recovery probes (
prober.mirror) send a non-internal request, soupstreamBreakerEligiblewastruefor them — i.e. a probe went through the circuit breaker. The probe's documented purpose is to "reach the upstream so it can prove itself" and feed a fresh health signal to the selection policy. But if the upstream's breaker is open, the probe is denied a permit, the prober records that breaker-open denial as a probe failure, the upstream's error rate stays high, and it stays excluded forever — a self-reinforcing wedge that no real traffic can break.Fix
Mark selection-recovery probes as breaker-ineligible: they neither acquire a permit nor record an outcome into the breaker. The breaker still recovers on its own via HalfOpen trials on real traffic; the probe's signal flows through the selection tracker instead.
common.WithSelectionProbe(ctx)/IsSelectionProbe(ctx)marker. We tag the ctx, not the request —prober.mirrorshares thereqobject with live client traffic.prober.mirrortags its probe ctx.upstreamBreakerEligiblenow takesctxand returnsfalsefor a selection probe (mirroring the existing hedge / internal-probe / composite skips).breakerEligibleonce and reuses it for both permit-acquire and outcome-record, so the two can never disagree.Behaviour change to note
Probes no longer contribute to the breaker's success/failure counters at all. Recovery is driven purely by real-traffic HalfOpen trials (after selection re-includes the upstream on the probe signal). This is the intended, more-correct behaviour, but it is a genuine change from today — worth watching for any upstream that has an open breaker and near-zero real traffic.
Tests
upstream/breaker_eligibility_test.go— a selection probe is breaker-ineligible; ordinary traffic stays eligible; hedge stays ineligible; ctx marker round-trips (incl. nil).Verified: probe branch builds and tests pass standalone (without #3's
breaker.gochange), confirming the two PRs are independent.🤖 Generated with Claude Code