feat(proxy): add CN health circuit breaker to skip overloaded CNs - #25073
Conversation
Add a per-CN health circuit breaker that tracks connection failures and
temporarily skips unhealthy CNs during route selection, preventing new
connections from paying auth timeout costs on known-bad CNs.
- cn_health: new circuit breaker with configurable fail threshold,
exponential cooldown (base→max), half-open probe window, and stale
entry sweeping.
- router: integrate health.pick() into Route(); add ConnectRouteSelected
to distinguish Route-selected connects (feeds breaker) from
internal/admin connects (does not).
- client_conn: use RouteForTransfer for session migration to avoid
consuming breaker-managed probes; Route-selected new-session
connects feed timeout/overload failures into the breaker.
- config: expose CN-health-check-* configuration knobs.
- metrics: add proxy_cn_health_{unhealthy,trips,success_probe} counters.
Fixes #24919
Qodo reviews are paused for this user.Troubleshooting steps vary by plan Learn more → On a Teams plan? Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center? |
gouhongshen
left a comment
gouhongshen
left a comment
There was a problem hiding this comment.
I rechecked the latest head (bb1ad656). The code-side cache bypass issue is addressed: cached reuse now checks CN health before SET CONNECTION ID, and plugin routing no longer allows fresh-login cache reuse.
I still cannot approve this exact diff yet because two previously raised cleanup/metadata issues remain in the current head:
- The PR body still lists
#24919under "Which issue(s) this PR fixes", but that issue is about a cross-CN shuffleINSERT ... SELECTexecution deadlock, not proxy CN health routing. pkg/sql/colexec/table_function/filter_helper_gpu_test.gois still in the diff with whitespace-only comment alignment changes, despite the reply saying it was dropped.
Please remove those from the current PR state and I can approve the code changes.
gouhongshen
left a comment
gouhongshen
left a comment
There was a problem hiding this comment.
LGTM. Rechecked the latest head; the proxy CN health breaker now gates cached connection reuse before SET CONNECTION ID, keeps plugin fresh-login routing from being bypassed by connCache, and fixes the cooldown overflow path.
5 participants
What type of PR is this?
Which issue(s) this PR fixes:
#24919
What this PR does / why we need it:
Introduce a CN health circuit breaker in Proxy routing to skip temporarily unavailable CN nodes, so fresh logins stop repeatedly paying auth timeout on known-bad CNs.
Highlights:
ErrAllCNServersBusywhen all candidates are temporarily unhealthy.cn-health-check-fail-thresholdto 2, so a single transient blip does not immediately sideline the only CN.SET CONNECTION ID.