Skip to content

Wire A32 circuit breaking into xDS channels - #2712

Open
mingley wants to merge 1 commit into
grpc:masterfrom
mingley:mingley/a32-circuit-breaking-wiring
Open

Wire A32 circuit breaking into xDS channels#2712
mingley wants to merge 1 commit into
grpc:masterfrom
mingley:mingley/a32-circuit-breaking-wiring

Conversation

@mingley

@mingley mingley commented Jun 30, 2026

Copy link
Copy Markdown
Contributor

Summary

This PR connects gRFC A32 circuit breaking to the xDS request path in tonic-xds. The circuit breaker works like a shared pool of slots: before an endpoint attempt begins, the channel reserves one slot for the selected {cluster, EDS service}. The CDS max_requests value sets the pool size, with the A32 default of 1024 when the field is omitted. If no slot is available, the client returns a local, non-retryable gRPC UNAVAILABLE response without contacting an endpoint.

  • Circuit-breaking limits follow live CDS updates. A permit stays reserved until the response body completes, errors, or is dropped, so the process-wide counter represents active RPC attempts across channels.
  • Local circuit-breaker responses are identified as pre-endpoint outcomes. Retry handling does not retry them, and outlier detection does not count them as endpoint failures.
  • The xDS cache preserves resource removals with generation and revision tracking. Removing CDS or EDS state clears obsolete breaker state and endpoints; re-added endpoints wait for a connector from the matching generation instead of reusing stale limits, connectors, or endpoint state.

Testing Done

  • cargo fmt --all -- --check — completed successfully.
  • cargo test -p tonic-xds client::loadbalance::loadbalancer::tests::circuit_breaker_drops_are_not_endpoint_outlier_outcomes -- --exact — 1 passed, 0 failed, 376 filtered out.
  • ci

@linux-foundation-easycla

linux-foundation-easycla Bot commented Jun 30, 2026

Copy link
Copy Markdown

CLA Signed
The committers listed above are authorized under a signed CLA.

@mingley
mingley force-pushed the mingley/a32-circuit-breaking-wiring branch 3 times, most recently from 4b4f77c to 01c865f Compare June 30, 2026 17:45
@mingley
mingley marked this pull request as ready for review July 30, 2026 23:00
@mingley
mingley force-pushed the mingley/a32-circuit-breaking-wiring branch from 53fe418 to e97a6ed Compare July 31, 2026 03:39
Comment thread tonic-xds/src/client/loadbalance/loadbalancer.rs
Comment thread tonic-xds/src/client/loadbalance/loadbalancer.rs
Enforce CDS max_requests across xDS channels with process-wide request counters. Preserve CDS and EDS removal generations so re-added clusters cannot reuse stale limits, connectors, or endpoints.
@mingley
mingley force-pushed the mingley/a32-circuit-breaking-wiring branch from 221f221 to 8e685d7 Compare August 7, 2026 18:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants