feat(service): add CircuitBreaker middleware#2545
Open
Mattbusel wants to merge 3 commits intohyperium:masterfrom
Open
feat(service): add CircuitBreaker middleware#2545Mattbusel wants to merge 3 commits intohyperium:masterfrom
Mattbusel wants to merge 3 commits intohyperium:masterfrom
Conversation
gRPC channels can fail silently — a downstream that is overwhelmed or
crashing will keep accepting connections and returning errors rather than
refusing them. Without a circuit breaker, callers retry into a failing
service, wasting resources and extending outages.
Add tonic::service::circuit_breaker::{CircuitBreaker, CircuitBreakerLayer}:
- Three-state machine: Closed → Open → HalfOpen
- Closed: requests flow through normally
- Open: requests are immediately rejected with Status::unavailable
("circuit breaker is open") until `timeout` elapses
- HalfOpen: one probe request is allowed; success above
`success_threshold` closes the circuit, any failure reopens it
The implementation is pure Tower middleware — no async runtime dependency
in poll_ready/call, state guarded by std::sync::Mutex for zero overhead.
Usage:
let channel = ServiceBuilder::new()
.layer(CircuitBreakerLayer::new(5, 0.6, Duration::from_secs(30)))
.service(channel);
let mut client = MyClient::new(channel);
Uses pin_project and tower_layer/tower_service already in [dependencies].
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
gRPC channels can fail silently. A downstream that is overwhelmed or crashing continues to accept connections and return errors rather than refusing them. Without a circuit breaker, callers flood a failing service with retries, wasting resources and extending recovery time.
Tonic already exposes Tower's
Interceptor,RecoverError,LayerExt, andLayeredhelpers throughtonic::service. A circuit breaker is the natural complement — it is the canonical resilience pattern for RPC clients.Change
Adds
tonic::service::circuit_breaker::{CircuitBreaker, CircuitBreakerLayer}and re-exports both fromtonic::service.State machine
Status::unavailable("circuit breaker is open").timeout; success abovesuccess_thresholdcloses the circuit, failure reopens it.Usage
Implementation notes
poll_ready/call— state is guarded bystd::sync::Mutex(nottokio::sync), so it works correctly in single-threaded test executors.poll_ready(notcall), respecting the Tower service contract.pin_projectandtower_layer/tower_servicealready in[dependencies]— no new deps.