Circuit Breakers

Circuit breakers

A circuit breaker stops calling a dependency that is already failing. Without one, every request keeps paying the full timeout to a service that will not answer, and the retries pile on top of whatever is wrong.

cb := health.NewCircuitBreakerWithStore(
	health.DefaultCircuitBreakerConfig(),
	ext.StateStore(),
	"payments-api",
)

The three states

State	Behaviour	Leaves when
CircuitClosed	calls pass through	FailureThreshold consecutive failures
CircuitOpen	calls are refused immediately	Timeout elapses → half-open
CircuitHalfOpen	at most HalfOpenMaxCalls probes	SuccessThreshold successes → closed; any failure → open

type CircuitBreakerConfig struct {
	FailureThreshold int           // 5
	SuccessThreshold int           // 2
	Timeout          time.Duration // 30s
	HalfOpenMaxCalls int           // 1
}

DefaultCircuitBreakerConfig() returns those values.

Half-open with HalfOpenMaxCalls: 1 is what stops the thundering herd: when the timeout expires, exactly one request probes the dependency rather than every waiting request arriving at once.

Using one directly

if !cb.Allow() {
	return nil, &health.CircuitOpenError{DepID: "payments-api"}
}

resp, err := call(ctx)
if err != nil {
	cb.Failure()
	return nil, err
}
cb.Success()
return resp, nil

Reset() forces it closed — useful in tests, and for an operator endpoint that clears a breaker after a fix is deployed.

Wiring it to dependency state

NewCircuitBreakerWithStore keeps the breaker and the state store in step: transitions call SetCircuitState, so the current state appears on /status and the gate can act on the same information.

Use it in preference to the bare NewCircuitBreaker whenever a state store exists — an open breaker that nothing can observe is an outage with no diagnostic.

With a wrapped HTTP client

The usual way to get one, with no bookkeeping of your own:

client := health.WrapHTTPClient(http.DefaultClient, "payments-api", ext.StateStore(),
	health.WithCircuitBreaker(cb),
	health.WithHTTPTimeout(3*time.Second),
	health.WithRetries(2, 200*time.Millisecond),
)

resp, err := client.Get(ctx, url)
var open *health.CircuitOpenError
if errors.As(err, &open) {
	// refused without a call — serve stale, queue, or fail fast
}

Every call reports success or failure with its latency, and the breaker refuses calls while open.

Choosing thresholds

FailureThreshold against your traffic rate. Five consecutive failures at 1000 rps is five milliseconds of trouble. On a low-traffic path it may be minutes. Match it to how long you are willing to keep trying.

Timeout against recovery time. Too short and the breaker reopens repeatedly while the dependency is still restarting; too long and you stay dark after it has recovered. 30 seconds is a reasonable default for a service that restarts in seconds.

SuccessThreshold above 1 for a dependency that can answer one request and then fail again — a database that accepted a connection from a pool that is still draining, say.

Breaker or state store thresholds?

They overlap and answer different questions:

• The state store's FailureThreshold decides what /status and the gate believe about a dependency.
• The breaker's FailureThreshold decides whether the next call is even attempted.

You usually want both, and wiring the breaker to the store with NewCircuitBreakerWithStore is what keeps their views consistent.