agileLimiter

An adaptive, latency-driven rate limiter that protects a downstream dependency.

Standalone Go module (module agilelimiter), zero external dependencies. It is deliberately small: a simple leaky bucket (one mutex) is the actuator, and a feedback controller decides the rate. A static ceiling is enforced in two independent places, so the controller can never exceed a number you are willing to promise.

downstream RTT ──> histogram (p50/p99) ──> fast/slow EWMA ──> rate factor ──> clamp[min,max] ──> bucket.SetRate
   (Observe)                                     (baseline)      gradient/AIMD

lim := agilelimiter.New(agilelimiter.Config{
	MinRate:     20,         // never go below this many req/s
	MaxRate:     500,        // hard ceiling you promise the downstream
	InitialRate: 50,         // optional; defaults to MaxRate/8
	SLOTarget:   200 * time.Millisecond, // optional: p99 tail guard
})
lim.Start()
defer lim.Stop()

// Hot path: wrap the downstream call in Do. It takes a slot, times the call, and
// feeds the measured round-trip time back to the controller — you never touch
// Take/Observe, so you cannot feed it the wrong latency.
err := lim.Do(func() error { return doRequest(ctx) })

When the call returns a value, use the generic form:

resp, err := agilelimiter.Do(lim, func() (*http.Response, error) {
	return client.Do(req)
})

Do observes the round trip even when fn returns an error: a timeout is a real signal that the downstream is struggling, so it still moves the controller.

The signal the controller wants is the downstream round-trip time — measured from the moment the request was actually sent, not from the moment it was planned. Do gets this right for you: it starts its clock the instant the slot is granted, so what it feeds the controller is exactly that round trip.

If you instead feed end-to-end latency that includes this limiter's own throttle wait, you create a self-reinforcing loop: backing off raises the measured latency, which triggers more backing off, which collapses the rate to MinRate. Do is the default precisely because it removes that footgun. Reach for the manual Take/Observe pair only when Do cannot measure the round trip — for an async send, or a call that retries internally, where the value must be one attempt's real round trip rather than the wrapper's. When you do, feed Observe the downstream RTT and keep Take's planned-send timestamp for a coordinated-omission-safe reporting metric only:

t0 := lim.Take()               // planned send time (blocks until granted)
start := time.Now()            // actually sent
err := doRequest(ctx)
lim.Observe(time.Since(start)) // downstream RTT -> drives the controller
report(time.Since(t0))         // CO-safe end-to-end metric -> reporting only

A failure is not automatically a congestion signal, and getting this wrong makes the limiter actively harmful. A refused connection returns in microseconds; the controller reads that as "the downstream is fast and healthy" and raises the rate into a dependency that is already down. Measured: starting at 50 req/s against a port that refuses every call, the rate climbed to 335 req/s in ten seconds — and kept going.

The question that matters is not "did it fail?" but "did it reach the downstream?". Say so once at the boundary — Do/Do[T] take Config.Classify; the HTTP adapters take a Verdict (httpclient/httpserver below), and each call path has exactly one of the two:

lim := agilelimiter.New(agilelimiter.Config{
	MinRate:   20,
	MaxRate:   500,
	SLOTarget: 200 * time.Millisecond,
	Classify: func(err error) agilelimiter.Outcome {
		switch {
		case errors.Is(err, syscall.ECONNREFUSED), errors.Is(err, errBreakerOpen):
			return agilelimiter.Unreachable // never arrived
		case errors.Is(err, context.Canceled):
			return agilelimiter.Aborted // not the downstream's opinion
		default:
			return agilelimiter.Reached // timeout, 5xx: it answered
		}
	},
})

Details that matter:

• The default is Reached for every error (i.e. today's behaviour), so this is opt-in. It is right for anything the downstream answered; it is wrong only for calls that never arrived, and only you can tell the difference.
• The synthetic sample is placed above the tolerated window and above SLOTarget, so the proportional backoff and the p99 tail guard both fire. It is a no-op before there is anything to anchor against (no SLOTarget, no sample yet) — inventing a baseline value would be worse than staying silent.
• MinRate is the outage probe rate. Backing off to 20 req/s is how the limiter keeps testing whether the downstream came back. Make sure it is a rate the downstream can absorb (× your instance count).
• On the manual Take/Observe path, call lim.Unavailable() instead of Observe for that attempt. Never both — that records two samples for one call.
• ConcurrencyLimiter has the same options, for handlers that answer 503 without doing any work: Config.Classify and lim.Unavailable().

Pair this with a circuit breaker. The breaker decides "stop calling"; the limiter decides "how much of the remaining capacity to use". Without a breaker you still pay a full network timeout on every call while the limiter sits at MinRate.

A cache in front of a downstream splits its traffic into two populations sharing one pipe: a hit answers in ~1ms and consumes none of the origin's capacity, a miss costs whatever the origin costs. Feed both into one p50 and the controller tracks the cache's hit ratio instead of the origin's load. Measured (compare/cache_test.go, 95% hits, origin saturated):

The aggregate was green while the origin was timing out. So route hits past the controller:

client := &http.Client{
	Transport: httpclient.Wrap(http.DefaultTransport, lim),
}

Wrap takes a slot, measures the round trip, and hands the result to a Verdict: one function, called once per call, returning one of three answers.

type Verdict func(resp *http.Response, err error, d time.Duration) agilelimiter.Outcome

client := &http.Client{
	Transport: httpclient.Wrap(http.DefaultTransport, lim,
		httpclient.WithVerdict(myVerdict)),
}

That is the whole policy surface. There is no option per case, because the cases are not enumerable: a new proxy header, a new async endpoint, a circuit-breaker fallback — each is a line in your function, not a release of this library.

DefaultVerdict is what you get without one. It is small enough to read:

Copy it and add the cases you know about. Two it deliberately leaves to you:

case errors.Is(err, ErrBreakerOpen):
	return agilelimiter.Unreachable   // your own error, not a *net.OpError
case resp.StatusCode == http.StatusAccepted:
	return agilelimiter.Aborted       // the work has not happened yet

A custom verdict replaces DefaultVerdict entirely — Wrap does not consult Config.Classify (that one is for Do/Do[T], so each call path has exactly one policy entry point).

One thing a verdict cannot change: a canceled call is dropped by Wrap before the verdict runs. It is a truncated measurement, not a fast response, so a verdict that returns Reached for everything still cannot turn it into evidence.

Two things this does not do:

• It does not raise the ceiling for hits. A hit still takes a slot, because the caller cannot know a request will hit before sending it. To give hits a higher ceiling, split them onto a second limiter where a local cache already tells you the answer.
• It does not stop a cold cache from stampeding the origin. That is a step change, and an adaptive limiter is a lagging controller. Coalesce misses (single-flight so one fill serves many), size MaxRate for a cold origin, and keep a small cache-bypassing probe feeding the controller so the origin's real latency stays observable.

If the cache is yours (in-process, or Redis you query yourself), do not classify anything: put the limiter around the call that actually costs capacity — the cache hit never reaches it, so it never produces a sample.

Each tick:

1. Take p50/p99 from the per-tick histogram.
2. Update fast (short half-life) and slow (long half-life) EWMAs of p50.
3. Compare fast against a window around the tolerated latency high = slow * (1 + TargetInflation), low = high * (1 - Deadband) — or against 0.8 * SLOTarget outright under AnchorSLO (see below):
   • fast <= low → healthy, probe up: +UpStep, or ×SlowStartGain while the limiter has never seen congestion (cold start — see below).
   • fast >= high → congested, proportional decrease 1 - Gain*(excess/high), capped at -DownStep.
   • in between → hold (deadband).
4. If p99 exceeds SLOTarget, force a full -DownStep.
5. Clamp to [MinRate, MaxRate] and push to the bucket.

Asymmetry is intentional: back off fast (30%/tick), ramp up slowly (5%/tick). It is friendlier to the downstream and resists oscillation around the knee. The baseline EWMA itself is also asymmetric — it falls quickly and rises slowly (see Pitfalls).

Multiplicative probing happens in two cases. Before the first backoff the controller does not know where the knee is, so +5%/tick is guessing very slowly: 10→500 costs 80 ticks. It doubles instead (SlowStartGain, default 2.0), which costs 6. The asymmetry above is exactly what makes this safe — overshooting costs two ticks to correct, undershooting costs eighty.

The cold-start flag clears on the first backoff and never re-arms, which would leave a limiter at MinRate climbing at UpStep: a dip that took 8 ticks to fall into would take 80 to climb out of. Config.RecoverRamp is the opt-in answer — set it to 0.5 and the multiplicative probe returns whenever the measured latency sits at or below half the tolerated window. It is off by default, and the matrix below shows why: it turns 25 req/s into 52 when capacity comes back, and 59 into 46 when capacity drops, because doubling is also how you overshoot a cliff.

Config.Anchor decides the threshold, and it changes what the controller optimizes:

The default protects the knee, and that is the right default for a downstream that has one. But the SLO cap is one-way: min() can only lower the window, never raise it, so with a small baseline the SLO contributes nothing and the effective target is "2x the fastest response ever observed", whatever the SLO allows:

20ms service, 200ms SLO  ->  window = min(2 x 20ms, 160ms) = 40ms

A healthy-but-slower 50ms period is 2.5x that baseline, so the rate walks down to MinRate and stays: the baseline may only climb MaxBaselineRise per tick (~12%/min at the defaults) and the ramp back is UpStep per tick. On a virtual clock that is 500 -> 10 req/s in 26 ticks, and 88 ticks to return (TestBaselineAnchorCollapsesOnBenignRise). AnchorSLO reads 50ms against a 160ms threshold and does not move (TestSLOAnchorIgnoresBenignRise).

What it costs: the absolute controller keeps probing while the SLO has headroom, so it will not find a knee that sits below the SLO. If your downstream degrades sharply before its own latency target, keep the relative anchor — or keep it and set MaxRate to the knee instead.

Head-to-head against go.uber.org/ratelimit (cd compare && go test -run TestSLOAnchorVsUber -v), 20s runs, a downstream that answers 20ms and then 20ms -> 70ms at t=5s with capacity held at 150 req/s and a 100ms SLO:

The default law spends the rest of the run at MinRate on a latency that never breaks the SLO. When capacity really drops (150 -> 40 req/s) the picture reverses and the SLO anchor is the one that behaves: it settles at ~39 req/s with 83% of requests inside the SLO, while the fixed rate keeps sending 100/s into 40/s (619ms mean, 40% inside).

TestSLOAnchorMatrix runs the same comparison over seven scenarios — steady, benign bump, an SLO that really breaks, capacity dropping, capacity recovering, flapping latency and a diurnal curve — two rounds each, and prints a summary of which arm held up where (cd compare && go test -run TestSLOAnchorMatrix -v -timeout 40m). Goodput/s, best in bold:

Read %<=SLO alongside goodput, not goodput alone. What it says:

• No arm holds everywhere. A fixed rate is excellent while capacity is what it was and poor the moment it moves: uber(100) is at 52% inside the SLO on the diurnal curve and 38% when capacity drops, and uber(140) drops to 38% on a benign 20 -> 60ms bump.
• The adaptive arms stay inside the SLO across the board (78-95%) without being told the capacity. That is what the trade buys.
• Their weak spot is recovery, not backoff. 40 -> 150 req/s costs them the scenario outright (25 vs 98), because coldStart does not re-arm and the climb is UpStep per tick. Config.RecoverRamp is the opt-in answer: at 0.5 that scenario goes 25 -> 52, and the capacity-drop one goes 59 -> 46, because doubling is also how you overshoot a cliff. It ships off for that reason.
• AnchorSLO is the steadier of the two modes (flapping latency: 112-113 req/s in both rounds, against 95-105), and it is the one that survives a benign bump once the SLO has real headroom.

One caveat on the model: it prices load from the limiter's target rate, so a fixed-rate arm that actually completes 1.1-1.5x its nominal value is priced as if it sent exactly that. That flatters the uber rows slightly.

Both anchors move the rate on a single tick. BackoffTicks and RecoverTicks require that many consecutive ticks to agree before a step is spent — CoDel's interval, reduced to a counter. Defaults are 1/1 under AnchorBaseline (the historical behaviour) and 2/2 under AnchorSLO, where the threshold is fixed and jitter across it is the main false-positive risk.

This is not just smoothing. The steps are asymmetric (-30% down, +5% up), so a noisy signal does not average out — one bad tick and one good tick ratchet the rate down. Requiring agreement is what stops that. A tick inside the deadband resets both counters, so the agreement has to be in a row, not merely cumulative.

The limiter above protects a downstream. To protect a service you run — the inbound direction — use ConcurrencyLimiter instead. It shares the same brain (the fast/slow EWMA baseline, target window and p99 guard) but swaps the actuator from a leaky bucket to a concurrency semaphore, because the inbound problem is different:

• Rate is not your knob. Clients decide how fast requests arrive; what you control is how many you serve at once. By Little's Law in-flight = rate × latency, so a fixed rate cap does not bound concurrency — the same rate at 10× the latency is 10× the in-flight requests.
• Blocking would amplify an overload. A middleware that blocks on admission builds a queue holding the very goroutines, sockets and connections it is meant to protect. So admission here is non-blocking: at the limit it refuses at once, and you shed load with a fast 429 / ResourceExhausted.
• The signal is clean by construction. You feed the handler's own duration, and since admission never waits there is no throttle time to contaminate it.

lim := agilelimiter.NewConcurrency(agilelimiter.ConcurrencyConfig{
	MinLimit:     10,   // never admit fewer than this many concurrent
	MaxLimit:     500,  // hard ceiling you promise the service
	InitialLimit: 50,   // optional; defaults to MaxLimit/8
	SLOTarget:    200 * time.Millisecond, // optional: p99 tail guard
})
lim.Start()
defer lim.Stop()

// Hot path: Do admits the request if there is room, times the handler, and feeds
// the handling time back to the controller. At the limit it returns
// ErrOverloaded at once — refusing, not queuing.
err := lim.Do(func() error { return handle(w, r) })
if errors.Is(err, agilelimiter.ErrOverloaded) {
	http.Error(w, "overloaded", http.StatusTooManyRequests)
}

When the slot must span more than one call, or you want to release it yourself, drive the primitives directly. Every successful TryTake must be paired with exactly one Release:

if !lim.TryTake() {
	http.Error(w, "overloaded", http.StatusTooManyRequests)
	return
}
defer lim.Release()            // released even if the handler panics
start := time.Now()
handle(w, r)
lim.Observe(time.Since(start)) // this request's handling time -> drives the controller

Limit() returns the current concurrency limit and InFlight() the number of requests being served — export both. It takes the same tuning knobs as the outbound limiter; MinLimit/MaxLimit/InitialLimit take the place of MinRate/MaxRate/InitialRate. (MaxLimit is required.) For the four HTTP/gRPC integration seams, see docs/integrations.md.

agilelimiter/httpserver wraps a net/http handler (stdlib only, so the module stays zero-dependency). Apply it once, around the whole protected subtree — every route beneath shares one concurrency budget, because they compete for the same process resources (do not build a limiter per endpoint):

lim := agilelimiter.NewConcurrency(agilelimiter.ConcurrencyConfig{
	MaxLimit:     500,
	InitialLimit: 50,
	SLOTarget:    200 * time.Millisecond,
})
lim.Start()
defer lim.Stop()

mux := http.NewServeMux()
mux.HandleFunc("/api/things", getThings)
http.ListenAndServe(":8080", httpserver.Wrap(mux, lim,
	httpserver.WithRetryAfter(time.Second))) // sends Retry-After on a 429

Wrap runs the handler only for admitted requests and observes only those, so the signal stays clean; a shed request never reaches the handler and is never measured. Wrap releases the slot even if the handler panics, and httpserver.StatsHandler(lim) exposes the current limit and signals as JSON — mount it outside the wrapper so it stays readable while the service is saturated.

A handler that fails instantly reports a ~0 handling time, which would otherwise read as "this service is fast and healthy" and raise the limit. So the response is inspected before it is measured.

Same one-hook design as the outbound side:

type Verdict func(r *http.Request, status int, h http.Header, d time.Duration) agilelimiter.Outcome

DefaultVerdict gives you:

Everything else is your call, in one function:

httpserver.Wrap(mux, lim, httpserver.WithVerdict(
	func(r *http.Request, status int, h http.Header, d time.Duration) agilelimiter.Outcome {
		switch {
		case strings.HasPrefix(r.URL.Path, "/health"), r.URL.Path == "/metrics":
			return agilelimiter.Aborted // cheap by design, and noisy on a quiet service
		case r.Method == http.MethodOptions:
			return agilelimiter.Aborted // CORS preflight
		case httpserver.IsCacheHitHeader(h):
			return agilelimiter.Aborted
		default:
			return httpserver.DefaultVerdict(r, status, h, d)
		}
	}))

IsCacheHitHeader reads only headers your handler already wrote, so marking a cache hit is one line: w.Header().Set("X-Cache", "HIT") before the body. The same reasoning as the outbound side applies — with 95% of requests served from memory in 0.2ms, p50 tracks the cache, not the 5% that reach the database.

Two cases the default does not handle, because they need knowledge it does not have: health/metrics/preflight (exclude by path, as above) and long-lived streams (SSE, websocket, long poll — give them their own budget, or the median of every other request pays for them).

Everything above is one rule: a duration is evidence only while it measures the resource you are protecting. A cache hit, an instant 503, a /healthz probe and a circuit-breaker fallback all return in about a millisecond, and none of them say anything about remaining capacity. Fed into one p50 they make a saturated dependency look fast — and the controller's answer to "fast" is to raise the rate.

It cannot be fixed inside the controller: deciding what a response means needs the response. It cannot be a fixed set of built-in cases either, for two reasons:

• The list does not converge. Every year adds a proxy header, an async endpoint, a framework convention. One option per case is a treadmill.
• Every built-in case is wrong for somebody. A 202 is free if your queue is cheap and expensive if enqueueing is the bottleneck; a 4xx flood is noise from a client bug and signal from a broken auth service. The library does not know which company it is in.

So it is split into three layers:

The middle layer ships as a function rather than as options because a function is readable, testable as a pure function, and composable: your verdict can call DefaultVerdict for everything it does not care about.

These are properties of the control law and of the deployment, not of the signal:

• Capacity that moves faster than the controller. Steady-state ramping is UpStep per tick (5% by default), so a swing with a period of a few seconds outruns it.
• Cold start and recovery. A fresh limiter doubles while it has never seen congestion (SlowStartGain, default 2.0), so it finds the knee in log2(MaxRate/InitialRate) ticks — 6 instead of 80 for 10→500. Recovery from a deep dip is the same shape (RecoverRamp): once the observed latency is at or below half the tolerated window the multiplicative probe returns, because nothing about that reading is ambiguous. What is left to you is a limiter that restarts cold and has to re-learn a knee it already knew — persist the last good rate across restarts for that.
• A cache going cold. That is a step change; the backstop is single-flight and sizing MaxRate for a cold origin, not a better classifier.
• Many instances. Each process converges on its own; there is no shared view of a shared dependency.

• Baseline pollution. If the slow EWMA is allowed to rise freely, a long overload drags the baseline up to the congested latency, the tolerated window rises with it, and the controller freezes in the overloaded state. This implementation avoids it two ways: the baseline is updated asymmetrically (falls quickly, rises slowly) and its upward drift is hard-capped per tick by MaxBaselineRise, so a sustained overload cannot move the target at all.
• Fast failures look like health. A connection that is refused, a DNS failure, an open breaker: all return in microseconds, which the controller reads as a fast response and answers by raising the rate. Classify them as Unreachable (see above). This is the one mistake that makes the limiter actively harmful rather than merely suboptimal.
• Coordinated omission. In the measurement path, never stop sampling when the downstream slows. If your workers block and stop issuing calls, you go blind exactly when it matters most — a stalled caller produces no sample at all.
• Oscillation. If the rate rings, lengthen the half-lives, increase Deadband, shrink UpStep, or lower Gain.
• Observability. Export Rate(), Stats() (baseline, fast, the last tick's p50/p99) — and Stats().Samples above all: with no samples a step changes nothing, so a frozen rate and a steady one look identical. Alert on Samples == 0 for more than a few ticks; it is what tells you the controller went blind (low traffic, or a pipeline that stopped observing because every response was a cache hit).

Tests use a virtual clock and drive the control step directly, so they are fully deterministic:

go test ./...
go vet ./...
gofmt -l .

A runnable end-to-end demo lives in example/:

go run ./example            # outbound: pacing requests to a downstream
go run ./example/concurrency # inbound: capping concurrency on a service
go run ./example/httpserver  # inbound: a service shedding load under a swarm