Skip to content

v0.5.0 — Adaptive

Pre-release
Pre-release

Choose a tag to compare

View all tags
@jamesgober jamesgober released this 06 Jun 00:36
· 9 commits to main since this release
v0.5.0
e4762fe

throttle-net v0.5.0 — Adaptive

Find the right limit without being told it. Every limiter so far needed you to know the downstream's capacity up front. v0.5.0 adds the clever one: an adaptive concurrency limiter that learns the capacity from observed outcomes — opening up while requests succeed, pulling back when they fail or slow — and never exceeds the hard ceiling. No breaking changes.

What is throttle-net?

A general-purpose outbound throttling and resilience library. Where rate-net protects your service from being overwhelmed (inbound), throttle-net protects your service from overwhelming the downstreams it calls — and from being banned by them. It paces outbound work, composes limits across dimensions and scopes, retries transient failures, fails fast when a dependency is sick, queues callers fairly when it is busy, and — as of this release — adapts its concurrency to what the downstream can actually take.

What's new in 0.5.0

AdaptiveLimiter — concurrency that learns

A token bucket needs a number. The adaptive limiter discovers one: it caps in-flight requests at a limit it adjusts from feedback, bounded by a floor and a ceiling. When requests succeed and the limit is saturated, it grows; when they fail or slow, it shrinks. The limit never exceeds the ceiling, so the adaptation can only ever be more conservative than your hard cap.

use throttle_net::{Aimd, AdaptiveLimiter};

let limiter = AdaptiveLimiter::builder()
    .floor(2)
    .ceiling(50)
    .initial(10)
    .build(Aimd::default()); // +1 on a saturated success, halve on failure

if let Some(permit) = limiter.try_acquire() {
    // ... call the downstream, then report the outcome ...
    if downstream_ok { permit.success() } else { permit.failure() }
}

Outcomes are reported through an AdaptivePermit: success() measures the round-trip time from acquisition, failure() signals a back-off, and dropping the permit unsettled counts as a failure — so an early return or a panic is treated conservatively. The waiting acquire().await blocks on a slot freeing rather than on a timer.

Two strategies, plus your own

  • Aimd — additive increase, multiplicative decrease. Probe upward gently on saturated successes, retreat sharply on failure. The classic congestion response.
  • Vegas — latency-based, after TCP Vegas. It estimates the downstream's queue depth from the round-trip time against the best (no-load) latency it has seen — limit * (rtt - min_rtt) / rtt — and grows when the estimate is small, shrinks when it is large. Latency degradation is felt before failures even start.
  • AdaptiveStrategy — implement adjust(current, in_flight, outcome) -> new_limit for a custom policy; the limiter clamps the result to [floor, ceiling].

The runnable examples/adaptive_concurrency.rs shows the limit climbing while a downstream is healthy, collapsing to the floor the moment it degrades, holding there through the outage, and climbing back on recovery — never crossing the ceiling.

Breaking changes

None. Everything in this release is additive, behind the adaptive feature.

Verification

Run on Windows x86_64, Rust stable 1.93.1; the same commands run in the CI matrix on Linux, macOS, and Windows across stable and MSRV 1.85:

cargo fmt --all -- --check
cargo clippy --all-targets --all-features -- -D warnings
cargo clippy --all-targets --no-default-features -- -D warnings
cargo test --all-features
cargo test --no-default-features
cargo doc --no-deps          # default features — now warning-free too
RUSTDOCFLAGS="-D warnings" cargo doc --no-deps --all-features
cargo deny check
cargo audit

All green. Counts at this tag (--all-features): 111 unit tests, 7 property tests, 3 circuit-breaker integration tests, 2 retry integration tests, 64 doctests. The adaptive limiter's exit criteria are covered: a run of failures drives the limit to the floor and a run of successes drives it to the ceiling (and no further); the AIMD and Vegas rules are unit-tested, and the in-flight count never exceeds the limit.

This release also fixes a documentation warning: the crate-level docs no longer link the feature-gated CircuitBreaker type, so cargo doc is clean on the default feature set, not only under --all-features.

What's next

  • v0.6.0 — Provider integration. Be a good API citizen out of the box: rate-limit header parsers (OpenAI/Anthropic/Cohere, AWS, GitHub, Stripe, generic RFC 6585), state synchronization from response headers to prevent client/server drift, and LLM provider presets behind provider-llm.

Installation

[dependencies]
throttle-net = "0.5"

# The adaptive and circuit-breaker limiters are behind feature flags:
throttle-net = { version = "0.5", features = ["adaptive", "circuit-breaker"] }

MSRV: Rust 1.85.

Documentation

Changelog: CHANGELOG.md.

Full Changelog: v0.4.0...v0.5.0

Assets 2
Loading