v9.0.0
- Breaking: adds the public
BackendLockContentionErrorexception and stops
leaking rawredis.exceptions.LockError. Redis per-bucket lock contention now
surfaces as this library exception:await_for_capacity/wait_for_capacity
with no caller timeout retry through contention instead of raising (logging a
throttled warning), andconsume_capacity,refund_capacity,
set_max_capacity, and reconfiguration raiseBackendLockContentionError
(chained from the underlying redis error) on lock starvation or mid-operation
lock loss. Handlers that caughtredis.exceptions.LockErrormust catch
BackendLockContentionErrorinstead; seeMIGRATION.mdand
the per-bucket locking section indocs/operations.md. - Breaking:
RedisBackendBuilder.build()/SyncRedisBackendBuilder.build()
now raiseValueErrorat build time when any configured quota's
per_secondswindow is longer thanbucket_ttl_seconds. That combination
previously built without error but silently reset a drained long-window
quota back to full capacity once an idle gap outlived the TTL. Widen
bucket_ttl_seconds, or shorten the offending quota'sper_seconds, for any
configuration the check now rejects; seeMIGRATION.mdand
the key-TTL guidance indocs/operations.md. - Breaking:
OpenAIUsageCounter/get_encodingno longer guess a
tokenizer from a hardcoded model-family fallback table for models the
installedtiktokencannot resolve on its own (for example a very new model
release). They now raise aValueErrorwith upgrade/workaround guidance
instead of either a possibly-wrong guessed encoding or a rawKeyError
escaping fromtiktoken. Code that specifically caughtKeyErroraround
token counting must catchValueErrorinstead; upgradetiktokenor pass an
explicitget_encoding_functoOpenAIUsageCounterfor models it does not
yet recognize. SeeMIGRATION.md. - Breaking:
UsageQuotasno longer accepts the private
_allow_empty_quotasconstructor keyword; passing it now raisesTypeError
(unknown keyword argument) instead of silently building an empty quota set.
UsageQuotas([])still raises the sameValueErrorpointing you to
UsageQuotas.unlimited(), which remains the supported way to build an
explicit no-limit quota set. SeeMIGRATION.md. - Fixes Redis
await_for_capacity/wait_for_capacitywith a caller
timeout: lock contention now retries acquisition until the caller's
deadline instead of raisingTimeoutErrorafter
lock_blocking_timeout_seconds(default 5s).timeout=0still fails fast,
and the timeout message now names lock contention as the cause instead of
misleading capacity fields. See the per-bucket locking section in
docs/operations.md. - Fixes async
callback_timeoutso it returns at the deadline even when a
callback swallows cancellation, including when it is torn down via
GeneratorExit(for example an async generator that uses the limiter being
closed early). Previously such a callback could blockacquire_capacity/
refund_capacityfor its full runtime and, on a swallowed cancellation,
without ever logging the documented "callback exceeded timeout" warning; the
async path now abandons the callback the same way the synchronous path
already does, logging any error the callback raises afterward. See
docs/observability.md. - Fixes a
SyncRateLimiterdeadlock when aPerModelConfigGettercalls back
into the limiter (for exampleclear_unused_model_families) while shared
model-family validation is in progress; the internal validation lock is now
reentrant.acquire_capacity_for_requestalso now emits the same
RuntimeWarningasacquire_capacitywhen called from inside a running
event loop. - Fixes a spurious shutdown warning: closing a limiter with zero in-flight
reservations no longer logs a "reservations still outstanding" warning. - Fixes the Redis backend hard-failing every rate-limit operation whenever the
host's local clock lags behind the Redis server clock (for example an NTP
outage, a paused/resumed VM, or container clock drift). Refill math already
uses Redis server time exclusively, so a lagging local clock is harmless to
correctness; the library now detects a genuine server-side clock jump by
comparing consecutive RedisTIMEreadings against locally-elapsed
monotonic time instead of the local wall clock, and raises only on a real
forward jump between readings (the realistic trigger is a Sentinel/managed
failover to a clock-skewed primary). A large divergence between the Redis
server clock and the local wall clock now logs a one-time warning about
possible NTP trouble instead of raising. - Fixes two error messages: the
ValueErrorraised when usage exceeds a
bucket's max capacity during acquire now names the failing quota window
(for example "for the 60s window"), disambiguating cases where two windows
on the same metric share a limit value; andset_max_capacity's validation
now reports a dedicated "must be an int or float" message for wrong-typed
inputs instead of misleadingly reusing the finite/positive-value message. - Fixes cancellation-path capacity refunds that fail: they now log a warning
identifying the affected reservation instead of failing silently; the
original cancellation error still propagates and the reserved capacity
still recovers through normal refill. - Adds
RateLimiter.reserve()/SyncRateLimiter.reserve(): a context
manager over the acquire -> call -> refund cycle. It yields a handle with
.reservationand.set_actual_usage(), refunds the unused remainder on
normal exit (warning and conservatively refunding the full reserved usage if
set_actual_usagewas never called), and on an exception refunds with an
optionalusage_on_error(or conservatively) before re-raising the original
exception. If a non-criticalusage_on_errorrefund itself fails (for
example its metric keys do not match the reservation), the reservation
still falls back to the conservative refund instead of leaking as
in-flight; the failure is logged, and the caller's original exception
still propagates. See the README's "Reserve capacity around a call"
example. - Fixes
OpenAIUsageCounterundercounting Responses API requests that use
text={"format": {...}}for structured output: that config is now counted
by JSON-serializing it likeresponse_format/tools/functions, instead of
being walked as plain text fragments that dropped the JSON structural
tokens (previously undercounting affected requests by roughly 62%). - Adds a weekly
tokenizer-driftCI canary (no API key required) that checks
the OpenAI token counter against the latest unpinnedopenai/tiktoken
releases for newly-unresolvable models or untriaged request parameters. - Fixes the Redis ACL command list in
MIGRATION.mdand
docs/operations.md: it was missingPEXPIRE(used
by redis-py's lock extend/reacquire script) andMULTI/EXEC/
DISCARD(used by redis-py's transaction pipelines), so a user provisioned
strictly per the old list could pass an initial smoke test but fail under
ordinary multi-quota usage. - Expands documentation coverage: the Redis ACL command list in
MIGRATION.mdnow includesPTTL; its validation-error
guidance more precisely distinguishes pydanticValidationErrorfrom
CardinalityLimitExceededError; the README's OpenAI example sets an
explicit output-token budget and notes the zero-token refund on error as an
approximation;docs/configuration.mdgains a
"Choosing reservation sizes" subsection; and
docs/operations.mdgains an "Application-facing
errors" reference section coveringDuplicateRefundError,
UnknownReservationError,AcquireRefundFailedError, and
CardinalityLimitExceededError. - Adds a test-suite safety gate that refuses to run when
--redis-urlpoints at
a non-empty Redis database. The suite flushes that database around every test,
so it now aborts with an actionable message instead of silently wiping data;
setTOKEN_THROTTLE_TESTS_ALLOW_FLUSH=1to opt in to running against a
non-empty database. - Adds a test-suite thread-leak detector that fails the session if a test leaves
a non-daemon thread or a thread-pool worker alive after a short grace period,
catching cross-test interference that previously surfaced only as full-suite
flakiness. SetTOKEN_THROTTLE_THREAD_LEAK_MODE=reportto investigate a leak
without failing the run. - Adds a stdlib-only acquire-path benchmark harness under
benchmarks/
(uv run python -m benchmarks.run, ortask bench) that reports p50/p90/p99
and ops/sec for the memory and Redis backends across sync/async and
uncontended/contended workloads, with optional JSON output. It is not part of
the test suite and adds no runtime dependency; absolute numbers are
machine- and Redis-locality-dependent and meant to be read relatively. See
benchmarks/README.md. - Adds a weekly scheduled soak/stress workflow (
.github/workflows/soak.yml,
also runnable on demand) that repeats the concurrency stress suites many times
back to back, runs the property-based accounting suite, and runs a
tightened-timing conformance pass. It exists to catch load- and soak-class
regressions (contention and accounting bugs that only appear under sustained,
repeated load) that the single-pass PR CI does not exercise. It changes no
library behavior. - Widens the recommended pip install version bounds in the README from a
next-minor cap to a next-major cap (for example>=8.0.8,<9.0.0instead of
>=8.0.8,<8.1.0), so installs can pick up minor and patch releases within the
same major version without re-pinning. This reflects the project's semantic
versioning guarantee that no breaking changes ship within a major. - Removes a stray empty
__init__.pyfrom the repository root that was never
part of the publishedtoken_throttlepackage; it changes no library
behavior.
Full upgrade guidance for the breaking changes: MIGRATION.md