Repository navigation
Technology Rate Limiting and Queues
mavrovde edited this page Nov 1, 2026
·
1 revision
Part of the Technology Evaluation. Where should rate-limit state, quotas and priority queues live?
In the API process at the first stage: token buckets per key, semaphores per priority tier under one database-concurrency cap, bounded queues with a deadline-aware admission decision. Quotas and usage are counted in memory and flushed to PostgreSQL in batches. Design details: Rate Limits and Priority.
| Option | Verdict | Reason |
|---|---|---|
| In-process buckets, tier semaphores, adaptive limits | Recommended | exact on one instance; no network hop; admission before a connection is taken |
| PostgreSQL for metering, quotas and key state | Viable for batched counters; rejected for per-request limiting | a write per request would load the shared database |
| Redis / Valkey | Rejected now; Valkey viable later | nothing to share with one instance; standard choice once there are several |
| NATS JetStream | Rejected now | the API is synchronous; a PostgreSQL job table comes first if bulk exports arrive |
- What it is. In-memory token buckets per key and per tier; a semaphore per priority tier under a global database cap; bounded queues with a queue deadline; optionally a gradient or Vegas-style adaptive limit.
-
Strengths. Admission happens before a database connection is taken; shedding answers
429withRetry-After; decisions take microseconds; deterministic tests with a fake clock. - Weaknesses. State is lost on restart (at most one window of extra allowance, accepted); not shared across replicas.
- Verdict: Recommended. When a second replica arrives, move shared state to Valkey.
- What it is. A bucket row per key updated on every request, or usage counters flushed in batches.
- Weaknesses for per-request use. Write amplification, WAL and vacuum load on a database shared with other services, and latency on every request.
- Verdict: Viable only for quotas and metering (hourly counters flushed at an interval) and for persistent key state. Rejected for per-request rate limiting.
- What it is. An in-memory store with atomic counters and scripted token buckets, shared across replicas.
- Strengths. The standard way to share limits between replicas; sub-millisecond.
- Weaknesses. Another stateful service with persistence and backups, and nothing to share while there is one instance.
- Licence note. Valkey is BSD-3. Redis has changed licences in recent versions (now offering AGPLv3 beside source-available licences).
- Verdict: Rejected now; Valkey viable later with several API replicas.
- A messaging system with persistence and a key-value store.
- The API is request/response, so there is nothing to queue across processes. If asynchronous bulk exports are added, a PostgreSQL job table (
SKIP LOCKED) comes first. - Licence. Apache-2.0. Verdict: Rejected now.
- Netflix, performance under load (adaptive concurrency limits): https://netflixtechblog.medium.com/performance-under-load-3e6fa9a60581; https://github.com/Netflix/concurrency-limits
- PostgreSQL advisory locks: https://www.postgresql.org/docs/16/explicit-locking.html#ADVISORY-LOCKS
- Valkey: https://valkey.io/; licensing overview: https://blog.elest.io/valkey-vs-redis-vs-keydb-which-in-memory-store-after-the-license-change/
- NATS: https://docs.nats.io/
- RateLimit header fields draft: https://datatracker.ietf.org/doc/draft-ietf-httpapi-ratelimit-headers/
ViaFrei API: a public, keyed, read-only REST API to live German mobility data. Design phase; nothing on these pages is a measurement of a running service. Data on viafrei.de and in every API answer is attributed to its publishers under their licences.
About
Design
- Domain Model
- API Design
- Architecture Overview
- Security and Access Keys
- Rate Limits and Priority
- Performance and Caching
- Configuration
- Observability
Data Model
Technical specification
-
Technical Specification
- Interface
- Scope, Conventions and References
- Protocol
- Common Envelope
- Resource Model
- Endpoints, Places
- Endpoints, Roads
- Endpoints, Energy and Parking
- Endpoints, Public Transport and Accessibility
- Endpoints, Weather Risk, Driving Rules and Composed Features
- Service Endpoints
- Errors
- Authentication and Authorisation
- Rate Limiting, Quotas and Priority
- Data Licensing and Attribution
- Compatibility and Versioning
- OpenAPI Skeleton
- Quality
- Delivery and operations
- Interface
Testing
- Testing Strategy
- Testing and Quality (design summary)
Engineering
Technology choice
- Technology Evaluation
- Architecture: Go
- Architecture: Rust
- Architecture: TypeScript
- Stack Comparison
- Benchmark Plan
Reference