Skip to content

Technology Rate Limiting and Queues

mavrovde edited this page Nov 1, 2026 · 1 revision

Technology: Rate Limiting and Queues

Part of the Technology Evaluation. Where should rate-limit state, quotas and priority queues live?

Decision

In the API process at the first stage: token buckets per key, semaphores per priority tier under one database-concurrency cap, bounded queues with a deadline-aware admission decision. Quotas and usage are counted in memory and flushed to PostgreSQL in batches. Design details: Rate Limits and Priority.

Summary

Option Verdict Reason
In-process buckets, tier semaphores, adaptive limits Recommended exact on one instance; no network hop; admission before a connection is taken
PostgreSQL for metering, quotas and key state Viable for batched counters; rejected for per-request limiting a write per request would load the shared database
Redis / Valkey Rejected now; Valkey viable later nothing to share with one instance; standard choice once there are several
NATS JetStream Rejected now the API is synchronous; a PostgreSQL job table comes first if bulk exports arrive

In-process limits

  • What it is. In-memory token buckets per key and per tier; a semaphore per priority tier under a global database cap; bounded queues with a queue deadline; optionally a gradient or Vegas-style adaptive limit.
  • Strengths. Admission happens before a database connection is taken; shedding answers 429 with Retry-After; decisions take microseconds; deterministic tests with a fake clock.
  • Weaknesses. State is lost on restart (at most one window of extra allowance, accepted); not shared across replicas.
  • Verdict: Recommended. When a second replica arrives, move shared state to Valkey.

PostgreSQL-backed limits and quotas

  • What it is. A bucket row per key updated on every request, or usage counters flushed in batches.
  • Weaknesses for per-request use. Write amplification, WAL and vacuum load on a database shared with other services, and latency on every request.
  • Verdict: Viable only for quotas and metering (hourly counters flushed at an interval) and for persistent key state. Rejected for per-request rate limiting.

Redis or Valkey

  • What it is. An in-memory store with atomic counters and scripted token buckets, shared across replicas.
  • Strengths. The standard way to share limits between replicas; sub-millisecond.
  • Weaknesses. Another stateful service with persistence and backups, and nothing to share while there is one instance.
  • Licence note. Valkey is BSD-3. Redis has changed licences in recent versions (now offering AGPLv3 beside source-available licences).
  • Verdict: Rejected now; Valkey viable later with several API replicas.

NATS JetStream

  • A messaging system with persistence and a key-value store.
  • The API is request/response, so there is nothing to queue across processes. If asynchronous bulk exports are added, a PostgreSQL job table (SKIP LOCKED) comes first.
  • Licence. Apache-2.0. Verdict: Rejected now.

Sources

Home

About

Design

Data Model

Technical specification

Testing

Engineering

Technology choice

Reference

Clone this wiki locally