v0.5.0-beta

Spooky 0.5.0-beta — Distributed Quota & Advanced Rate Limiting

Minor release covering all changes since 0.4.3-beta. Rate limiting moves from per-node token buckets to cluster-wide budgets backed by a shared counter store, with composite identity and dual-window contracts.

Added

• Distributed quota policies — resilience.quota enforces cluster-wide request budgets at admission, before forwarding. Disabled by default, so an existing config engages none of it.
• Composite policies — resilience.quota.policies[], each with a name, optional route_allowlist, a selector, and one or both of a burst and sustained window (requests, window_secs). A policy's identity is the composite of its selected dimensions, so one policy can budget per tenant-and-route without collapsing distinct callers into a shared counter.
• Identity selection — selector.route (bool), plus tenant, token, and client, each resolved from a request key. tenant/token accept header:*, cookie:*, query:*, bearer_token; client adds peer_ip and client_ip.
• Redis counter store — backend.kind: redis with url, key_prefix (spooky:quota), connect_timeout_ms (250), command_timeout_ms (100), max_inflight (1024). Burst and sustained windows are incremented and tested in one atomic Lua evaluation, so a request never charges one window and abandons the other.
• In-memory backend — kind: in_memory (default) for single-node deployments and tests, sharing the fixed-window semantics of the Redis path.
• Bounded local fallback — local_fallback (Redis only) with key_prefix and a required max_entries. Engages only for timeouts and unavailability, never for protocol, config, or logic errors.
• Shadow mode — enforcement: shadow records what would have been denied without blocking, so a policy can be sized against live traffic before it turns requests away. enforce is the default.
• Backend failure policy — backend_failure_policy: fail_closed (default) rejects with 503 when the counter store is unreachable; fail_open admits.
• Metrics — spooky_quota_policy_outcomes_total{policy,decision,reason,selector_dimensions,backend_mode} and spooky_quota_backend_health_total{backend_mode,reason}. Decisions are allowed, denied, shadow_denied, failed_open, failed_closed, not_applied. Degraded operation appears in backend_mode as <kind>_local_fallback_<reason>, so running on fallback counters is distinguishable from running on the real backend.
• Runtime introspection — /admin/runtime gained a quota block: enabled, enforcement, backend_failure_policy, active_backend, a backend_status object (availability, degraded, health_reason, last_observed_at_unix_ms, recent_errors[]), and the resolved policies[] with selectors and windows.

Changed

• Scoped rate limiting was rebuilt on the quota pipeline's evaluation contract. Buckets now evaluate a request cost and return remaining tokens with a retry_after computed from the token deficit and refill rate, replacing the previous boolean consume. Existing resilience.scoped_rate_limits config is unchanged and behaves as before.
• A poisoned scoped rate-limit bucket lock is reported as backend unavailability rather than an implicit allow inside the bucket layer. The legacy path still resolves that to fail-open.

Fixed

• Every route-matching quota policy is evaluated, not just the first. A route matched by more than one policy previously consumed only the first policy's budget — later policies validated at startup and appeared in the runtime snapshot while enforcing nothing, so a narrow policy layered after a broad one was silently dead. A denial now short-circuits so remaining budgets aren't charged for a request about to be rejected; shadow denials keep evaluating so every policy records its outcome.
• A request missing the identity a later policy selects on is denied with selector_identity_missing instead of being admitted by an earlier, broader policy that matched first.

Security

• Quota enforcement is fail-closed by default. A counter-store outage rejects with 503 rather than admitting unmetered traffic, and local fallback is scoped to outages only — a misconfigured or protocol-mismatched backend fails hard instead of quietly enforcing a weaker, node-local budget.
• Startup rejects incoherent policy: a burst window not shorter than its sustained window, a selector with no dimensions, zero requests or window_secs, duplicate policy names, duplicate selector/window fingerprints, one request key bound to two identity dimensions, local_fallback against a non-Redis backend, and enabled: true with no policies.

Upgrade Notes

Nothing breaks. resilience.quota defaults to enabled: false with an empty policies list, so an existing config runs unchanged. Scoped rate limiting keeps working and is not deprecated.

⚠️ backend_failure_policy defaults to fail_closed. It stays dormant until you set enabled: true — but once you do, a node that cannot reach Redis rejects with 503 rather than admitting unmetered traffic. Use fail_open if availability matters more than the budget.

⚠️ One-way config compatibility: 0.5.0 reads a 0.4.3 config, but 0.4.3 rejects any resilience.quota block. Roll the binary back before the config.

⚠️ /admin/runtime gained a top-level quota object and /metrics gained two counter families. Automation asserting on exact response shape or metric-family sets should be updated.

Migration: start with enforcement: shadow and a generous budget, watch spooky_quota_policy_outcomes_total{decision="shadow_denied"} to size the policy against real traffic, then switch to enforce.

Known Limitations

• Fixed-window counters only — no sliding window or leaky bucket
• Two windows per policy (burst + sustained); no arbitrary window stacking
• Redis is the only distributed backend — no Memcached, DynamoDB, or gossip-based counting
• No Redis Cluster or Sentinel topology support; a single endpoint per backend
• Local fallback is node-local and bounded — during a Redis outage the effective budget is per-node, not cluster-wide
• Quota is evaluated at admission only; no mid-stream or response-size accounting
• No Retry-After or quota headers on denial responses
• Policies are config-driven — no runtime quota API and no per-tenant dynamic budgets
• Pre-GA — soak test before broad rollout