v0.6.0-beta

Impulse- v0.6.0-beta

Added

• Secret references — reference sensitive fields via literal: or file:// instead of plaintext, with a new secrets: provider block.
• Upstream mTLS — present a client certificate to backends via tls.client_certificate / tls.client_key (or their _ref variants).
• Secret & cert lifecycle observability — new Prometheus metrics, control-API fields, and audit events for secret resolution and certificate expiry.

Changed

• Project rebranded from Spooky to Impulse (binaries, packaging, docs, config paths). Naming change only — no behavior change.

Compatibility

• Fully backward compatible — all new fields are optional, existing configs work unchanged.
• Upgrading from a spooky-named install should account for new impulse binary/paths/systemd unit names.

Full changelog: CHANGELOG.md

v0.5.1-beta

Spooky 0.5.1-beta — Observability Operator Bundle

Patch release covering all changes since 0.5.0-beta. Ships a packaged operator observability experience — dashboards, alerts, and SLOs on top of the existing metrics — plus canonical audit correlation, hardened default packaging, and an OTLP startup fix.

Added

• Packaged operator observability bundle under deploy/observability/ — Prometheus recording rules, production alert rules, an SLO package, and six Grafana dashboards (edge traffic, admission overload, backend health, retries/hedges, TLS certificates, control-plane activity), so operators get a working dashboard/alert set instead of assembling one from raw metrics.
• Canonical correlation fields on admin audit events — event_id, schema_version, request_id, trace_id, span_id, listener — plus a stable failure_class (authentication, authorization, source_policy, request_validation, runtime_config, runtime_state, listener_tls, watchdog) on non-success events, so failures can be grouped and correlated across metrics/logs/traces without parsing free-form reason strings.
• GET /admin/runtime and the runtime history endpoints gained an observability block — contract version, audit schema version, current generation, backend and quota backend health summaries, recent tracked admin actions, and repository-relative dashboard/documentation references. This is the one canonical entry point into the packaged bundle for operators and automation.
• h3_client now accepts --method (default GET) and repeatable --header name=value (including pseudo-headers like :protocol), for exercising non-GET and header-sensitive traffic in the observability lab.
• Documentation — docs/operations/observability-bundle.md covers the shipped dashboards, alerts, SLOs, and incident-correlation workflow; docs/architecture/observability-contract.md and docs/operations/control-plane.md document the audit schema and the new runtime observability block.

Changed

• The Debian package's default config.yaml now ships the control API disabled (observability.control_api.enabled: false) with a placeholder-token comment block, instead of enabled with a literal replace-with-strong-token credential — a fresh install no longer boots with a live, weakly-credentialed admin surface.
• The systemd unit restarts with Restart=always instead of Restart=on-failure (the watchdog's own clean-exit drain would otherwise not be restarted), adds StartLimitIntervalSec=60/StartLimitBurst=5 to guard against crash-looping, raises LimitNOFILE=65535 for QUIC-scale file descriptor use, and tightens sandboxing (ProtectKernelTunables, ProtectKernelModules, ProtectControlGroups, RestrictAddressFamilies, RestrictNamespaces, RestrictSUIDSGID, LockPersonality). /etc/spooky is no longer in ReadWritePaths — only /var/log/spooky stays writable.

Fixed

• OTLP tracing is now initialized inside the Tokio runtime instead of before it. The OTLP tonic exporter spawns a background task while building its gRPC channel, which previously panicked with "there is no reactor running" whenever tracing was enabled.
• Docker image build and clippy warnings introduced during observability bundle development.

Upgrade Notes

Nothing breaks. The new observability block on /admin/runtime and the audit event fields are additive JSON — existing consumers that don't read them are unaffected.

⚠️ Fresh installs/packages built from this version ship the control API disabled by default. If you rely on the Debian package's default config enabling it out of the box, you must now explicitly set a real bearer token and flip enabled: true.

Existing deployed configs and units are not modified in place by this release — these changes affect newly generated packages only.

Known Limitations

• Observability contract and audit schema are both fixed at v1 — no versioned migration path yet if the schema needs to change.
• Dashboard/documentation references in /admin/runtime are repository asset paths, not resolvable URLs — operators must map them into their own Grafana/runbook tooling.
• Pre-GA — soak test before broad rollout.

v0.5.0-beta

Spooky 0.5.0-beta — Distributed Quota & Advanced Rate Limiting

Minor release covering all changes since 0.4.3-beta. Rate limiting moves from per-node token buckets to cluster-wide budgets backed by a shared counter store, with composite identity and dual-window contracts.

Added

• Distributed quota policies — resilience.quota enforces cluster-wide request budgets at admission, before forwarding. Disabled by default, so an existing config engages none of it.
• Composite policies — resilience.quota.policies[], each with a name, optional route_allowlist, a selector, and one or both of a burst and sustained window (requests, window_secs). A policy's identity is the composite of its selected dimensions, so one policy can budget per tenant-and-route without collapsing distinct callers into a shared counter.
• Identity selection — selector.route (bool), plus tenant, token, and client, each resolved from a request key. tenant/token accept header:*, cookie:*, query:*, bearer_token; client adds peer_ip and client_ip.
• Redis counter store — backend.kind: redis with url, key_prefix (spooky:quota), connect_timeout_ms (250), command_timeout_ms (100), max_inflight (1024). Burst and sustained windows are incremented and tested in one atomic Lua evaluation, so a request never charges one window and abandons the other.
• In-memory backend — kind: in_memory (default) for single-node deployments and tests, sharing the fixed-window semantics of the Redis path.
• Bounded local fallback — local_fallback (Redis only) with key_prefix and a required max_entries. Engages only for timeouts and unavailability, never for protocol, config, or logic errors.
• Shadow mode — enforcement: shadow records what would have been denied without blocking, so a policy can be sized against live traffic before it turns requests away. enforce is the default.
• Backend failure policy — backend_failure_policy: fail_closed (default) rejects with 503 when the counter store is unreachable; fail_open admits.
• Metrics — spooky_quota_policy_outcomes_total{policy,decision,reason,selector_dimensions,backend_mode} and spooky_quota_backend_health_total{backend_mode,reason}. Decisions are allowed, denied, shadow_denied, failed_open, failed_closed, not_applied. Degraded operation appears in backend_mode as <kind>_local_fallback_<reason>, so running on fallback counters is distinguishable from running on the real backend.
• Runtime introspection — /admin/runtime gained a quota block: enabled, enforcement, backend_failure_policy, active_backend, a backend_status object (availability, degraded, health_reason, last_observed_at_unix_ms, recent_errors[]), and the resolved policies[] with selectors and windows.

Changed

• Scoped rate limiting was rebuilt on the quota pipeline's evaluation contract. Buckets now evaluate a request cost and return remaining tokens with a retry_after computed from the token deficit and refill rate, replacing the previous boolean consume. Existing resilience.scoped_rate_limits config is unchanged and behaves as before.
• A poisoned scoped rate-limit bucket lock is reported as backend unavailability rather than an implicit allow inside the bucket layer. The legacy path still resolves that to fail-open.

Fixed

• Every route-matching quota policy is evaluated, not just the first. A route matched by more than one policy previously consumed only the first policy's budget — later policies validated at startup and appeared in the runtime snapshot while enforcing nothing, so a narrow policy layered after a broad one was silently dead. A denial now short-circuits so remaining budgets aren't charged for a request about to be rejected; shadow denials keep evaluating so every policy records its outcome.
• A request missing the identity a later policy selects on is denied with selector_identity_missing instead of being admitted by an earlier, broader policy that matched first.

Security

• Quota enforcement is fail-closed by default. A counter-store outage rejects with 503 rather than admitting unmetered traffic, and local fallback is scoped to outages only — a misconfigured or protocol-mismatched backend fails hard instead of quietly enforcing a weaker, node-local budget.
• Startup rejects incoherent policy: a burst window not shorter than its sustained window, a selector with no dimensions, zero requests or window_secs, duplicate policy names, duplicate selector/window fingerprints, one request key bound to two identity dimensions, local_fallback against a non-Redis backend, and enabled: true with no policies.

Upgrade Notes

Nothing breaks. resilience.quota defaults to enabled: false with an empty policies list, so an existing config runs unchanged. Scoped rate limiting keeps working and is not deprecated.

⚠️ backend_failure_policy defaults to fail_closed. It stays dormant until you set enabled: true — but once you do, a node that cannot reach Redis rejects with 503 rather than admitting unmetered traffic. Use fail_open if availability matters more than the budget.

⚠️ One-way config compatibility: 0.5.0 reads a 0.4.3 config, but 0.4.3 rejects any resilience.quota block. Roll the binary back before the config.

⚠️ /admin/runtime gained a top-level quota object and /metrics gained two counter families. Automation asserting on exact response shape or metric-family sets should be updated.

Migration: start with enforcement: shadow and a generous budget, watch spooky_quota_policy_outcomes_total{decision="shadow_denied"} to size the policy against real traffic, then switch to enforce.

Known Limitations

• Fixed-window counters only — no sliding window or leaky bucket
• Two windows per policy (burst + sustained); no arbitrary window stacking
• Redis is the only distributed backend — no Memcached, DynamoDB, or gossip-based counting
• No Redis Cluster or Sentinel topology support; a single endpoint per backend
• Local fallback is node-local and bounded — during a Redis outage the effective budget is per-node, not cluster-wide
• Quota is evaluated at admission only; no mid-stream or response-size accounting
• No Retry-After or quota headers on denial responses
• Policies are config-driven — no runtime quota API and no per-tenant dynamic budgets
• Pre-GA — soak test before broad rollout

v0.4.3-beta

Spooky 0.4.3-beta — Asymmetric JWT & JWKS

Patch release since 0.4.2-beta. JWT validation moves beyond a shared secret to public-key verification, with keys pinned locally or fetched from an issuer's JWKS endpoint.

Added

• RS256 and ES256 validation — joins HS256 on the local, synchronous request path. Signature checking never makes a network call.
• Static public keys — auth.jwt.static_keys[] as PEM (kind: pem) or JWK (kind: jwk), each with a kid and alg.
• JWKS key sources — auth.jwt.jwks_url fetches keys into a background-refreshed cache. Tune with jwks_refresh_interval_secs (300), jwks_cache_ttl_secs (900), jwks_stale_if_error_secs (3600), jwks_request_timeout_ms (2000). An unknown kid triggers a rate-limited refresh, not a per-request fetch.
• Explicit cache states — never_fetched, fresh, stale, refresh_failed_retained, quarantined_retained, empty_unusable. The *_retained states still serve last-known-good keys; only empty_unusable rejects everything.
• Startup gating — jwks_startup_behavior: require_ready (default) fails startup rather than admitting traffic with no keys; reload activation preflights the same condition. allow_degraded boots and retries in the background.
• Metrics — spooky_jwt_validation_failures_total{reason}, spooky_jwt_algorithm_rejections_total{algorithm}, and per-source spooky_jwks_* counters and gauges (refresh outcomes, age, state, key count, refresh timestamps).
• Runtime introspection — /admin/runtime gained jwks.sources[] and per-upstream JWT provider state under auth.providers[].

Changed

• JWT validation restructured into a canonical pipeline — JOSE parse → algorithm policy → key resolution → signature → claims — with a stable rejection vocabulary (algorithm_not_allowed, missing_verification_key, key_source_unavailable, ambiguous_verification_key, issuer_mismatch, audience_mismatch, token_expired).
• Key type is re-checked at verification time, so an asymmetric key can never satisfy an HS256 token and vice versa. alg: none never maps to a verification mode.
• allowed_algorithms and require_kid are explicit policy, configured independently of the key material present.

Fixed

• Retained cache entries survive the full TTL window instead of being dropped, so a transient issuer outage no longer causes an immediate auth outage.
• Two JWKS sources sharing a URL but differing in policy merge deterministically instead of depending on iteration order.
• Tokens with no kid are accepted only when exactly one algorithm-compatible key survives filtering — multiple candidates are rejected as ambiguous rather than guessed at.

Security

• JWKS telemetry is labelled by an opaque jwks_source_id, never the URL, and endpoints in logs and /admin/runtime are stripped of query strings. Credentials in a JWKS URL cannot leak through /metrics, logs, or the snapshot.
• RSA keys shorter than 2048 bits are rejected, static or JWKS-published.
• A refresh failure never widens access — last-known-good keys validate until jwks_stale_if_error_secs elapses, then requests are rejected, not admitted.
• Startup rejects incoherent policy: secret without HS256 in the allowlist, HS256 with an empty secret, or no key material at all.

Upgrade Notes

Nothing breaks. allowed_algorithms defaults to HS256 only, static_keys is empty, and jwks_url is unset, so an existing config behaves exactly as it did.

⚠️ jwks_startup_behavior defaults to require_ready. It stays dormant until you set jwks_url — but once you do, a node whose issuer is unreachable fails to boot rather than starting degraded. Use allow_degraded if you'd rather come up without keys.

⚠️ One-way config compatibility: 0.4.3 reads a 0.4.2 config, but 0.4.2 rejects static_keys, jwks_url, allowed_algorithms, require_kid, issuers, audiences, and the jwks_* fields. Roll the binary back before the config.

Migration: add allowed_algorithms and key material alongside your existing secret, watch spooky_jwt_algorithm_rejections_total while both are live, then drop HS256 and clear the secret.

Known Limitations

• No RS384/RS512, PS*, EdDSA, or non-P-256 curves
• JWKS is direct-URL only — no discovery-document resolution
• The cache is process-local, not shared across instances
• A source removed or repointed by reload isn't evicted for the process lifetime
• A token matching both a static key and a JWKS key is rejected as ambiguous
• Pre-GA — soak test before broad rollout

v0.4.2-beta

Spooky 0.4.2-beta — Admin-Plane Security & Audit

Patch release since 0.4.1-beta. The control API moves from one shared bearer
token to roles, client certs, and an audit trail.

Added

• Admin roles — viewer reads, operator mutates, admin restarts.
  Configure under observability.control_api.auth.bearer_tokens[].
• Control API mTLS — tls.client_auth.mode: disabled, optional, or
  required. Scoped to the control API; downstream client auth is untouched.
  Roles can come from a cert subject attribute via auth.identity_source.
• Audit stream — JSON record per privileged action (actor, action, target,
  generation, result, reason, timestamp). Covers auth success/failure, denials,
  reads, and attempt/result pairs for mutations. Writes to a dedicated
  spooky.control_api.audit log target or its own file.
• IP allowlisting — CIDR gate under ip_allowlist, checked before
  credentials, using the TCP peer address.

Changed

• Under-scoped callers now get 403 with reason: insufficient_role. Missing
  or bad credentials still get 401. Denial bodies gained reason and
  required_role.
• The control API builds its own TLS config instead of borrowing the primary
  listener's, and advertises only http/1.1 in ALPN.
• Plain-text log lines now include the log target. JSON output unchanged.

Security

• Token matching is constant-time across the full set.
• Tokens never appear in /admin/runtime and are redacted in debug output.
• Startup rejects: mTLS without CA material, optional mTLS as the only auth,
  inverted role ordering, file audit sink with no path, malformed CIDRs.

Upgrade Notes

Nothing breaks. New settings default to off — mTLS disabled, no static tokens,
no audit, empty allowlist. auth_token still works as an admin identity.

⚠️ Once you add role-scoped tokens, automation that treats every rejection as
401 will miss the new 403.

⚠️ One-way config compatibility: 0.4.2 reads a 0.4.1 config, but 0.4.1 rejects
the new admin-plane fields. Roll the binary back before the config.

Migration: add role-scoped tokens next to auth_token, turn on audit to see who
still uses the legacy token, then drop it.

Known Limitations

• No CRL/OCSP — a stolen client cert works until its CA rotates
• Bearer and cert roles are unioned: a viewer token with an admin cert
  gets admin
• Audit is per-process and local — no aggregation, no tamper-evidence
• trust_proxy_headers is accepted but ignored
• Pre-GA — soak test before broad rollout

v0.4.1-beta

Spooky 0.4.1-beta — Staged Activation & Rollback

Patch release covering all changes since 0.4.0-beta.

Added

• Staged runtime activation — POST /admin/runtime/validate, /preview, and /activate split config changes into plan-then-commit. Validate and preview read, validate, compatibility-gate, and produce a per-domain diff without touching the running runtime; activate commits it as an explicit transaction returning the diff, structured rejections, and a history entry.
• Rollback by generation id — POST /admin/runtime/rollback restores a previously retained runtime generation. Rollback moves forward to a new generation number while restoring the target's content and its original config source.
• Runtime generation history — GET /admin/runtime/history and /history/{generation} expose the operation log alongside retained-generation records (status, rollback_candidate, has_bundle, and a note explaining failed staged prepares), so operators can discover which generations are valid rollback targets.
• Alternate config source activation — activation accepts a config_path in the request body, enabling canary workflows that switch between config files without a restart.
• Optimistic concurrency — expected_generation on activate and expected_active_generation on rollback reject with 409 when the runtime has moved underneath the caller.
• Runtime activation observability, and canonical rejection reasons normalized across operator surfaces.

Fixed

• Activating an alternate config_path now makes that file the active runtime source. Previously the runtime kept advertising the startup path, so a later reload-without-body silently re-read the wrong file and the control-plane view misreported the active config source.
• Generation history records only the operation actually requested. Previously every staged call wrote both a validate and a synthetic preview entry, so a single activation consumed three history slots and the audit trail claimed preview activity that never occurred.
• Retained-generation state is now surfaced by the history API. Failed staged prepares were tracked internally but never exposed, leaving operators unable to see why a candidate never committed.

Changed

• Operator conflicts are now classified instead of collapsing into 500. Stale-generation and restart-required conflicts return 409; an unknown rollback target returns 404; an invalid config returns 400. Genuine runtime faults — resource preparation failure, or a failed runtime swap — still return 500.
• POST /admin/runtime/reload now defaults to the currently active runtime config source rather than always re-reading the startup path. It remains supported as a shortcut over the same staged pipeline as /activate.
• Config defaults co-located with the types they belong to, and serde default patterns normalized.
• Test suite restructured by behavioral domain, with shared integration harnesses for the request path, bootstrap/QUIC parity, runtime swap, and backend lifecycle.
• Documentation updated for the staged control-plane surface: control API reference, API overview, production readiness, migration guide, and roadmap.

Upgrade Notes

⚠️ Behavioral change for automation matching on HTTP status codes. Recoverable control-plane conflicts that previously returned 500 Internal Server Error now return 409 Conflict or 404 Not Found. Tooling that treats any non-2xx from /admin/runtime/reload as an infrastructure failure should be updated to distinguish retryable operator conflicts from genuine faults.

Note that validate and preview return 200 even when a candidate is rejected — the status code reflects the planning request, not the verdict. Read candidate_status and rejected_changes.

Known Limitations

• No dynamic backend discovery (static config only)
• Listener bind address and protocol changes still require a restart
• Retained generations are capped at a fixed bound; rollback reaches recent generations, not arbitrarily old ones
• Retention is not yet configurable, and there is no automatic rollback on post-activation health regression
• Pre-GA — extended soak testing recommended before broad production rollout

v0.4.0-beta

Spooky 0.4.0-beta — Runtime Lifecycle & Canonical Observability

Minor release covering all changes since 0.3.1-beta.

Added

• Deterministic runtime lifecycle state machine — reload commits, drain, and shutdown now flow through explicit lifecycle states rather than ad-hoc paths, making generation swaps predictable under load.
• Canonical observability vocabulary — admission outcomes, local reason enums, overload metric labels, and backend health-failure reasons all map to one canonical set, with bounded connect-attempt label cardinality to keep metric series stable.
• Unified backend refresh classification — DNS/health refreshes report explicit traffic-continuity outcomes, and the canonical backend health-failure reason is surfaced in the control-plane snapshot.
• Structured reload rejections — operator-facing reload preflight failures now carry structured messages, backed by a typed operational-policy vocabulary for runtime ownership and reload rules.
• Live log-level reload — log.level changes apply on config reload via a runtime set_log_level, no restart required.

Fixed

• Process-shared watchdog and DNS resolver are now carried across reload instead of being rebuilt, preserving state through generation swaps.
• Bundle-lock panic and control-API unreachable paths replaced with fail-safe recovery; backend client-rotation failure is now an explicit, operator-visible outcome.
• Pre-response requests still reading the body now time out under the request-body bucket.
• Restored stateful round-robin selection, failover availability on circuit-open retries, and streaming guardrail failure semantics.
• WebSocket upgrade headers preserved in bootstrap 101 responses.
• Upstreams iterated in name order so route conflicts report deterministically.

Changed

• Public API surface locked down across all crates — crate-internal items demoted, unreachable_pub enforced, dead code removed, and public runtime/listener/connection surfaces documented.
• Documentation realigned to the post-refactor architecture (edge runtime ownership, runtime generation & backend lifecycle, request lifecycle, transport boundaries, observability, and reload behavior).
• Test suite hardened with per-contract cases asserting QUIC/bootstrap parity, metric label vocabularies, and retry/hedge accounting.

Known Limitations

• No dynamic backend discovery (static config only)
• Listener bind address and protocol changes still require a restart
• Pre-GA — extended soak testing recommended before broad production rollout

v0.3.1-beta

Spooky 0.3.1-beta — Full Config Hot Reload

Minor release covering all changes since 0.3.0-beta.

Added

• Full config hot reload — POST /admin/runtime/reload atomically swaps the runtime bundle without restarting the process or dropping active connections. Routes, backends, upstream pools, and resilience settings all take effect immediately.
• Listener group reconciliation on reload — new listener groups defined in the incoming config are started live; removed groups are retired gracefully with a configurable drain timeout.
• Live admin endpoint rebinding — control API and metrics endpoint addresses update in place on reload; no restart required for admin plane address changes.
• RuntimeTaskRegistry — generation-aware background task tracking ensures retired tasks from the previous generation drain cleanly before the new generation takes full ownership.

Fixed

• Hot reload now rejects configs that remove a listener or change its bind address, returning 409 Conflict with a clear error instead of partially applying the change.
• Hot reload now rejects changes to startup-owned settings (log level, thread counts, listen protocol/address), returning 409 Conflict — reload refused, restart required.
• Control API and cert reload endpoints now target the live runtime bundle after a hot reload, not the original startup bundle.
• Metrics endpoint, bootstrap listener, and control API settings are now refreshed from the live runtime bundle on each generation swap.

Known Limitations

• No dynamic backend discovery (static config only)
• Listener bind address and protocol changes still require a restart
• Pre-GA — extended soak testing recommended before broad production rollout

v0.3.0-beta

Spooky 0.3.0-beta — HTTP/1.1 Upstream Support

Minor release covering all changes since 0.2.1-beta.

Added

• HTTP/1.1 upstream transport — http:// backends are now forwarded over a pooled HTTP/1.1 connection. No backend changes required; use http://host:port in your config and Spooky selects the right transport automatically.
• Scheme-aware dispatch — backend URL scheme determines transport: https:// backends use HTTP/2, http:// backends use HTTP/1.1. Mixed deployments are supported within the same upstream pool.
• DNS refresh for H1 backends — when backend hostnames re-resolve, HTTP/1.1 pooled connections rotate alongside HTTP/2 connections.
• H1 health checks — active health probes now use the same transport as the target backend scheme.
• TE: trailers header forwarded on H1 upstream requests to preserve trailer semantics.

Fixed

• Config validator no longer rejects http:// upstreams due to unused TLS CA path checks — HTTP-only configs boot cleanly without requiring any TLS material for those backends.

Known Limitations

• No dynamic backend discovery (static config only)
• No configuration hot reload (restart to apply config changes)
• Pre-GA — extended soak testing recommended before broad production rollout

v0.2.1-beta

Spooky 0.2.1-beta — Diagnostics & Error Clarity

Patch release covering all changes since 0.2.0-beta.

Fixed

• ProxyError::Pool was displayed as "transport error: ..." — now correctly shows "pool error: ..." so pool and transport failures are distinguishable in logs
• Watchdog mutex poison is now logged and recovered — coordinator remains operational after a worker panic instead of silently skipping state updates
• OTLP tracing endpoint is now configurable via OTEL_EXPORTER_OTLP_TRACES_ENDPOINT or OTEL_EXPORTER_OTLP_ENDPOINT environment variables; resolved source is logged at startup
• validate() returns a structured ValidationError instead of bool, making the first validation failure available to callers as a typed error
• take_validation_error now clears the slot on read, preventing stale validation errors from leaking across test cases
• Logger fallback error messages now include both the file path and the parent directory when log directory creation fails

Changed

• validate_config call site updated to handle the new Result return type from the validator