Skip to content

v0.5.1-beta

Choose a tag to compare

@nishujangra nishujangra released this 14 Aug 18:49
· 200 commits to master since this release
314fae8

Spooky 0.5.1-beta — Observability Operator Bundle

Patch release covering all changes since 0.5.0-beta. Ships a packaged operator observability experience — dashboards, alerts, and SLOs on top of the existing metrics — plus canonical audit correlation, hardened default packaging, and an OTLP startup fix.

Added

  • Packaged operator observability bundle under deploy/observability/ — Prometheus recording rules, production alert rules, an SLO package, and six Grafana dashboards (edge traffic, admission overload, backend health, retries/hedges, TLS certificates, control-plane activity), so operators get a working dashboard/alert set instead of assembling one from raw metrics.
  • Canonical correlation fields on admin audit events — event_id, schema_version, request_id, trace_id, span_id, listener — plus a stable failure_class (authentication, authorization, source_policy, request_validation, runtime_config, runtime_state, listener_tls, watchdog) on non-success events, so failures can be grouped and correlated across metrics/logs/traces without parsing free-form reason strings.
  • GET /admin/runtime and the runtime history endpoints gained an observability block — contract version, audit schema version, current generation, backend and quota backend health summaries, recent tracked admin actions, and repository-relative dashboard/documentation references. This is the one canonical entry point into the packaged bundle for operators and automation.
  • h3_client now accepts --method (default GET) and repeatable --header name=value (including pseudo-headers like :protocol), for exercising non-GET and header-sensitive traffic in the observability lab.
  • Documentation — docs/operations/observability-bundle.md covers the shipped dashboards, alerts, SLOs, and incident-correlation workflow; docs/architecture/observability-contract.md and docs/operations/control-plane.md document the audit schema and the new runtime observability block.

Changed

  • The Debian package's default config.yaml now ships the control API disabled (observability.control_api.enabled: false) with a placeholder-token comment block, instead of enabled with a literal replace-with-strong-token credential — a fresh install no longer boots with a live, weakly-credentialed admin surface.
  • The systemd unit restarts with Restart=always instead of Restart=on-failure (the watchdog's own clean-exit drain would otherwise not be restarted), adds StartLimitIntervalSec=60/StartLimitBurst=5 to guard against crash-looping, raises LimitNOFILE=65535 for QUIC-scale file descriptor use, and tightens sandboxing (ProtectKernelTunables, ProtectKernelModules, ProtectControlGroups, RestrictAddressFamilies, RestrictNamespaces, RestrictSUIDSGID, LockPersonality). /etc/spooky is no longer in ReadWritePaths — only /var/log/spooky stays writable.

Fixed

  • OTLP tracing is now initialized inside the Tokio runtime instead of before it. The OTLP tonic exporter spawns a background task while building its gRPC channel, which previously panicked with "there is no reactor running" whenever tracing was enabled.
  • Docker image build and clippy warnings introduced during observability bundle development.

Upgrade Notes

Nothing breaks. The new observability block on /admin/runtime and the audit event fields are additive JSON — existing consumers that don't read them are unaffected.

⚠️ Fresh installs/packages built from this version ship the control API disabled by default. If you rely on the Debian package's default config enabling it out of the box, you must now explicitly set a real bearer token and flip enabled: true.

Existing deployed configs and units are not modified in place by this release — these changes affect newly generated packages only.

Known Limitations

  • Observability contract and audit schema are both fixed at v1 — no versioned migration path yet if the schema needs to change.
  • Dashboard/documentation references in /admin/runtime are repository asset paths, not resolvable URLs — operators must map them into their own Grafana/runbook tooling.
  • Pre-GA — soak test before broad rollout.