v0.22.0

What's new in v0.22.0

POST /api/incidents has been documented since 0.20.0: hand the daemon the moment an incident happened, and it freezes the findings of the window that preceded it before the ring evicts them. On Kubernetes, the thing that knows that moment is Alertmanager, and the way to configure Alertmanager is an AlertmanagerConfig or a VMAlertmanagerConfig. Neither of those CRDs can send an arbitrary header, so neither can send X-API-Key, and the route was unreachable from the receiver an operator actually generates.

v0.22.0 accepts the credential those CRDs can carry, and ships the rules and the receiver to go with it.

The header a generated receiver cannot send

Every key-gated route now takes Authorization: Bearer <key> beside X-API-Key: POST and DELETE /api/findings/{signature}/ack, GET /api/acks, and both verbs of /api/incidents. Either one, the same key, the same constant-time comparison, and the same rules about which key opens which verb, so [daemon] read_api_key still reads without ever writing.

Bearer is only consulted once X-API-Key has failed. An existing deployment sending the header keeps deciding on its own, and nothing about it changes. The scheme is matched case-insensitively, as RFC 7235 requires, while the credential itself is compared byte for byte.

The CORS layer still advertises x-api-key and deliberately not authorization. Bearer is for server-to-server callers, so a cross-origin browser client is refused at preflight and keeps using the header, where a second replayable credential would buy nothing.

The rules nobody could copy

examples/incident-alerts-prometheus-operator.yaml and examples/incident-alerts-victoriametrics-operator.yaml carry four alert rules, oom_kill, restart, memory_saturation and deploy, plus the receiver that posts them. The rules produce the three labels the route reads, service, perf_sentinel_kind and namespace, which the shipped kube-prometheus rules never do: those carry namespace and pod, never a service and never a kind.

service is derived fleet-wide from the pod name rather than written one rule at a time, and the two cluster assumptions behind that derivation are stated at the top of the file instead of being left implicit. The deploy rule keys on a new ReplicaSet rather than on the Deployment's generation, which an HPA also advances, so an autoscale is not posted as a release.

Both files also name the trap that costs the most: each operator appends a matcher on its own resource's namespace by default, while the alerts carry the observed workload's. Nothing matches, no incident arrives, and no refusal is counted anywhere either, because the delivery never leaves. The prometheus-operator file names the route ordering that decides whether an earlier route swallows the alerts first.

The join a second kube-state-metrics breaks

The deploy rule aggregates both sides of its many-to-one join. Without that, a kube-state-metrics scraped on two instances publishes kube_replicaset_owner twice per replicaset, and PromQL refuses the match: the rule does not alert late, it fails to evaluate and posts nothing at all, while prometheus_rule_evaluation_failures_total climbs where nobody is watching.

The simulation lab pins both halves. Against a real Prometheus with kube-state-metrics at two replicas and 18 duplicated owner series, the shipped rule evaluates with zero failures and its pre-review expression fails three times. The shipped PrometheusRule was then applied byte for byte, a real Alertmanager delivered through the generated receiver with its bearer credential, and the resulting incident came back carrying 72 frozen findings, while the released 0.21.0 running beside it under the same ConfigMap and the same receiver refused the same delivery at the same instant.

Upgrade impact

• The body of a 401 on the query API now reads missing or invalid X-API-Key or Authorization: Bearer, where it named the header alone. A client that matches on that string needs updating. The status code, the headers and every other response are unchanged.
• Nothing changes for a deployment that sends X-API-Key. The header is tried first and short-circuits, so bearer is a second way in, never a replacement, and no key needs rotating or re-issuing.
• A key reaching a key-gated route in bearer form has the same power as in header form, which includes the write keys. [daemon] read_api_key still opens GET /api/acks and GET /api/incidents and is still refused on every write, in either form.
• Additive. No configuration key is added or removed, no route, metric or wire format changes, no public signature in perf-sentinel-core moves, the acknowledgment signatures and the detection verdicts are untouched, and MSRV stays 1.98.1.

Full detail in CHANGELOG.md.

Verifying this release

# Binary integrity via SLSA Build L3 attestation
gh attestation verify perf-sentinel-linux-amd64 \
  --repo robintra/perf-sentinel

# A periodic disclosure produced by this binary
perf-sentinel verify-hash --report perf-sentinel-report.json \
  --expected-identity "https://github.com/robintra/perf-sentinel/.github/workflows/release.yml@refs/tags/v0.22.0" \
  --expected-issuer "https://token.actions.githubusercontent.com" \
  --verify-binary ./perf-sentinel-linux-amd64

gh CLI 2.49 or newer required for gh attestation verify.