v0.20.0

What's new in v0.20.0

A service is OOM-killed at 14:03. What was already firing on it in the five minutes before? perf-sentinel held the answer and lost it: the findings ring is a FIFO with no TTL, which on a busy fleet turns over in minutes, and the listing had no way to ask for a period. By the time anyone opened a terminal the window was gone.

perf-sentinel cannot detect the crash itself: it has no OTLP metrics path, and a saturating process keeps emitting spans. So the moment comes from the alerting that already knows it, and what perf-sentinel owns, the findings of a period, is frozen before the ring lets go.

A window on the findings listing

GET /api/findings takes since_ms and until_ms, both Unix epoch milliseconds against stored_at_ms. With both, the listing is a window, folded over the detections inside it alone, so first_seen_ms and seen_count describe the window rather than the whole retained history. That fold order is the point: applied after it, an upper bound would keep every group whose lifetime overlaps, and a chronic pattern running all week would match every incident window ever asked for. since_ms alone stays a delta poll with whole-buffer counts, which is what a collector resynchronising after a restart needs.

GET /api/status reports oldest_finding_ms, which separates "nothing fired" from "the ring no longer reaches that far back". Both were the same empty answer before.

perf_sentinel_service_last_span_timestamp_seconds{service} publishes when each service was last heard from, a Unix stamp on the process_start_time_seconds convention rather than an age, so time() - gauge is the age and a scrape gap cannot make it lie. Deliberately not a liveness signal: a crash, a scale to zero, a rolling deploy and a quiet cron all read the same.

The moment comes from your alerting

POST /api/incidents receives an Alertmanager webhook and freezes the findings of the window that preceded it. GET /api/incidents reads them back. Both sit behind the opt-in [daemon.incidents] section and its required api_key.

window   [at_ms - lookback_ms, at_ms + 2 * trace_ttl_ms]
freeze   at reception, then one settle pass per delivery
merge    by signature, so the record grows and never loses a row

The window closes after the incident on purpose. A finding is stamped when its trace is analysed, one TTL after its last span, so the traces live at the crash land past startsAt and a window closed there would miss exactly what a post-mortem wants. The settle pass re-resolves the same window once the analysis has caught up and merges by signature, so whatever the ring evicted in between, the capture can only grow. Each record carries the oldest_finding_ms read under the same lock as the fold, which is what separates a complete capture from one the ring had already eaten into.

The Alertmanager envelope is the only accepted shape, webhook_config having no body template while any script emits it with curl. Three labels are read. The service is the join key to the findings, so an alert without it is refused. The kind must be one of oom_kill, memory_saturation, restart, deploy or other, never guessed from alertname, and a deploy is posted for the same reason as a restart: to freeze what was already firing before the rollout, so a restart it causes is not read as a crash. The namespace, defaulting to the label kube-prometheus alerts already carry, is optional, is carried on the record and filters the listing.

Reposting is idempotent on a content-derived id: a repeat re-resolves a fixed window against a ring that only evicts, so the first capture stands and a resolved delivery only closes it. perf_sentinel_incidents_total{kind} counts incidents rather than deliveries.

[daemon.incidents] archive_path appends every record to a newline-delimited JSON file through one writer opened at startup, because the ring dies with the daemon and a node-level memory event that kills the observed service often takes a co-located daemon with it.

perf_sentinel_incidents_rejected_total{reason} counts what the routes refused: a missing key, a missing service label, an unparsable startsAt, the alerts past the per-delivery cap. The intake body says the same, but Alertmanager discards it and never retries a 4xx, so a receiver with the wrong header used to lose every capture with nothing to show for it.

A key that reads and cannot write

[daemon] read_api_key, with PERF_SENTINEL_READ_API_KEY winning over it, opens the two GETs a write key gates, /api/acks and /api/incidents. It never satisfies a POST or DELETE, never adds a gate where no write key is set, and must differ from both write keys, a read key equal to a write key being that key. It exists so a dashboard or a collector, each carrying one header value, never holds the key that can ack a finding or fabricate an incident.

Two related corrections. The CORS wildcard refusal knew the ack key alone and now covers both write keys, so [daemon.cors] allowed_origins = ["*"] next to [daemon.incidents] api_key fails at load where it used to pass. And the environment overrides now apply with no config file present, where they were silently skipped: a container started with its keys in the environment and no mounted config ran with its write routes open.

Reading it back from a terminal

perf-sentinel query monitor gains an Incidents tab, its sixth, and perf-sentinel query incidents lists the same records. The tab polls the 20 newest on every tick: one row per incident with its start as local time, the service as ns/service when the alert carried a namespace, the kind, firing or ended, then the finding count and a capture marker, then one indented row per finding with whether it fired before or after the restart. The daemon's refusals stay hints on the tab rather than a stale monitor, a 401 naming the flag that passes the key, a 503 naming the disabled section, a 404 naming a daemon older than 0.20.0.

It is the one tab that needs a key, so query monitor gains --api-key-file and the read-only key suffices, the monitor never holding a key that can POST.

Never following the last path component

The daemon's three appenders, the analysis archive, the incident archive and the ack store, now open with O_NOFOLLOW on Unix, where the analysis archive had a symlink check alone and a link planted between the check and the open was written through. On Windows they open with FILE_FLAG_OPEN_REPARSE_POINT and refuse a handle carrying a reparse point, the ack store's symlink refusal having been Unix-only. That refusal covers every reparse point, a cloud sync placeholder or a deduplicated file included, so point the three paths at a plain local file.

Upgrade impact

• Nothing is on by default. [daemon.incidents] is opt-in and its routes answer 503 when absent, [daemon] read_api_key is unset, and the two new GET /api/findings parameters are optional. A 0.19.0 configuration runs unchanged.
• [daemon.incidents] enabled = true without an api_key is a load error, not a warning, the route being an inbound write surface.
• [daemon.cors] allowed_origins = ["*"] beside [daemon.incidents] api_key now fails at load. It already failed beside [daemon.ack] api_key.
• A weakened archive or ack file is now refused at startup. Mode 0600 applies on creation only, so a file that gained group or world bits afterwards fails the daemon rather than being written to.
• perf-sentinel query findings exits 1 on a 2xx body it cannot parse, naming the route, where it printed No findings from daemon. and exited 0. A proxy's error page no longer reads as a clean run in CI.
• Breaking, perf-sentinel-core only. DaemonConfig gains read_api_key and incidents, and DaemonIncidentsConfig is new. MetricsState gains three incident counters and the last-span gauge. The acknowledgment signatures, the detection verdicts and the rest of the daemon API are unchanged, MSRV stays 1.97.1.
• /api/config reports read_api_key_set and incidents_enabled, both visible in the query monitor Config tab.

Full detail in CHANGELOG.md.

Verifying this release

# Binary integrity via SLSA Build L3 attestation
gh attestation verify perf-sentinel-linux-amd64 \
  --repo robintra/perf-sentinel

# A periodic disclosure produced by this binary
perf-sentinel verify-hash --report perf-sentinel-report.json \
  --expected-identity "https://github.com/robintra/perf-sentinel/.github/workflows/release.yml@refs/tags/v0.20.0" \
  --expected-issuer "https://token.actions.githubusercontent.com" \
  --verify-binary ./perf-sentinel-linux-amd64

gh CLI 2.49 or newer required for gh attestation verify.