v0.18.0

What's new in v0.18.0

The daemon's metrics break down by service. This started from a report that the shipped Grafana dashboard had a Service filter which changed nothing, and it ended in three places where a service name was missing, blank, or attributed to the wrong owner.

The filter that reached one panel out of twenty-one

perf_sentinel_findings_total gains a service label, and perf_sentinel_slow_duration_seconds gains one next to type. Until now only service_io_ops_total carried the dimension, so the dashboard's Service variable filtered that single panel. Selecting a service left Critical findings untouched, which is how this was found on a production install.

The cardinality rule this repo enforces still holds, through a per-run cap on the analysis side:

findings        128 distinct services
slow duration    64 services, 14 series per (type, service) pair
past a cap      folded into service="_other", never dropped

Folding rather than dropping is what keeps sum over the label equal to the old unlabeled series, and the global p95 exact. OpenMetrics exemplars follow the split, keyed on (type, severity, service), so a service's series links to a trace of its own instead of a neighbour's. They also expire 15 minutes after the batch that recorded them, so a service that goes quiet stops pointing at a trace the tracing backend has already dropped.

The avoidable I/O no counter could attribute

perf_sentinel_service_avoidable_io_ops_total{service} is the per-service share of perf_sentinel_avoidable_io_ops, derived in the same scoring pass under the same dedup rule, so the two cannot diverge. A finding whose spans come from several services is split between them, each charged its own repeats, the finding's own service one less for the call that was necessary. The split travels on a new optional pattern.occurrences_by_service.

Its denominator ships beside it, perf_sentinel_service_analyzed_io_ops_total{service}, so a per-service waste ratio never mixes populations: the ingest-side counter counts before analysis, drops past its own 1024 cap instead of folding, and includes batches later shed.

The credit costs nothing measurable. Against the commit before it, at 16 and at 128 services, on a single-service corpus and on one where every group is shared by two services:

-1.3% to +0.4%   the change
      up to 1.9%   the same binary between two runs

disclose uses the same split for per-application avoidable_io_ops, hence efficiency_score and the G1 estimated_waste_*, on windows archived by 0.18.0 or later. Older archives keep first-span attribution. Occurrence counts stay on the owning service, so a service can now show waste with zero occurrences for a pattern it took part in under another service's finding.

The service name two readers left blank

A span carrying no service name resolves to "unknown" on every ingestion path, settled once in sanitize_span_event. The Zipkin and Jaeger readers left the field empty where OTLP already fell back, so the same anonymous trace produced a blank component in its acknowledgment signature and, now that metrics carry the dimension, an empty service label. An empty label is not a neutral one: a scrape that honors exposed labels drops it, and a plain scrape overwrites it with the target's own name.

Finding.service, the report payloads, the query API and the metric label now agree on one value.

unmatched_acknowledgment also stops misreading its own signal. When a current finding shares the detector and the template hash under a different service or endpoint, the attribution moved rather than the query, and the warning now names that successor instead of reading as "the problem looks fixed, drop the entry".

The variance bar that called a cached repeat an N+1

[detection] sanitizer_aware_min_cv exposes the coefficient of variation the sanitizer heuristic compares per-span durations against, default 0.5, the value it hard-wired until now.

The case came from the simulation lab. Under sanitizer_aware_classification = "strict", ten identical Doctrine lookups served from cache on a PHP-FPM worker measured a CV near 0.75 once the runner was loaded. That crossed the bar, turned a redundant_sql finding into n_plus_one_sql, and attached a leftJoin hint to a query that needs memoizing. None of the four modes separates that case from a real N+1, so the threshold had to move rather than the mode.

strict, CV 0.75, bar 0.5    n_plus_one_sql, leftJoin hint
strict, CV 0.75, bar 1.0    redundant_sql, memoize hint

Validated to a finite (0, 10], recorded in detection_config, and a report written before the knob reads back as 0.5. The same threshold feeds the HTTP heuristic, which tells n_plus_one_http from redundant_http by timing spread alone, so raising it moves both verdicts.

Upgrade impact

• Breaking for any alert or dashboard matching the two labeled series without aggregating. perf_sentinel_findings_total{severity="critical"} now returns one series per service, and an unaggregated alert on it fires per service. Wrap it in sum() to keep the old behavior. [daemon] per_service_labels = false restores the pre-0.18 shape, the label stays declared but empty on every series, which PromQL treats as absent.
• A scrape that does not honor the labels a target exposes renames service to exported_service on these series, the same collision service_io_ops_total already had.
• Acknowledgments and baselines taken from Zipkin or Jaeger spans with no service name must be re-taken. Both identities are keyed on the service, and it changed from blank to unknown. Everything else keeps its signature.
• Breaking, perf-sentinel-core only. DetectConfig gains a public sanitizer_aware_min_cv and has no #[non_exhaustive], so a struct literal downstream needs the field. detect_n_plus_one and timing_variance_suggests_n_plus_one take the threshold as a parameter. Finding.pattern gains occurrences_by_service, GreenSummary an in-process avoidable_per_service.
• The shipped Grafana dashboard is reorganized into Analysis (filtered by service), eleven panels the service filter now reaches, and Daemon health (global), ten the filter could never slice. The namespace variable is relabeled Daemon namespace, since it selects the install rather than the analysed workloads.
• The HTML report's help named [detection] n_plus_one_threshold, a key removed in 0.6.0 that the loader refuses. It now names n_plus_one_min_occurrences.
• CLI surface otherwise unchanged, daemon API untouched, MSRV stays 1.97.1.

Full detail in CHANGELOG.md.

Verifying this release

# Binary integrity via SLSA Build L3 attestation
gh attestation verify perf-sentinel-linux-amd64 \
  --repo robintra/perf-sentinel

# A periodic disclosure produced by this binary
perf-sentinel verify-hash --report perf-sentinel-report.json \
  --expected-identity "https://github.com/robintra/perf-sentinel/.github/workflows/release.yml@refs/tags/v0.18.0" \
  --expected-issuer "https://token.actions.githubusercontent.com" \
  --verify-binary ./perf-sentinel-linux-amd64

gh CLI 2.49 or newer required for gh attestation verify.