perf-sentinel chart v0.17.0

What's new in chart-v0.17.0

The chart itself changes this time. The shipped PrometheusRule drops from seven always-on alerts to five and three of the survivors change shape, which is breaking for anyone running prometheusRule.enabled: true. appVersion moves from 0.16.0 to 0.17.0, and that move carries a fix a chart user feels directly: the daemon no longer gets OOM-killed by a wide topology.

No other template moves. values.yaml gains comments and not one key, so a values file renders the same objects apart from the PrometheusRule, the image tag and the version labels.

The alert set

The old set alerted on conditions that were routine, already predicted by a second rule, or already a panel on the Grafana dashboard this repo ships, and two of them carried severity: info, which routes to nobody. What remains fires only on data the daemon lost and cannot recover.

PerfSentinelDown was blind to the topology this repo documents. absent(perf_sentinel_active_traces) fires only when no series matches anywhere and carries no labels at all. On the sharded layout in examples/docker-compose-sharded.yml, killing one of two replicas left the survivor publishing the metric, so the alert stayed green while half the spans went into a hole. It was equally silent when one Prometheus scraped two installs. It is now up{job="<release fullname>"} == 0, for: 15m, at severity: warning.

serviceMonitor.interval 15s  ->  15m is 60 consecutive failed scrapes
workload.strategy maxSurge 1 / maxUnavailable 0
  a rolling update retires the old target from discovery, it never reports down

PerfSentinelIngestRejecting could be fired by a stranger. It matched reason!="memory_pressure", which swept in parse_error and unsupported_media_type, raised by any malformed HTTP request reaching :4318. It is now scoped to reason="channel_full", the only reason that is span loss on this side, with a > 0.05 floor over [10m] held for: 15m. As shipped, a single increment held rate([5m]) non-zero for five minutes, so > 0 with for: 10m was satisfied by roughly three rejected requests spread across ten.

PerfSentinelAnalysisShedding gains a floor of one trace per second over [10m], held for: 15m, so it fires past roughly 900 traces whose findings will never exist. At > 0 it fired on about three shed traces in ten minutes.

PerfSentinelArchiveDropping is new, on rate(perf_sentinel_archive_windows_dropped_total[15m]) > 0. A dropped disclosure window leaves the archive hash chain contiguous, so verify-hash recomputes clean over a record that is short, and this counter is the only witness. It carries no floor, unlike the two above, because the unit is a whole scoring window rather than one request. Inert on an install without [daemon.archive].

Three rules are gone. PerfSentinelAnalysisQueueSaturated predicted the shedding alert, said so in its own description, and is already the Runtime headroom and shedding panel under the same division. PerfSentinelCorrelatorEvicting carried severity: info and fired on the documented steady state of the feature it watched. PerfSentinelServiceCardinalityOverflow carried severity: info and its own description ended Expected on very wide topologies, while MAX_SERVICE_CARDINALITY is a compile-time constant, so it named no knob to turn.

Upgrade impact

• Grep your Alertmanager config for the three removed names before upgrading: PerfSentinelAnalysisQueueSaturated, PerfSentinelCorrelatorEvicting, PerfSentinelServiceCardinalityOverflow. Alertmanager does not warn about a route that stops receiving, so a silence, route or inhibition rule keyed on them simply goes quiet.
• PerfSentinelDown reroutes and regroups. It moves from critical to warning, and where absent() carried no labels it now carries instance, pod and namespace, so grouping and deduplication keys change and a multi-replica install gets one alert per replica.
• It also depends on the scrape job name, up{job="<release fullname>"}, which is what serviceMonitor.enabled: true produces since no jobLabel is set. Scraping with your own scrape_config under a different job_name leaves that one rule silent, so override it through prometheusRule.additionalRules.
• Two rules get quieter and that is the point. PerfSentinelIngestRejecting no longer fires on client-side rejections at all, and both it and PerfSentinelAnalysisShedding gain floors and longer holds. Those conditions stay visible on the OTLP span intake and Runtime headroom and shedding panels.
• Unchanged: PerfSentinelMemoryPressureRejecting and the five PerfSentinelEnergyScraperStale rules are byte-identical, and no values.yaml key is added or removed, so no values file needs editing.

Behavior

• The upgrade rolls the pods. The image tag falls back to .Chart.AppVersion when image.tag is empty, which is the default, so the pod spec moves from 0.16.0 to 0.17.0. The checksum/config annotation changes with it. Pinning image.tag keeps the old image and skips the rollout, but see the next point before you do.
• The new image fixes a crash. 0.17.0 stops the cross-trace correlator from OOM-killing a memory-limited daemon on a wide topology. Measured in the simulation lab at 1500 services and 80 traces per second, one batch built roughly 1.6 million distinct keys, a 34 MiB table and 51 MiB live across the rehash, inside a 256 MiB container. A deployment that worked around this by setting [daemon.correlation] enabled = false can turn it back on.
• Nothing else in the deployment moves. Routes and wire formats are unchanged: /api/findings, /api/correlations, /api/export/report, /metrics and the OTLP listeners answer in the same shapes, /metrics gains no series, and the disclosure schema stays at perf-sentinel-report/v1.7. Findings, thresholds and signatures are the same, so no acknowledgment and no baseline moves on this upgrade.

Install

The chart is published as an OCI artifact on GHCR, install it directly with no helm repo add step:

helm install perf-sentinel oci://ghcr.io/robintra/charts/perf-sentinel --version 0.17.0

Upgrade an existing release:

helm upgrade perf-sentinel oci://ghcr.io/robintra/charts/perf-sentinel --version 0.17.0

Read docs/HELM-DEPLOYMENT.md for sizing, Ingress postures and the acknowledgments ConfigMap mount, and docs/METRICS.md for the five rules as a plain groups: block, which is what an operator outside Kubernetes needs since PrometheusRule is a Prometheus Operator CRD. The v0.17.0 binary notes cover the daemon and batch changes.

If you are upgrading from chart-v0.12.0 or earlier, read the chart-v0.13.0 notes first: that release makes a mounted acknowledgments ConfigMap reload without a pod restart. Mount it as a directory, a subPath mount defeats the reload.

Full Changelog: chart-v0.16.0...chart-v0.17.0