What's new in v0.13.0
v0.13.0 is about what a Prometheus scrape actually reports, and about what happens to an acknowledgment once the daemon is running. The pg-stat and mysql-stat scrapes were reading a call count no exporter publishes, so three of the four rankings sorted on zeros. And a team decision recorded in the acknowledgments file only applied at startup, which is not an answer when that file is a mounted ConfigMap.
Both change what you read on identical data, and both have a paragraph in the upgrade section below.
The call count was read as a label
calls was read as a label on the timing series. No exporter publishes it that way, the postgres_exporter built-in included: it exposes pg_stat_statements_calls_total alongside the time series, as a series of its own. So the count silently fell back to zero on every deployment, which emptied the ranking by calls and collapsed the mean onto the total, and no test caught it because the fixture carried no calls either.
The count now comes from a query of its own, joined on queryid on PostgreSQL and on digest on MySQL, never on the statement text, which exporters commonly truncate: a wrong join is worse than a missing count. MySQL joins SUM_ROWS_SENT and SUM_ROWS_EXAMINED the same way, so the ranking by rows examined stops sorting an all-zero column it advertised as a placeholder. --calls-metric, --rows-sent-metric and --rows-examined-metric name each series, and an empty value skips its query, which is how an operator opts out for an exporter that publishes no such counter. A failed counter query is never fatal: the timings still produce a report, and a warning names the series that came back empty rather than leaving a silent zero.
One statement, one ranked row
The exporters label one statement once per database, user or schema. topk over the raw series therefore ranked the same statement several times, split its execution time across those rows, and left the call count describing a different set of rows than the time it divides. Every query now folds on the statement identity before ranking, sum by (queryid, <query label>) on PostgreSQL and sum by (digest, <schema label>) on MySQL, so one statement is one ranked row whichever database it ran against.
instance and job stay inside that identity. A Prometheus scraping several database servers must not sum them into one row that names no server, and the report carries no column that would reveal the blend. On MySQL the schema stays in the identity for the same reason and remains visible on the row, with --schema-label naming it when a recording rule renames it.
The unit is not something to assume
pg_stat_statements counts milliseconds while the postgres_exporter built-in converts to seconds, and a hand-written exporter query usually forwards the column untouched. Performance Schema counts SUM_TIMER_WAIT in picoseconds, which the mysqld_exporter collector converts to seconds. Reading one for the other is off by a factor of a thousand on PostgreSQL and by 10^12 on MySQL, with nothing in the output to say so.
--unit states which it is, seconds|milliseconds on pg-stat and seconds|milliseconds|picoseconds on mysql-stat, with --pg-stat-unit and --mysql-stat-unit as the report counterparts. Defaults keep the exporter built-ins, so a scrape that names nothing behaves as before.
What the scrape sends
Each counter query is intersected with the ranked set, and on(queryid), so it stays bounded by --top-n instead of pulling the whole counter series. Unfiltered, an instance with a large pg_stat_statements.max spread over several databases overruns the HTTP client's body cap. The error is caught and downgraded to a warning, and the counts fall back to the zero this join exists to prevent.
Series names and label names both land unencoded in the query string, so each goes through its own PromQL grammar guard rather than being escaped: the metric-name grammar for a series, the tighter label grammar for a label, which carries no :. An operator typo carrying & or # would otherwise smuggle a second parameter into the URL instead of failing outright.
An acknowledgments file read once goes stale
The TOML acknowledgments file is re-read every minute instead of only at startup. Telling an operator to restart a daemon so it notices a text edit is not an answer when that file is the sanctioned way to record a team decision, and when it is usually a mounted ConfigMap. The map moves behind an ArcSwap, matching the energy cache already in the tree, so reads stay lock-free and a query in flight keeps a coherent view rather than one spanning two revisions of the file.
A mounted ConfigMap is projected through a ..data/ indirection, so the configured path is a symlink, and the loader refused to follow it: the daemon would not start at all. A link resolving under its own directory is now accepted, since following it grants no reach the caller did not already have by naming that directory. One resolving anywhere else stays refused, which is the case the check was written for.
Absence is told from an empty file. A deleted ConfigMap or an unmounted volume keeps the previous acks and warns once, and so does a file that is present but unreadable, because neither must un-acknowledge what the team decided to accept. [daemon.ack] enabled = false stops the poll rather than repopulating a map that startup deliberately left empty. Only an explicitly configured toml_path is polled, never the CWD-relative default, and the interval matches the kubelet's ConfigMap sync period, since polling faster would only re-read a file the platform has not refreshed.
One deployment note: mount the ConfigMap as a directory. A subPath mount freezes the file at mount time and defeats the reload entirely.
Upgrade impact
calls and mean_exec_time change on identical data, on every Prometheus scrape. They were zero and a repeat of the total respectively, so the rankings by calls and by mean execution time were placeholders rather than signal. They now carry real figures, which means a ranking read from a 0.12.0 scrape and the same ranking read from 0.13.0 are not comparable. The same holds for rows_sent and rows_examined on the MySQL scrape.
A statement reported once per database now occupies one ranked row instead of several. A --top-n 10 scrape that spent its slots on one statement seen from four databases now shows ten distinct statements, and that statement's time is the sum across them. Rows from different servers are still kept apart.
An edit to the acknowledgments file now applies without a restart. A daemon with [daemon.ack] toml_path configured picks up an edit within a minute, where it previously held the startup snapshot for its whole lifetime. Nothing else about acknowledgments moves: signatures are untouched, and a file the daemon cannot read leaves the previous acks in place.
The file-based --input path of both pg-stat and mysql-stat is unchanged, and the minimum supported Rust version stays 1.97.1.
Validation
The full Rust suite passes (3235 tests), clippy is clean at -D warnings across all targets, cargo fmt --check passes, and the default and --no-default-features builds both pass.
The Prometheus work rests on its unit and end-to-end tests: the counter joins, the identity fold across databases and across servers, the query-string encoding, and the flag surface of both subcommands and of report. The acknowledgments reload is covered the same way, absence told from an empty file included.
Verifying this release
# Binary integrity via SLSA Build L3 attestation
gh attestation verify perf-sentinel-linux-amd64 \
--repo robintra/perf-sentinel
# A periodic disclosure produced by this binary
perf-sentinel verify-hash --report perf-sentinel-report.json \
--expected-identity "https://github.com/robintra/perf-sentinel/.github/workflows/release.yml@refs/tags/v0.13.0" \
--expected-issuer "https://token.actions.githubusercontent.com" \
--verify-binary ./perf-sentinel-linux-amd64gh CLI 2.49 or newer required for gh attestation verify.