v0.51.1

This release fixes Prometheus scrape timeouts and disappearing gauge series on installs with large client snapshot tables. Client analytics is now computed by a background aggregator into persisted 5-minute rollups and served from memory, so the metrics endpoint does no analytics storage I/O. Upgrades are drop-in: the new indexes build in the background and history rolls up while the endpoint keeps serving.

Fixes

Prometheus scrapes no longer time out on large client snapshot tables

On installs with a large client_snapshots table (around 10 million rows on PostgreSQL in the reported case), every scrape of /api/prometheus/metrics recomputed client analytics over the whole table inside the request: a COUNT(DISTINCT) plus roughly 70 per-name and per-user queries, taking 17 to 50 seconds. Prometheus gave up at its 10s timeout, and because the next INFO read waited behind that storage work, betterdb_poll_stale flipped to 1 for every connection and the staleness sweep removed their gauge series. The UI's client analytics endpoints hit the same queries.

The scrape now does no client analytics storage I/O at all. A background aggregator folds new raw snapshots into a persisted table of 5-minute rollups (client_analytics_rollups, on PostgreSQL, SQLite and the memory adapter) every 15 seconds, reading only rows newer than its watermark through a new (connection_id, captured_at) index, and keeps the latest stats per connection in memory for the exporter to read. The ACL audit gauges (betterdb_acl_denied*) get the same treatment: the audit poller refreshes an in-memory copy after each ACL LOG poll that saves new entries and at least every 10 minutes otherwise, so rows removed by retention drop out; their values are unchanged all-time totals. betterdb_poll_stale now reflects INFO reachability only; the storage-based refresh runs separately and can no longer hold up INFO polls or the scrape response.

On the stats endpoint, explicit ranges up to 6 hours for a single connection are still computed exactly from raw snapshots, now in 2 queries instead of about 70; longer ranges, and any request without a connection header, are served from the rollups and aligned to 5-minute bucket boundaries. GET /client-analytics/history defaults its start to one window ago, and idle-connections looks at the last window instead of all history. Client IP addresses are now derived by stripping the trailing port, so IPv6 addresses such as [::1]:6379 no longer collapse to [ in unique-IP counts and the time series.

Behaviour change: the unique name/user/IP counts, the average connection age and the per-name and per-user figures in GET /client-analytics/stats called without a range now cover the last 24 hours (CLIENT_ANALYTICS_WINDOW_MS) instead of all stored history. An all-history distinct count has to scan every stored row on each refresh, which is exactly what made scrapes time out; a window keeps the cost proportional to the window, not the table. The headline peak (betterdb_client_connections_peak, and peakConnections/peakTimestamp in the stats response for one connection without a range) covers the rollup retention (CLIENT_ANALYTICS_ROLLUP_RETENTION_DAYS, default 90 days): it is read from the peak stored on each rollup, so it stays cheap over the whole retention. Current connection counts (betterdb_client_connections_current, _by_name, _by_user) still reflect the newest snapshot batch. Stats merged across connections count a connection toward "current" only if its newest batch is within the larger of 5 minutes and two poll intervals of the newest batch overall, so a connection that stopped reporting does not inflate the total. Right after startup, the client connection series are absent for up to 15 seconds until the aggregator's first run completes.

New configuration (all optional):

• CLIENT_ANALYTICS_WINDOW_MS (default 86400000, 24h): window for unique counts, average age and per-name/per-user figures.
• CLIENT_ANALYTICS_ROLLUP_RETENTION_DAYS (default 90): rollup retention and the span of the headline peak. Rollups are also swept by the general retention window like every other store; whichever is shorter wins.
• CLIENT_ANALYTICS_RAW_RETENTION_DAYS (unset by default, keeping raw snapshots as before): set it to prune raw snapshots older than that many days, hourly and in chunks. Pruning starts only once the backfill has rolled that history up.
• CLIENT_ANALYTICS_QUERY_BUDGET_MS (default 2000): a client analytics or audit stats storage call slower than this logs one warning per method, then stays quiet for that method for 5 minutes.

The aggregator cadence (15 seconds), the backfill chunking (1-hour chunks, 4 per connection per run) and the 6-hour limit for raw-computed ranges are fixed.

Upgrading an existing install: nothing blocks startup. On PostgreSQL the two new indexes (idx_client_connection_captured on client_snapshots, idx_acl_connection_captured on acl_audit) are built in the background with CREATE INDEX CONCURRENTLY once the app is ready; on a 9.4 million row table this took 14 seconds and produced a 68 MB index. If another instance holds the build lock (for example the previous pod during a rolling update), the build waits and retries rather than giving up, and an index left invalid by an interrupted build is dropped and rebuilt. On SQLite the indexes are created during schema setup before the HTTP server listens, at roughly 1.5 seconds per million rows (15 seconds for 8 million rows). The aggregator produces current values from the last hour of raw data in its first run, then, once the index is valid, backfills older history backwards in 1-hour chunks back to the rollup retention horizon; until that finishes, the headline peak covers only the history folded so far. After downtime, one indexed lookup finds the next stored snapshot and the scan jumps straight to it, so the aggregator catches up in a single run. On the 9.4 million row test dataset (4 connections, 8 days of history) the backfill took 12 minutes, about 1.3 minutes per million rows, while the metrics endpoint stayed under 150 ms throughout. Rollups stay small: 288 rows per connection per day, 18 MB for the whole dataset.

Measured on that dataset, v0.51.0 compared with v0.51.1:

Measurement	v0.51.0	v0.51.1
/api/prometheus/metrics scrape latency	p50 29.4s, max 51s	p50 16 ms, p95 75 ms
Scrapes timing out at 10s	12 of 12	0, including during backfill
Connections flagged stale (betterdb_poll_stale)	4 of 4	0
/client-analytics/stats, no range	35.6s	under 3 ms
/client-analytics/stats, 1-hour range	6.4s	5 ms
/client-analytics/stats, 7-day range	not measured	0.3s
/client-analytics/spike-detection	3.7s	1.6s
/client-analytics/idle-connections	1.3s	0.8s
Aggregator run, steady state	n/a	p50 3 ms, max 150 ms

Documentation

• The FAQ now states that a single monitor instance per storage database is the supported deployment; several instances sharing a database or the same Valkey connections are not.
• docs/updating.md describes how schema changes behave on large PostgreSQL and SQLite databases during an upgrade.

What's Changed

• fix(migration): count keys on every cluster master during validation by @amitkojha05 in #545
• feat(semantic-cache): cancel judgeFn on timeout via AbortSignal by @amitkojha05 in #220
• feat(settings): configurable anomaly detector thresholds via PATCH /settings/anomaly/detectors by @amitkojha05 in #206
• chore(deps): bump @modelcontextprotocol/sdk from 1.31.0 to 1.32.0 by @dependabot[bot] in #554
• fix(client-analytics): serve Prometheus from persisted rollups instead of full-table aggregates by @KIvanow in #557

Full Changelog: v0.51.0...v0.51.1