v0.54.0

v0.54.0

This release adds mutual TLS: BetterDB can present a client certificate, verify a private CA, and authenticate by certificate alone, with no AUTH. Anomaly detection adds three failure patterns seen in write-heavy production clusters. Replication divergence tracks each replica's byte gap to the primary, not just lag. Retry amplification separates a retry storm that feeds itself from a transient burst and names the clients behind it. Incident timelines put related events in order and show whether the instance recovered or settled into a degraded state. The MONITOR health gate now reads replication lag correctly, and key analytics no longer sets off BetterDB's own error-reply alerts.

Upgrades are drop-in. The new storage columns are created on startup and existing connections keep working with password auth. A few behaviour changes to know about:

• Anomaly-triggered and scheduled MONITOR captures can now be skipped for replica lag or an unhealthy replica link. Before, lag always read as about 0 (see MONITOR health gate).
• Command-mode migrations from a cluster now include masters flagged fail?. An unreachable one fails the job instead of being skipped silently.
• error_replies is a new anomaly metric. It alerts at 20 error replies per second or more.

Highlights

Mutual TLS and certificate-only auth (#630, #633, #646)

• Client certificate and key: each connection gets a TLS panel under Use TLS. Paste PEM inline, where the private key and passphrase are encrypted at rest with ENCRYPTION_KEY, or point at files on the server inside BETTERDB_TLS_CERT_DIR, with the same allowlist and symlink checks as SSH keys.
• Private CA: a custom CA bundle verifies the server certificate. Verification stays on; there is no "verify: none".
• Certificate-only auth: authMode: 'certificate' (DB_AUTH_MODE=certificate, Client certificate (mTLS) in the UI) sends no AUTH, so the server maps the certificate to an ACL user (tls-auth-clients-user). After connecting, BetterDB runs ACL WHOAMI and shows the authenticated user in the test result and connection status, so a certificate that silently fell back to default is visible.
• Env default connection: DB_TLS_CA_FILE, DB_TLS_CERT_FILE, DB_TLS_KEY_FILE, DB_TLS_KEY_PASSPHRASE and DB_AUTH_MODE.
• Cluster and Sentinel: discovered nodes inherit the seed's certificate and auth mode.
• Certificate rotation:
  • File-based material is re-read before every reconnect, including on cluster node clients and migration clients.
  • A half-written pair caught mid-rotation keeps the previous material and is retried on the next check.
  • A broken file logs one warning per distinct error, and one line when the files are usable again.
• Migration: command-mode migrations use the same TLS and auth options per cluster master. Each target cluster node gets its own TLS servername (SNI) when it is reached by hostname. RedisShake modes are blocked when either side uses a client certificate.
• Test connection matches save: the test now validates the same cleaned-up TLS config that saving does. Before, a config could fail the test and still save, or the other way round.
• Not covered yet:
  • The standalone agent (--valkey-tls) still supports plain TLS only.
  • Cluster nodes reached by bare IP get no TLS servername.

Replication divergence (#618)

lag says how recently a replica acknowledged the stream, not how much it has applied, so a replica can fall behind while lag stays at 0. BetterDB now reads each replica's offset from the primary's INFO replication and tracks the gap in bytes and in seconds at the current write rate. It works in windows of 30 seconds, or three poll intervals if that is longer.

• WARNING when the gap grows across 4 windows, or the replica stays at least 30 s behind for 2 windows.
• CRITICAL at 120 s behind.
• CRITICAL when writes are back near their baseline, or have stopped, and the gap still hasn't halved, so the replica isn't catching up.
• Missing acknowledgements: WARNING after 2 windows without them and CRITICAL after 4, for a gap of at least 1 MiB.
• Noise floor: gaps under 1 MiB or 5 s behind are ignored.
• Webhooks: new events replication.divergence and replication.recovered. When a replica disconnects, its open alert closes without sending recovered.
• UI and MCP: a per-replica divergence panel on the anomaly dashboard, and the MCP tool get_replication_divergence.
• Coverage: detection runs on connections that report a primary role. Sentinel connections and cluster shards whose primary isn't registered aren't covered; set SENTINEL_AUTO_REGISTER_NODES or CLUSTER_AUTO_REGISTER_NODES.

Retry amplification (#620, #645)

Slow or failed calls cause retries, the retries add load, and the server gets slower, so the storm doesn't settle on its own. The new detector looks for that loop in 60-second windows. It needs commands per client climbing while the client count stays flat, plus a stress signal: rising error replies, connection churn or a P99 regression.

• Two shapes:
  • rising: load climbs and moves with error replies or churn.
  • step: load jumps to 2× baseline and holds, with an error or reconnect burst.
• Latency alone isn't enough. A P99 regression with flat clients looks the same as organic growth through a fixed connection pool, so it can't qualify a run on its own.
• Severity: WARNING when a run qualifies. CRITICAL when the server is also saturated: load or client saturation, rejected connections or lockout risk.
• Transient runs: runs that end within 3 windows are recorded as transient and don't alert.
• Culprit clients: the alert names up to five clients, ranked by command-rate growth (from CLIENT LIST tot-cmds, Valkey 8+) and by reconnect counts. BetterDB's own connections (BetterDB-Monitor*, BetterDB-Agent*, BetterDB-CLI) are left out.
• Webhook: retry_amplification.detected.
• MCP tool: get_retry_amplification.
• New metric: error_replies (from total_error_replies) is now tracked as an anomaly metric and feeds this detector.

Incident timelines and compound failure patterns (#621)

• Incident timelines: related events are grouped per instance, or per cluster, and kept in order. Each incident has a status: active, recovering, recovered or degraded. Degraded means the instance didn't return to its state before the incident. Available at GET /anomaly/incidents and the MCP tool get_incident_timeline. There is no web view yet.
• Recovery amplification: a short replica disconnect turns into a full resync when the backlog no longer covers the gap. BetterDB raises a resync-escalation event when that happens. It matches the sequence disconnect → full resync → fork or output-buffer pressure into one incident.
• Backlog coverage in seconds: repl-backlog-size divided by the current write rate. That is how long a replica can drop off before it needs a full resync. WARNING under 60 s, CRITICAL under 15 s.
• Reshard pressure: during a slot migration, a source or target shard that is far above the cluster median, with its write P99 regressing, raises RESHARD_PRESSURE, even when total throughput looks normal. This needs at least three primaries registered as connections.
• Connection churn: connection_churn alerts when clients reconnect instead of reusing connections: 10 or more new connections per second while connected_clients stays flat. Workloads that open a connection per request, such as PHP-FPM or serverless functions, stay quiet during a traffic surge.
• Correlator inputs: per-command P99 regressions and connection churn now feed the correlator.

MONITOR health gate

• Replica lag is read from the primary (#619): the gate used to compute lag from a replica's own INFO replication, where both offsets are the replica's own position, so it always read about 0. On a primary, lag now comes from the furthest-behind online replica. Replicas still in their initial sync are ignored.
• Unhealthy replica links skip captures: a new replica_link_unhealthy reason, controlled by MONITOR_REPLICA_IO_STALE_SEC (default 30).
• Preflight panel: it shows lag and link status. Manual captures are never blocked by them.

Key analytics

• Key analytics stops generating error replies (#645): it sent all six size commands (STRLEN, LLEN, HLEN, SCARD, ZCARD, XLEN) and both OBJECT IDLETIME and OBJECT FREQ for every key, then kept the one reply that matched. The rest came back WRONGTYPE or ERR, about 60,000 error replies per 10,000-key scan. That fired a CRITICAL error_replies alert on every connection every 5 minutes and inflated the errorstat counts in your own dashboards. Now:
  • TYPE runs once per scan batch, then each key gets only its own size command.
  • Only one of OBJECT IDLETIME and OBJECT FREQ is sent, depending on the eviction policy.
  • The same change is in @betterdb/agent. Agent connections pick it up with the next agent release (still 1.8.0 in this repo).
• Keys deleted mid-scan: a key deleted between the batch TYPE and its probes is recorded as gone, not as its old type with size 0.

Migration

• Cluster masters parsed with the shared parser (#646): command-mode migration now reads CLUSTER NODES with the same parser as the rest of the app. Masters flagged fail? (suspected down) are included instead of being skipped with their keys.

Connections

• Plaintext secrets are re-encrypted before the environment sync at startup, so a failed sync can't leave a row unencrypted (#646).
• CA-only TLS configs no longer require ENCRYPTION_KEY (#633).

What's Changed

• Mutual TLS: client certificates, private CA, certificate-only auth by @KIvanow in #630
• fix(connections): follow-ups to mutual TLS (#630) by @KIvanow in #633
• fix: mTLS review follow-ups by @KIvanow in #646
• Detect replication divergence per replica by @jamby77 in #618
• Detect retry amplification and attribute it to clients by @jamby77 in #620
• Track connection churn and build incident timelines by @jamby77 in #621
• Read replica lag from the primary in the health gate by @jamby77 in #619
• fix: stop BetterDB's own commands from triggering error and retry alerts by @KIvanow in #645
• Isolate the license spec from the real data directory by @jamby77 in #632
• chore(deps): bump opentelemetry-api from 1.42.1 to 1.45.1 in /packages/cache-benchmark by @dependabot[bot] in #628
• chore(deps): bump setuptools from 83.0.0 to 84.1.0 in /packages/cache-benchmark by @dependabot[bot] in #629

Full Changelog: v0.53.0...v0.54.0

What's Changed

• chore(deps): bump setuptools from 83.0.0 to 84.1.0 in /packages/cache-benchmark by @dependabot[bot] in #629
• chore(deps): bump opentelemetry-api from 1.42.1 to 1.45.1 in /packages/cache-benchmark by @dependabot[bot] in #628
• Mutual TLS: client certificates, private CA, certificate-only auth by @KIvanow in #630
• fix(connections): follow-ups to mutual TLS (#630) by @KIvanow in #633
• Isolate the license spec from the real data directory by @jamby77 in #632
• Read replica lag from the primary in the health gate by @jamby77 in #619
• Detect replication divergence per replica by @jamby77 in #618
• Detect retry amplification and attribute it to clients by @jamby77 in #620
• Track connection churn and build incident timelines by @jamby77 in #621
• fix: stop BetterDB's own commands from triggering error and retry alerts by @KIvanow in #645
• fix: mTLS review follow-ups by @KIvanow in #646

Full Changelog: v0.53.0...v0.54.0