Skip to content

v0.55.0

Latest

Choose a tag to compare

@github-actions github-actions released this 10 Nov 15:42
· 23 commits to master since this release
ee9d173

v0.55.0

v0.54.0 added replication divergence, retry amplification and incident timelines, but they only saw the nodes you had registered as connections, and incidents only existed in the API and MCP. This release closes both gaps. A cluster registered by its seed, or a Sentinel connection with data-node credentials, now gets divergence, retry amplification, reshard skew, backlog coverage, resync escalation, fork memory risk and output-buffer checks on every primary. The anomaly dashboard has an Incidents tab with a detail page per incident. Replication gap, backlog coverage, connection churn, retry amplification state and open incidents are now Prometheus gauges you can graph.

The release also fixes clusters behind a hostname that resolves to several nodes (the Valkey Kubernetes operator's headless Service), where slow log entries were dropped without an error and slot stats showed one shard.

Upgrades are drop-in apart from the items below. New columns are added on startup on SQLite and Postgres.

  • No downgrade after upgrading. slow_log_entries and command_log_entries get a source_run_id column that joins their unique keys. An older build can't write to the migrated tables.
  • Slot series removed. betterdb_cluster_slot_expires, betterdb_cluster_slot_reads_total and betterdb_cluster_slot_writes_total are gone. They were always 0, because Valkey never returns those fields. See Cluster slot stats for what replaces them.
  • Anomaly summary gauges are labelled per connection. betterdb_anomaly_by_metric, betterdb_anomaly_by_severity, betterdb_anomaly_events_current, betterdb_correlated_groups_by_pattern and betterdb_correlated_groups_by_severity were all exported under the default connection's label. Queries that filtered on that one label now see only that connection's counts.
  • One-time slow log replay. The first poll after upgrading re-reads up to 128 entries per log type per connection, because older rows carry no run id.
  • Minimum ACL: atomic slot migrations need +cluster|getslotmigrations (Valkey 9+). Without it they don't show.

Highlights

Detection on every primary of a cluster or Sentinel deployment (#654, #655, #656, #659, #673, #677)

Before, a cluster had to have each primary registered as its own connection, and Sentinel connections weren't covered at all. Now each poll fans out to the other primaries:

  • Cluster: every primary in CLUSTER NODES, through the cluster-discovery node clients, with the seed's TLS settings.
  • Sentinel: every primary in SENTINEL MASTERS, but only when the connection has its own data-node username or password. Sentinel credentials are never sent to an address Sentinel reports.
  • Which checks run per node: replication divergence, retry amplification, reshard skew, backlog coverage, resync escalation, fork memory risk and replica output-buffer pressure.
  • Scoping: events keep the seed's connectionId and add nodeAddr (new node_addr column). Incidents group by node, so events from different shards don't chain. The webhook instance is the shard primary.
  • Failover: state is kept per node. A node's open alerts close when it leaves the topology, is claimed by another connection, or is registered as its own connection.
  • Limits:
    • Up to 64 primaries per connection, 2 s per read.
    • Z-score detectors (memory, ops/sec and the rest) still run only on registered connections, so divergence_under_pressure can't match on a non-seed shard.
    • Retry amplification on a fan-out node has no client names or reconnect counts (topContributors: null), because BetterDB doesn't read CLIENT LIST from it. Register the node as a connection to get them.
    • Reshard skew on a fan-out shard uses ops/sec and memory only, with no CPU. It can't be tagged RESHARD_PRESSURE unless BetterDB has a write P99 for it, which it reads from INFO latencystats at most every 30 s.

Coverage on the dashboard (#673). When the fan-out covers less than it should, the anomaly dashboard says so and says why. For example, "Sentinel data nodes not covered" names the setting to fill in, and "3 of 7 primaries checked (2 unreachable, 1 refused the credentials, 1 past the 64-primary cap)" covers a partial read. Nothing shows when coverage is full. Before, these cases left at most a log line, and primaries past the cap were skipped without one.

Replica-side cross-check (#659, #679). Divergence used only the primary's view of each replica, which comes from the replica's acknowledgements. When the replica can be read, BetterDB now reads its own slave_repl_offset and stores it next to the primary's view on each window.

  • Replica caught up: if its link is up and it's under 1 MiB behind, the alert is skipped. The primary was looking at stale acknowledgements.
  • Replica agrees: the alert goes out and says whether the replica has data it hasn't applied (slow apply) or has applied everything it received (the stream isn't reaching it). The webhook gets a replicaView field.
  • Views disagree: when the two gaps differ by at least 1 MiB and 5 s of writes for 3 windows in a row, a WARNING is raised.
  • Recovery: recovered waits until the replica's own view shows it caught up with its link up.
  • How the replica is reached: through its own connection if it's registered, by node id on a cluster, or by address with the data-node credentials on Sentinel. For a standalone primary, BetterDB dials the replica with the primary's credentials, but only at an IP that CLIENT LIST TYPE replica shows a replica connected from.
  • API and MCP: GET .../replication-divergence and get_replication_divergence return both views. The UI is unchanged.

Incident timeline in the dashboard (#662, #676, #683, #684, #685)

Incidents were only in GET /anomaly/incidents and the MCP tool. The anomaly dashboard now has Overview and Incidents tabs (?tab=incidents).

  • List: pattern, scope (connections and nodes), start, duration and status for the dashboard's date range, refreshed every 30 s.
  • Detail page at /anomalies/incidents/:incidentId:
    • the events in order, with when each signal returned to baseline, or that it hasn't, or that recovery is unknown
    • each matched sequence rule with its diagnosis, steps and recommendations, with timeline events tagged by step
    • a "recovery debt" callout on degraded incidents, naming the signals still away from baseline
  • From a correlated group: an expanded group on the Anomalies page has a "View incident" link. The new endpoint GET /anomaly/incidents/by-event/:anomalyId?at=<ms> finds the incident that holds an event now, since incident ids change when events regroup.
  • The list and detail page load through one shared window, so they no longer disagree about where an incident starts or ends.
  • One failed lookup no longer fails the whole list with "Could not load incidents".

Prometheus gauges for replication, churn, retries and incidents (#660, #674)

These detectors only reached Prometheus as event counters, so you could count alerts but not graph what was behind them. All new gauges have a connection label. node is the primary's host:port, including nodes reached through a cluster or Sentinel connection.

Gauge Labels Value
betterdb_replication_gap_bytes node, replica latest divergence window gap
betterdb_replication_seconds_behind node, replica latest window; absent when unknown
betterdb_replication_divergence_level node, replica 0 none, 1 warning, 2 critical
betterdb_replication_write_rate_bytes_per_second node smoothed write rate
betterdb_replication_backlog_coverage_seconds node backlog size ÷ write rate
betterdb_connections_received_per_second node new connections per second
betterdb_connection_churn_per_op node new connections per op
betterdb_retry_amplification_state node 0 none, 1 candidate, 2 amplifying
betterdb_incident_active pattern, status open incidents (active, recovering, degraded)
  • They're exported with the full profile only, and also through OTel.
  • Replication and retry gauges update on every poll, for the seed and every fan-out node. Series for a replica that drops out or a node that leaves are removed.
  • betterdb_retry_amplification_state shows the live state. Whether an episode was transient is only known after it ends, so it isn't a gauge value.
  • Incidents are counted over the last hour, under every connection an incident spans.
  • docs/prometheus-metrics.md has the cardinality and example PromQL.

Recommendations with numbers (#678)

Recommendations used to be fixed text such as "Raise repl-backlog-size". Three detectors now compute a target from the instance's own numbers:

Finding Setting Target
backlog_coverage_risk repl-backlog-size 75 s of writes at the current rate, past the 72 s at which the warning clears
repl_buffer_pressure client-output-buffer-limit replica 2 × the omem of the replica that raised it
fork_memory_risk total_system_memory RAM at which the projected fork peak sits at 70%

The incident detail page shows each as setting: current → target, and get_incident_timeline returns them. The retry amplification recommendation now names the top clients, for example Top clients: name checkout (cmds 4.2×, 12 reconnects). Values aren't stored. After the anomaly cache TTL (1 hour by default), rebuilt incidents show the text without them.

More incident sequence rules (#657, #680)

Rule Steps
divergence_under_pressure ops/sec or input ↑ → replication divergence → memory or replica buffer pressure
reshard_under_writes reshard skew → replication divergence
reshard_divergence_pressure reshard skew → replication divergence → memory or replica buffer pressure
retry_storm P99, error replies or connection churn → retry amplification
backlog_exhaustion backlog coverage risk → resync escalation
  • reshard_divergence_pressure replaces the two single rules when they share an event, so the overlap shows one diagnosis, not two.
  • Steps that need a rise no longer match a drop. A fall in ops/sec, memory or churn won't fill a "rising" step.
  • A later step joins an incident only when every earlier step happened, in order, within the rule's span. Before, only the step right before it had to be there.
  • docs/anomaly-detection.md has the full table, with optional signals and spans.

Cluster fixes for multi-node seeds (#651)

Found while running Monitor against a cluster run by the Valkey operator (v0.8.0, Valkey 9.0.0), where the natural seed is a headless Service that resolves to every pod.

  • Auto-registration churn: each 30 s sync reached a random pod and retired it as "the seed", then brought it back on the next sync (13 retires in 8 minutes on a stable cluster). BetterDB now resolves the seed host and checks which nodes it points at. A ClusterIP, load balancer VIP or NAT endpoint is detected by which node answers across syncs.
  • Slow log and command log entries dropped: log IDs are per node, so after the seed moved to another pod, entries with IDs already stored from the previous node were dropped (16+ minutes of Saved 0). Entries are now keyed by the server's run_id.
  • Slot migrations: Valkey 9 atomic slot migration (CLUSTER MIGRATESLOTS), which the operator uses, is now detected on every primary and shown with its slot range and phase.
  • Health probe: new GET /api/health/live answers without touching any database. The Docker image HEALTHCHECK uses it. The Helm chart probes still use /api/health and will move in a later chart release.
  • Cluster discovery: a node counts as unhealthy only when it's flagged fail.
  • AI observability: a MOVED reply on the discovery registry read is logged once per client, not every 15 s.

Cluster slot stats

  • Every shard (#651): CLUSTER SLOT-STATS only covers the slots of the node that answers, so a cluster showed one shard (5462 of 16384 slots). BetterDB now asks every primary.
  • Hot slots by CPU (#681): with cluster-slot-stats-enabled, slots are ranked by CPU as well as key count, since a hot slot is usually a few busy keys and rarely makes the top 100 by keys. New counters betterdb_cluster_slot_cpu_usec_total and betterdb_cluster_slot_network_bytes_{in,out}_total (use rate()). The Cluster page's Top Slots card gets a Keys/CPU toggle, and the cluster-slots Grafana board gets "Hottest slots by CPU" and "Busiest slots by traffic" panels. With slot stats off (the Valkey default), a cluster connection exports 1 series per slot instead of 4.

Retry amplification

  • One client per connection (#675): a client captured before it ran CLIENT SETNAME was counted under its IP in one capture and its name in the next, which split one reconnect storm across two clients. Each connection is now one client for the whole episode. Reconnects are counted by when the connection opened, and long episodes compare their newest 30 minutes with the 30 minutes before they started.
  • Back-to-back storms (#634): the baseline for a storm that starts soon after the last one skips readings from the earlier one. If no clean readings are left, BetterDB reports attribution as unavailable instead of saying no client stood out.

Webhooks

  • Sequence alerts (#686, Pro): a new anomaly.sequence.detected event is sent when an incident sequence rule matches, and again when a compound rule replaces it with a more severe diagnosis. It carries the diagnosis, recommendations and member events. Nothing is sent on recovery. It goes out under the connection of the first event, so a webhook scoped only to another connection in the same sequence doesn't get it.
  • Incident links: with FRONTEND_URL set, anomaly.detected and anomaly.sequence.detected carry an incidentUrl, and "View in BetterDB" in Slack and Discord opens the incident instead of the anomalies page.
  • One alert per replica stall (#653): when the primary and a stalled replica are both monitored, a webhook subscribed to both events used to get replication.lag from the replica and replication.divergence (not_acking) from the primary for the same stall. Now whichever arrives first wins for that webhook. Other divergence reasons are always sent.

Other fixes

  • active-replica (#647): lag, link health, resync-loop and config/ACL drift checks now treat KeyDB/Valkey active-replica nodes as replicas. Failover alerts don't, because an active-replica stays writable. A malformed offset such as offset=12abc now reads as missing, not as 12.
  • Alert logic no longer reads message text (#648): latency recovery and resync escalation adoption read stored fields (kind, replicaAddr) instead of parsing the alert message. Existing command_p99 rows are backfilled on startup.
  • Incident history (#634, #649): a long, noisy incident keeps the first event of every signal when it hits the 200-event cap, so sequence rules still match. Latency regressions keep their place in the timeline after a restart. A legacy resolved row without resolvedAt reports recovery as unknown, not instant.
  • Absolute-threshold alerts (#634) say "above threshold" and name the threshold, instead of "spike detected … below normal".

Docs

  • Minimum ACL (#650): new docs/minimum-acl.md with a read-only monitoring user that has no key access, add-ons per feature, and the commands that change data and what triggers each.

MCP

  • @betterdb/mcp 1.6.0 shipped as a Claude Code plugin (#661, #658, #687): claude plugin marketplace add https://raw.githubusercontent.com/BetterDB-inc/monitor/master/.claude-plugin/marketplace.json, then claude plugin install betterdb@betterdb. It includes seven skills for triage, slow commands, hot keys, memory, latency, durability and cache misses.

What's Changed

  • Fix retry baselines, alert wording and incident ordering follow-ups by @jamby77 in #634
  • Load the repo env file in tests and block only license keys by @jamby77 in #635
  • Read alert kind and replica from stored fields, not message text by @jamby77 in #648
  • Tidy retry amplification and incident timeline review follow-ups by @jamby77 in #649
  • Share one parser for replica lines in INFO replication by @jamby77 in #647
  • Send one webhook per replica stall when both nodes are monitored by @jamby77 in #653
  • Document the minimum ACL for monitoring by @jamby77 in #650
  • Detect replication divergence on every shard and Sentinel group by @jamby77 in #654
  • Evaluate retry amplification on each cluster node by @jamby77 in #655
  • Judge reshard skew with only the seed registered by @jamby77 in #656
  • Add four more incident sequence rules by @jamby77 in #657
  • Cross-check replication divergence against the replica's own offset by @jamby77 in #659
  • Export replication, churn, retry and incident signals as Prometheus gauges by @jamby77 in #660
  • Add an incident timeline view to the anomaly dashboard by @jamby77 in #662
  • Add BetterDB MCP skills for the plugin listing by @KIvanow in #658
  • fix(prometheus): label anomaly summary gauges per connection by @KIvanow in #674
  • feat(anomaly): surface fan-out coverage for Sentinel and unreachable primaries by @KIvanow in #673
  • Keep the incident list loading when a next-firing lookup fails by @jamby77 in #676
  • fix(anomaly): one client identity across CLIENT SETNAME in retry attribution by @KIvanow in #675
  • Run backlog, recovery, fork and output-buffer detectors on fan-out primaries by @jamby77 in #677
  • Track the replica's own offset in replication divergence by @jamby77 in #679
  • Combine overlapping reshard, divergence and memory-pressure diagnoses, and make sequence steps direction-aware by @jamby77 in #680
  • feat(anomaly): recommendations with computed values by @KIvanow in #678
  • Filter stored anomaly events by node address and commands by @jamby77 in #682
  • Fix incident loading state, page copy and recovery-debt wording by @jamby77 in #684
  • Load the incident list and detail page through one shared window by @jamby77 in #683
  • Add Claude Code plugin for the BetterDB MCP server by @jamby77 in #661
  • fix: monitor clusters whose seed host fronts several nodes (Kubernetes operator) by @KIvanow in #651
  • Export per-slot CPU and traffic, and rank slots by CPU as well as keys by @KIvanow in #681
  • Add listing links to the MCP plugin manifest by @KIvanow in #687
  • Link correlated groups to their incident by @jamby77 in #685
  • Send an alert when an anomaly sequence is matched by @jamby77 in #686

Full Changelog: v0.54.0...v0.55.0