Skip to content

v0.30.0

Choose a tag to compare

@KIvanow KIvanow released this 22 Jul 15:30
f31b7dc

Valkey Cluster V2 (Raft) health monitoring — quorum-loss and leadership-churn detection, plus a live Raft Health panel.

✨ Features

  • Cluster V2 (Raft) health detection. The anomaly engine now understands Valkey's Raft-based clustering (cluster-protocol raft) and watches the two failure modes that mode is uniquely prone to:
    • Quorum loss / no reachable leader (CRITICAL). Calibrated against a live 3-node Raft cluster: a node that has lost its majority keeps reporting cluster_state:ok (its slots stay "covered") and never inflates its term (pre-vote), so cluster_state is not a reliable signal. The detector instead keys on the node repeatedly seeking a leader it can't elect (candidate/pre-candidate) while its commit index stays frozen. Detection windows are derived from the cluster's cluster-node-timeout, so they hold across any tuning, and a healthy failover or a one-off election blip is correctly ignored. Incidents auto-resolve when quorum returns and stay pinned (surviving a dismiss) while the outage is live.
    • Election churn (WARNING). Repeated completed elections (term climbing) in a short window flag flapping leadership from an unstable/overloaded primary or a split network.
  • Cluster Overview → "Cluster V2 · Raft Health" panel. Shows the connected node's Raft role, current term, commit / last-applied / log-entries progress (with an apply-lag callout), and the elected leader. It self-hides on legacy gossip clusters, surfaces "Electing" while a node is seeking a leader, distinguishes a slot/serving cluster_state:fail from a true "no quorum", and pins the live outage from the authoritative detector.
  • Critical "Raft cluster has lost quorum" banner on the Anomaly dashboard, with a Raft-specific remediation runbook (majority math, don't force-promote, don't wipe survivors), plus a friendly "Raft Health" metric label.
  • Gossip-mode gating. On a Raft cluster the gossip-era topology detectors (duplicate-primary #2261, stuck-replica #2090) are skipped — topology is consensus-managed there — while remaining fully active on classic gossip clusters, even across transient CLUSTER INFO blips.
  • OTLP telemetry mirroring (#319). The monitor can mirror its existing Prometheus metrics and discrete monitoring events to an OTLP endpoint. Opt-in and no-op unless OTEL_EXPORTER_OTLP_ENDPOINT / OTEL_TELEMETRY_ENABLED are set.
  • Agent Memory consolidate() modes (#324). consolidate() now takes a required mode: 'summary' | 'facts', surfaces unmatched tombstones, and ships recall tuning. (Also released standalone as agent-memory-v0.7.0.)

⬆ Upgrade notes

  • Fully backwards compatible; no config changes. On non-Raft (gossip/standalone) deployments nothing changes.
  • Raft health detection runs inside the anomaly engine, which is a Pro/Enterprise feature — it activates with a valid license; Community builds are unaffected.
  • OTLP mirroring is off by default — enable it only via the OTEL_* env vars above.
  • Docker: betterdb/monitor:0.30.0

Full Changelog: v0.29.0...v0.30.0

What's Changed

  • feat(otel): mirror Prometheus metrics + discrete events to OTLP by @jamby77 in #319
  • feat(agent-memory): merge consolidate() modes, surface unmatched tombstones, tune recall by @KIvanow in #324
  • feat(anomaly): Raft cluster (Cluster V2) health detection + gossip-mode gating by @KIvanow in #326

Full Changelog: v0.29.0...v0.30.0