Skip to content

v0.34.0

Choose a tag to compare

@KIvanow KIvanow released this 31 Jul 07:48
· 27 commits to master since this release
5a94587

Three new memory & load anomaly detectors, plus stable telemetry instance identity.

This release adds three detectors targeting silent memory-budget and single-thread-saturation conditions - non-dataset overhead squeezing the maxmemory budget, event-loop saturation that raw CPU% hides, and copy-on-write OOM risk during forked saves - and fixes anonymous-telemetry instance counting so ephemeral containers stop looking like new installs.

✨ New anomaly detectors

  • Non-dataset memory overhead (MEMORY_OVERHEAD) - a maxmemory budget is meant to bound the dataset, but operational overhead (client output buffers, the replication backlog and per-replica buffers, the AOF rewrite buffer, scripts/functions, cluster-bus links) is charged against the same budget. When it grows it silently shrinks the room for user data and, under an eviction policy, drives eviction of keys that would otherwise fit. WARNING at ≥30% of maxmemory, CRITICAL at ≥50% - or a CRITICAL when eviction is active and overhead has grown larger than the dataset itself, so a normal capacity-bound cache evicting by design is not mistaken for a problem. Names the dominant overhead component and gives a targeted remedy (e.g. tune the relevant client-output-buffer-limit, lower repl-backlog-size); escalation-only hysteresis. (upstream valkey#1792)

  • Event-loop load saturation (LOAD_SATURATION) - on a largely single-threaded server, raw CPU% is a poor busyness indicator. This measures the fraction of wall-clock time the event loop actually spends working (instantaneous_eventloop_duration_usec × instantaneous_eventloop_cycles_per_sec). WARNING at ≥80% busy, CRITICAL at ≥95%, each requiring the band to hold for 3 consecutive polls so a momentary batch/pipeline burst doesn't alert. When CPU% reads misleadingly low next to the busy fraction, the alert calls that out explicitly and points at slow commands / big O(N) operations / a single-thread bottleneck rather than "add more CPU". Graceful no-op on servers that don't expose the event-loop metrics. (upstream valkey#2055)

  • Fork / copy-on-write OOM risk (FORK_MEMORY_RISK) - BGSAVE and AOF rewrite fork the server; writes arriving while the child runs dirty copy-on-write pages, so RSS climbs toward used_memory + bytes-written-during-save and can cross available RAM, letting the OOM killer terminate the server mid-save. Two paths: live (a save in progress, projecting used_memory_rss + current COW against total system memory) and projected (no save running, but a sustained write rate makes the next fork risky - gated on the per-poll write rate so it clears when writes stop, and estimated from the larger of the last RDB or AOF COW so AOF-only deployments are covered). WARNING at ≥80% of system memory, CRITICAL at ≥90% (live), with a slow-fork note from latest_fork_usec and advice to verify vm.overcommit_memory and RAM headroom. (upstream valkey#3609)

📡 Telemetry

  • Stable anonymous instance identity + ephemerality signals - the anonymous instance id was re-derived every boot from a hash including HOSTNAME, so ephemeral containers (random per-run hostname) each looked like a brand-new install and inflated distinct-instance counts. The id is now persisted under data/instance-id and reused across restarts, with a fallback to per-boot derivation when the data dir isn't writable, and a random UUID (instead of a shared constant hash) when no infrastructure identifiers are set. The ping also carries ci, container, and hostnameIsContainerId so throwaway boots (CI, e2e spin-ups, restart loops) can be segmented from real installs. (#350)

⬆️ Upgrade notes

  • Fully backwards compatible - no config or schema migrations. The three detectors are additive and emit over the existing webhook, OTLP, and dashboard channels automatically; they add three metric-type labels (memory_overhead, load_saturation, fork_memory_risk).
  • FORK_MEMORY_RISK compares projected RSS against total_system_memory, so it is most meaningful on memory-limited instances (a container/cgroup memory limit, where Valkey reports that limit). On an unconstrained host that reports full system RAM it will rarely fire.
  • Telemetry now writes an anonymous id to data/instance-id when the data directory is writable (still governed by BETTERDB_TELEMETRY; no new data is collected beyond the segmentation flags above).
  • Docker: betterdb/monitor:0.34.0

What's Changed

  • feat(telemetry): persist instance id and add ephemerality signals by @KIvanow in #350
  • feat(anomaly): memory-overhead, load-saturation and fork-OOM detectors by @KIvanow in #351

Full Changelog: v0.33.0...v0.34.0