Skip to content

Persist metric history to disk (survive restarts)#44

Merged
LarsLaskowski merged 1 commit into
mainfrom
claude/issue-2-poczw6
Jul 14, 2026
Merged

Persist metric history to disk (survive restarts)#44
LarsLaskowski merged 1 commit into
mainfrom
claude/issue-2-poczw6

Conversation

@LarsLaskowski

@LarsLaskowski LarsLaskowski commented Jul 13, 2026

Copy link
Copy Markdown
Owner

Summary

RingBuffer history was in-memory only, so every service restart or reboot wiped the dashboard sparklines. The collector now periodically snapshots all ring buffers to a single compact binary file and reloads it on startup, so GET /api/v1/metrics/history spans restarts.

  • Binary snapshot format (PIMH v1) in a new internal/collector/persist.go: magic + version + per-series records (kind byte, mountpoint/interface key, points as Unix-ms timestamp + float64 pairs), all little-endian. Decoding is bounds-checked (magic, version, series/key/point-count limits) so a corrupt or truncated file is logged and ignored rather than crashing or over-allocating; unknown series kinds are skipped for forward compatibility.
  • Atomic writes: temp file in the same directory + fsync + rename, so a crash mid-write never leaves a partial file.
  • Flush triggers: the existing slow tick in Collector.Run (default every 15 min, keeping SD-card wear low) plus a final flush on clean shutdown; main now waits for the collector goroutine so that last flush completes before the process exits.
  • Load on startup: points older than history_window_minutes are trimmed, and each buffer keeps at most its capacity (newest points win). Map series that end up empty after trimming are omitted, matching live-collection behavior.
  • RingBuffer.Fill: the import counterpart to Snapshot, restoring buffer contents oldest-first.
  • New config keys: history_persist_enabled (default true) and data_dir (default /var/lib/pimonitor); validation rejects an empty data_dir when persistence is enabled. Persistence failures are warnings only — metric collection and the API keep working (verified against an unwritable data_dir).
  • systemd: StateDirectory=pimonitor (creates /var/lib/pimonitor owned by the service user) + ReadWritePaths=/var/lib/pimonitor so the hardened unit (ProtectSystem=strict) can write its state dir.

No /api/v1/... response shape changes.

Verified end-to-end on this branch: started the service, sampled /api/v1/metrics/history, sent SIGTERM, restarted, and confirmed the pre-restart points were returned after the restart (restored timestamps are truncated to millisecond precision by design). Also exercised: corrupt history.bin (warns, starts empty), unwritable data_dir (warns, keeps serving), persistence disabled (writes nothing), empty data_dir with persistence enabled (rejected at startup). Note: verified on x86-64 Linux (plus arm/arm64 cross-compile); not yet run on physical Pi hardware.

Related Issue

Closes #2

Checklist

  • Tests added/updated for the change (go test ./... passes locally)
  • go vet ./... and golangci-lint run are clean
  • Documentation updated if this changes the REST API (docs/API.md),
    configuration (README.md, packaging/pimonitor.example.yaml), or
    installation/packaging (packaging/install.sh, systemd units)
  • No breaking change to /api/v1/... response shapes, or a new API
    version was introduced instead

RingBuffer history was in-memory only, so every service restart or
reboot wiped the dashboard sparklines - on a Pi that reboots for
updates this made history empty right when it was most useful.

The collector now snapshots all ring buffers to a single compact
binary file (data_dir/history.bin, PIMH v1 format) via an atomic
temp-file+rename write, flushed on the existing slow tick and once
more on clean shutdown to minimize SD-card wear. On startup the file
is reloaded, dropping points older than history_window_minutes.

A binary format was chosen over JSON per issue #2 for a smaller
footprint and less SD-card wear; corrupt or truncated files are
detected (magic/version/bounds checks) and ignored with a warning so
the service always starts.

New config keys: history_persist_enabled (default true) and data_dir
(default /var/lib/pimonitor). The systemd unit gains
StateDirectory=pimonitor and ReadWritePaths=/var/lib/pimonitor so the
hardened service can write its state directory. No API shape change.

@LarsLaskowski LarsLaskowski left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed against the project's Go/security/API-stability conventions (/review-pr). Everything checks out — no blocking issues.

Verification (on the PR branch)

  • go build ./..., go vet ./... — clean
  • go test ./... and go test -race on the touched packages — all pass
  • golangci-lint run — 0 issues, gofmt clean

Checklist review

  • Parsing robustness: the PIMH decoder is well done — sticky-error bounds checking, magic/version/series/key-length/point-count limits, and (importantly) point slices are only allocated after take() confirms the bytes exist, so allocation is bounded by actual file size, not by attacker-declared counts. maxPointsPerSeries * pointSize (2²⁵) also fits in a 32-bit int, so the GOARM=6 builds are safe. Negative-path tests cover bad magic, wrong version, truncation, trailing data, and unknown kinds.
  • Error handling: missing file, corrupt file, and unwritable data_dir all degrade to warnings with the collector still starting — matches the project's graceful-degradation convention.
  • Resource leaks: temp file is closed/removed on every failure path of writeFileAtomic; main now waits for the collector goroutine (bounded by the shutdown context) so the final flush isn't orphaned.
  • REST API stability: no /api/v1/... shape change; docs/API.md wording updated accordingly.
  • Dependencies: none added. Language: all English. Command execution: none.
  • Config/packaging: defaults-then-YAML load order means history_persist_enabled: false works; StateDirectory= + ReadWritePaths= is the right pairing for ProtectSystem=strict.

Non-blocking observations

  1. Directory durability: writeFileAtomic fsyncs the file but not the parent directory after the rename, so on power loss the latest snapshot may be lost (the previous one survives). For best-effort sparkline history on an SD card that's a reasonable trade-off against extra wear — fine as is, just noting it's a known gap.
  2. Encode side doesn't enforce maxSeries: a host with >4096 mountpoints/interfaces would write a file its own decoder rejects on next start. Unreachable on any real Pi, so not worth code; mentioning only for completeness.
  3. Brief empty-history window at startup: the HTTP server can serve /api/v1/metrics/history before the collector goroutine finishes loadHistory(). Benign (same as pre-PR behavior of starting empty) and properly locked, so no race — just a few milliseconds where restored points aren't visible yet.

Nice, focused change — the format comment doubling as a spec and the SD-wear-conscious flush cadence are both appreciated.

@LarsLaskowski
LarsLaskowski merged commit 447f95e into main Jul 14, 2026
3 checks passed
@LarsLaskowski
LarsLaskowski deleted the claude/issue-2-poczw6 branch July 14, 2026 10:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A1: Persist metric history to disk (survive restarts)

2 participants