V1 [OPS] Disk space monitoring and guardrails

[austin-barrington]

Context

If the WAL directory, metadata directory, or chDB data path fills up, there are no checks before I/O operations. This leads to:

• RocksDB crashes with unhandled I/O errors
• chDB flush failures
• Silent data loss if the WAL cannot append

Production deployments need proactive monitoring and protective action before disk exhaustion.

Deliverables

• Add periodic disk space checks (via fs::statvfs or equivalent) on the WAL, metadata, and chDB data directories.

• Expose available space as a Prometheus gauge: hyperbytedb_disk_free_bytes{path="wal|meta|chdb"}.

• Add configurable thresholds in a new [disk] section:

  Key	Type	Default	Description
  check_interval_secs	integer	60	How often to check disk space
  warn_threshold_mb	integer	1024	Alert when any data dir has < this much free space
  readonly_threshold_mb	integer	256	Enter read-only mode when below this threshold
  readonly_on_error	boolean	true	Enter read-only mode on I/O errors to prevent data loss

• When free space drops below readonly_threshold_mb:

  • Set node state to Draining-like read-only mode (reject writes with 507, serve queries)
  • Log a critical message every check interval
  • Increment hyperbytedb_disk_readonly_events_total counter

• Add a startup check: if any data directory has less than warn_threshold_mb free, log a warning.

References

• docs/user-guide/troubleshooting.md — no disk space section exists
• docs/user-guide/v1-stable-scope.md — production topology expectations

Priority

P0 — Production requirement, data-loss risk

[austin-barrington]

Fixed: DiskMonitor in disk_monitor.rs with statvfs, hyperbytedb_disk_free_bytes metric, configurable warn/readonly thresholds, write handler gate, startup check.