v14.7.0 (2026-07-16) - "Vigilant II"

[4nonX]

High Availability stabilization: Failover logic and split-brain guards move to beta status (stable, validated, field-ready). Load-testing framework, CTDB clustering, operator runbooks, monitoring, and hardware matrix are experimental additions; core HA is no longer demo-ware. PostgreSQL failover validated under sustained I/O workload, CTDB clustering enables SMB client persistence through failover, comprehensive runbooks document recovery for all 8 failure scenarios, Prometheus alerting and Grafana monitoring provide real-time cluster visibility, SCSI-3 PR compatibility matrix preempts fencing failure surprises.

High Availability Enhancements

• PostgreSQL HA Load Testing: Automated load-test framework (nixos/tests/ha-failover-load-test.nix) validates Patroni failover under sustained workload (pgbench 100 concurrent connections). Injects network partition mid-run and validates failover <60 seconds, cluster converges to single primary (no split-brain), database consistency (row count validated post-failover). EXPERIMENTAL: infrastructure ready, production validation pending.

• CTDB Clustering for SMB HA (NEW): Enables Samba clustering so SMB clients survive HA failover without disconnecting or losing byte-range locks. Configuration via Settings → High Availability → CTDB Clustering. Requires: HA enabled, Samba enabled, ZFS dataset for lock database. Automatic failover of public IPs (Keepalived + CTDB coordination). Frontend UI for configuration, real-time cluster status monitoring. Documentation: HA-CTDB-SETUP.md. EXPERIMENTAL: tested in lab, production validation in progress.

• Prometheus HA Alerting (NEW): 20+ production-grade alert rules for cluster health monitoring:

  • Failover readiness: quorum lost, peer unreachable, fencing disabled, witness unreachable
  • Replication health: lag high, replication stalled, Patroni no leader, WAL lag high
  • Safety nets: watchdog disabled, watchdog stuck, split-brain detection
  • System health: daemon unresponsive, etcd unreachable, SCSI-3 PR fencing failures
    Each alert includes detection criteria and remediation guidance. Integration: prometheus/ha-alerts.yml.

• Grafana HA Cluster Dashboard (NEW): Real-time monitoring dashboard (grafana/dashboards/ha-cluster.json) with:

  • Cluster quorum status, node roles, recent failover timeline
  • Peer heartbeat tracking (missed beats per node)
  • Replication lag graphs with color-coded thresholds
  • Node availability heatmap over time
  • Failover suppression flags (maintenance, hysteresis, subordinate mode)
  • CTDB cluster status and public IP ownership
  • Automated impact measurement (how long did failover take?)

• HA Operator Runbooks (NEW): Comprehensive guide (docs/admin/HA-FAILURE-MODES.md, 4000+ lines) covering 8 real failure scenarios:

  1. Node unreachable (network partition detection and recovery)
  2. Failover stalled (witness/maintenance/hysteresis blocking)
  3. Split-brain detected (emergency data reconciliation)
  4. Witness down (quorum witness loss)
  5. Replication lag high (bottleneck identification)
  6. Patroni no leader (etcd quorum restoration)
  7. SCSI-3 fencing failed (PR support validation)
  8. Subordinate mode (stale data catch-up)
     Each scenario includes: root causes, detection symptoms, step-by-step recovery commands, validation checklist, escalation procedures. Post-failover validation checklist included.

• SCSI-3 PR Hardware Matrix (NEW): Documented hardware compatibility (docs/reference/HA-HARDWARE-MATRIX.md):

  • Tested working hardware (enterprise SANs, JBODs, SAS controllers with SCSI-3 PR support)
  • Known issues and firmware workarounds
  • NVMe-oF support (enterprise targets yes, consumer targets no)
  • Testing procedures and fallback strategies
  • Community contribution process for undocumented hardware

HA Status Changes

• Failover Logic: Beta (stable, battle-tested)
• Split-Brain Guards: Beta (all 6 guards validated by CI)
• Load-Testing Framework: Experimental (infrastructure complete, field validation pending)
• CTDB Clustering: Experimental (optional module for SMB failover support)
• Monitoring: Beta (20+ alerts, real-time dashboard)
• Operator Runbooks: Beta (comprehensive recovery procedures for all failure scenarios)

Database Changes

• New Tables (CTDB configuration):
  • ctdb_config - CTDB settings (enable, pool, dataset, timeouts, log level)
  • ctdb_public_addresses - VIPs managed by CTDB across cluster nodes

API Changes

• New Endpoints (CTDB):
  • GET /api/ha/ctdb/configure - Get CTDB configuration
  • POST /api/ha/ctdb/configure - Save CTDB configuration (requires system:admin)
  • GET /api/ha/ctdb/status - Real-time CTDB cluster status
  • GET /api/ha/ctdb/databases - CTDB database replication status

UI Changes

• Settings → High Availability → CTDB Clustering (NEW): Configuration panel for:
  • Enable/disable CTDB
  • Configure data pool and dataset
  • Configure node timeout and recovery timeout
  • Manage public IP addresses (VIPs)
  • Monitor real-time cluster status and IP ownership

Known Limitations & Future Work

• PostgreSQL HA: Load-testing framework in place; field validation pending before v15.0.0 GA
• CTDB: SMB failover now works; requires broader field validation before v15.0.0 GA
• Hardware Matrix: Community-contributed (submit test results for your hardware)
• Monitoring: Alerts available for configuration; integration with PagerDuty/OpsGenie recommended for production

Documentation Changes

• Added: docs/admin/HA-FAILURE-MODES.md (operator runbooks, all 8 scenarios)
• Added: docs/admin/HA-CTDB-SETUP.md (CTDB installation and operational guide)
• Added: docs/reference/HA-HARDWARE-MATRIX.md (SCSI-3 PR compatibility)
• Updated: docs/admin/HIGH-AVAILABILITY.md (linked to new failure modes guide)
• Added: prometheus/ha-alerts.yml (20+ alert rules)
• Added: grafana/dashboards/ha-cluster.json (cluster monitoring dashboard)

No Breaking Changes

All changes are backward-compatible. HA remains off-by-default. Existing v14.6.0 deployments can upgrade without changes. CTDB is optional (enabled by setting services.dplaneos.ctdb.enable = true). New Prometheus alerts and Grafana dashboard are optional integrations.

---

---

🚀 Installation

curl -fsSL https://get.dplaneos.io | sudo bash -s -- --version 14.7.0

---

This discussion was created from the release v14.7.0 (2026-07-16) - "Vigilant II".