Skip to content

v14.7.0 (2026-07-16) - "Vigilant II"

Latest
Latest

Choose a tag to compare

View all tags
@github-actions github-actions released this 16 Jul 21:37
v14.7.0
29c4a1f

High Availability stabilization: Failover logic and split-brain guards move to beta status (stable, validated, field-ready). Load-testing framework, CTDB clustering, operator runbooks, monitoring, and hardware matrix are experimental additions; core HA is no longer demo-ware. PostgreSQL failover validated under sustained I/O workload, CTDB clustering enables SMB client persistence through failover, comprehensive runbooks document recovery for all 8 failure scenarios, Prometheus alerting and Grafana monitoring provide real-time cluster visibility, SCSI-3 PR compatibility matrix preempts fencing failure surprises.

High Availability Enhancements

  • PostgreSQL HA Load Testing: Automated load-test framework (nixos/tests/ha-failover-load-test.nix) validates Patroni failover under sustained workload (pgbench 100 concurrent connections). Injects network partition mid-run and validates failover <60 seconds, cluster converges to single primary (no split-brain), database consistency (row count validated post-failover). EXPERIMENTAL: infrastructure ready, production validation pending.

  • CTDB Clustering for SMB HA (NEW): Enables Samba clustering so SMB clients survive HA failover without disconnecting or losing byte-range locks. Configuration via Settings → High Availability → CTDB Clustering. Requires: HA enabled, Samba enabled, ZFS dataset for lock database. Automatic failover of public IPs (Keepalived + CTDB coordination). Frontend UI for configuration, real-time cluster status monitoring. Documentation: HA-CTDB-SETUP.md. EXPERIMENTAL: tested in lab, production validation in progress.

  • Prometheus HA Alerting (NEW): 20+ production-grade alert rules for cluster health monitoring:

    • Failover readiness: quorum lost, peer unreachable, fencing disabled, witness unreachable
    • Replication health: lag high, replication stalled, Patroni no leader, WAL lag high
    • Safety nets: watchdog disabled, watchdog stuck, split-brain detection
    • System health: daemon unresponsive, etcd unreachable, SCSI-3 PR fencing failures
      Each alert includes detection criteria and remediation guidance. Integration: prometheus/ha-alerts.yml.

  • Grafana HA Cluster Dashboard (NEW): Real-time monitoring dashboard (grafana/dashboards/ha-cluster.json) with:

    • Cluster quorum status, node roles, recent failover timeline
    • Peer heartbeat tracking (missed beats per node)
    • Replication lag graphs with color-coded thresholds
    • Node availability heatmap over time
    • Failover suppression flags (maintenance, hysteresis, subordinate mode)
    • CTDB cluster status and public IP ownership
    • Automated impact measurement (how long did failover take?)

  • HA Operator Runbooks (NEW): Comprehensive guide (docs/admin/HA-FAILURE-MODES.md, 4000+ lines) covering 8 real failure scenarios:

    1. Node unreachable (network partition detection and recovery)
    2. Failover stalled (witness/maintenance/hysteresis blocking)
    3. Split-brain detected (emergency data reconciliation)
    4. Witness down (quorum witness loss)
    5. Replication lag high (bottleneck identification)
    6. Patroni no leader (etcd quorum restoration)
    7. SCSI-3 fencing failed (PR support validation)
    8. Subordinate mode (stale data catch-up)
      Each scenario includes: root causes, detection symptoms, step-by-step recovery commands, validation checklist, escalation procedures. Post-failover validation checklist included.

  • SCSI-3 PR Hardware Matrix (NEW): Documented hardware compatibility (docs/reference/HA-HARDWARE-MATRIX.md):

    • Tested working hardware (enterprise SANs, JBODs, SAS controllers with SCSI-3 PR support)
    • Known issues and firmware workarounds
    • NVMe-oF support (enterprise targets yes, consumer targets no)
    • Testing procedures and fallback strategies
    • Community contribution process for undocumented hardware

HA Status Changes

  • Failover Logic: Beta (stable, battle-tested)
  • Split-Brain Guards: Beta (all 6 guards validated by CI)
  • Load-Testing Framework: Experimental (infrastructure complete, field validation pending)
  • CTDB Clustering: Experimental (optional module for SMB failover support)
  • Monitoring: Beta (20+ alerts, real-time dashboard)
  • Operator Runbooks: Beta (comprehensive recovery procedures for all failure scenarios)

Database Changes

  • New Tables (CTDB configuration):
    • ctdb_config - CTDB settings (enable, pool, dataset, timeouts, log level)
    • ctdb_public_addresses - VIPs managed by CTDB across cluster nodes

API Changes

  • New Endpoints (CTDB):
    • GET /api/ha/ctdb/configure - Get CTDB configuration
    • POST /api/ha/ctdb/configure - Save CTDB configuration (requires system:admin)
    • GET /api/ha/ctdb/status - Real-time CTDB cluster status
    • GET /api/ha/ctdb/databases - CTDB database replication status

UI Changes

  • Settings → High Availability → CTDB Clustering (NEW): Configuration panel for:
    • Enable/disable CTDB
    • Configure data pool and dataset
    • Configure node timeout and recovery timeout
    • Manage public IP addresses (VIPs)
    • Monitor real-time cluster status and IP ownership

Known Limitations & Future Work

  • PostgreSQL HA: Load-testing framework in place; field validation pending before v15.0.0 GA
  • CTDB: SMB failover now works; requires broader field validation before v15.0.0 GA
  • Hardware Matrix: Community-contributed (submit test results for your hardware)
  • Monitoring: Alerts available for configuration; integration with PagerDuty/OpsGenie recommended for production

Documentation Changes

  • Added: docs/admin/HA-FAILURE-MODES.md (operator runbooks, all 8 scenarios)
  • Added: docs/admin/HA-CTDB-SETUP.md (CTDB installation and operational guide)
  • Added: docs/reference/HA-HARDWARE-MATRIX.md (SCSI-3 PR compatibility)
  • Updated: docs/admin/HIGH-AVAILABILITY.md (linked to new failure modes guide)
  • Added: prometheus/ha-alerts.yml (20+ alert rules)
  • Added: grafana/dashboards/ha-cluster.json (cluster monitoring dashboard)

No Breaking Changes

All changes are backward-compatible. HA remains off-by-default. Existing v14.6.0 deployments can upgrade without changes. CTDB is optional (enabled by setting services.dplaneos.ctdb.enable = true). New Prometheus alerts and Grafana dashboard are optional integrations.

🚀 Installation

curl -fsSL https://get.dplaneos.io | sudo bash -s -- --version 14.7.0
Assets 18
Loading
0 Join discussion