HA and MultiRegion en

中文 | English

HA & Multi-Region

Maintained version: v0.51.2 ｜ Last updated: 2026-07-06

EnerOS HA evolved from v0.20.0 dual-node master-standby to v0.41.0 multi-region active-active, providing RPO<1s / RTO<60s disaster recovery. This page is an evolution and config summary; for full deployment see the main repo docs/deployment.md §3.6 + ADR-0009.

Evolution Path

Version	Mode	RPO	RTO	Topology
v0.20.0	Dual-node master-standby	seconds	30s	Master-standby, failover
v0.41.0	Multi-region active-active	<1s	<60s	3 regions × 3 nodes

Multi-Region Deployment Topology

Recommended 3 regions × 3 nodes cluster topology (same-city dual-active + remote DR):

┌──────────────────┐  ┌──────────────────┐  ┌──────────────────┐
│   region-a       │  │   region-b       │  │   region-c       │
│   dc-shanghai    │  │   dc-shanghai    │  │   dc-beijing     │
│   priority=100   │  │   priority=90    │  │   priority=80    │
│                  │  │                  │  │                  │
│  ┌────────────┐  │  │  ┌────────────┐  │  │  ┌────────────┐  │
│  │ node-a1    │  │  │  │ node-b1    │  │  │  │ node-c1    │  │
│  │ (Primary)  │  │  │  │ (Secondary)│  │  │  │ (Witness)  │  │
│  ├────────────┤  │  │  ├────────────┤  │  │  ├────────────┤  │
│  │ node-a2    │  │  │  │ node-b2    │  │  │  │ node-c2    │  │
│  │ (Secondary)│  │  │  │ (Secondary)│  │  │  │ (Secondary)│  │
│  ├────────────┤  │  │  ├────────────┤  │  │  ├────────────┤  │
│  │ node-a3    │  │  │  │ node-b3    │  │  │  │ node-c3    │  │
│  │ (Witness)  │  │  │  │ (Witness)  │  │  │  │ (Primary)  │  │
│  └────────────┘  │  │  └────────────┘  │  │  └────────────┘  │
└──────────────────┘  └──────────────────┘  └──────────────────┘
        │                     │                     │
        └───────── WAL cross-region continuous replication (RPO < 1s) ───┘

Key Components

Component	crate	Responsibility
RegionAwareRouter	eneros-multiregion	Routing strategy (local_first / latency_optimal / failover_only)
CrossRegionReplicator	eneros-multiregion	WAL cross-region continuous replication (RPO<1s)
DrOrchestrator	eneros-os/bins/eneros-dr-orchestrator	DR fault detection (5s period) + failover + drill orchestration
RollingUpgrader	eneros-os/bins/eneros-upgrader	Rolling upgrade (rolling / blue-green)
ConflictResolver	eneros-multiregion	Active-active conflict resolution (CRDT / LWW / none)

Enabling HA Enhancement

Disabled by default (enabled = false); when unconfigured, HA middleware passes through, HA endpoints return 501.

[ha_enhance]
enabled = true

[ha_enhance.multi_region]
local_region_id = "region-a"
routing_strategy = "local_first"

[[ha_enhance.multi_region.regions]]
id = "region-a"
datacenter = "dc-shanghai"
priority = 100
endpoints = ["https://eneros-a.example.com:8080"]

[ha_enhance.replication]
wal_dir = "/var/lib/eneros/wal"
max_wal_size_mb = 1024
batch_size = 100
rpo_warning_threshold_ms = 1000
rpo_degradation_threshold_ms = 5000

[ha_enhance.dr]
failure_detection_timeout_ms = 10000
failover_confirm_delay_ms = 10000
drill_enabled = true
drill_cron = "0 3 * * 0"

[ha_enhance.active_active]
conflict_policy = "crdt"
clock_skew_threshold_ms = 100

DR Drills

Regular drills are critical to ensure DR effectiveness:

• drill_enabled = true + drill_cron = "0 3 * * 0" (drill every Sunday 03:00 off-peak)
• Rotating drill scenarios: Week 1 CrossRegionFailover / Week 2 DataCenterLoss / Week 3 NetworkPartition
• Drill results recorded to audit log with measured RPO/RTO values

Manual drill and emergency failover:

# Manual drill
eneros-dr-orchestrator --drill

# Emergency failover to specified region
eneros-dr-orchestrator --failover region-c

Rolling Upgrade

# Start rolling upgrade
eneros-upgrader --target-version 0.52.0 --strategy rolling

# Blue-green deployment
eneros-upgrader --target-version 0.52.0 --strategy blue-green

# Abort upgrade
eneros-upgrader --abort

Upgrade config:

[ha_enhance.upgrade]
health_check_retry_interval_ms = 5000
health_check_total_timeout_ms = 60000
health_check_retries = 3
graceful_shutdown_timeout_ms = 30000
auto_rollback_on_failure = true
minor_jump_allowed = 1
major_jump_requires_manual = true

Known Limitations (v0.51.2)

• Active-active conflict resolution only CRDT and LWW; business-layer conflict resolution not supported
• Cross-region replication only WAL stream; object storage replication not supported
• DrOrchestrator fault detection is simple heartbeat; Quorum not supported
• Rolling upgrade only supports minor jump of 1 version; major jump requires manual confirmation

Related Documentation

• ADR-0009 Multi-Region HA
• Operations and Plugins — Operations automation with HA
• Configuration Reference — [ha_enhance] section config
• Main repo docs/deployment.md §3.6 — HA deployment config
• Main repo crates/eneros-multiregion/ — source