[Feature] Multi-DC clustering (DC-local failure detection)

[pathosDev]

Size / Priority

• Size: XL — needs its own design phase + likely a 6-week implementation slot.
• Reference: Akka Multi-DC Cluster.

Rationale

When a cluster spans multiple datacenters (geo-distributed deployment, multi-region SaaS, active-active disaster recovery), naive uniform failure detection causes spurious cluster splits:

• Cross-DC RTT: 50-200ms (AWS us-east-1 ↔ eu-west-1 ≈ 80ms).
• Intra-DC RTT: <1ms.
• defaultFailureDetectorSettings heartbeat-interval = 1s, downAfterMs = 4s. With 100ms cross-DC RTT, normal jitter (e.g. GC pause across a window) easily exceeds 4s perceived skew. Cross-DC nodes get marked unreachable → downing → cluster splits.

The fundamental issue: one global failure-detector can't simultaneously be aggressive enough for intra-DC speed and lax enough for cross-DC latency.

Akka's approach: DC-aware FD + sharding.

• Each member tags with its DC.
• FD runs at separate rates per DC: aggressive intra-DC, lax cross-DC.
• Cross-DC unreachability is reported differently — not eligible for the same downing decisions as intra-DC unreachability.
• Sharding placement respects DC tags (allocate-to-local-dc-first, with cross-DC failover).

The framework's responsibility is non-trivial: this touches cluster, FD, sharding, downing, gossip. Hence XL + own design phase.

Reference: what Akka does

akka.cluster {
  multi-data-center {
    self-data-center = "us-east"
    cross-data-center-connections = 3   # FD heartbeats sent to N peers per other DC
    cross-data-center-gossip-probability = 0.2
    failure-detector {
      acceptable-heartbeat-pause = 10s   # vs 3s default for intra-DC
    }
  }
  sharding.passivate-idle-after = 120s
}

Behaviour:

• Heartbeats stay intra-DC by default (small mesh per DC).
• Cross-DC heartbeats sent only to a fixed-size random subset (cross-data-center-connections = 3); ensures one slow DC doesn't blow up the heartbeat mesh.
• Cross-DC unreachability does NOT trigger downing — only intra-DC unreachability does. A DC that becomes unreachable from another DC is "isolated", not "down".
• Sharding allocates entities preferentially to the local DC; cross-DC failover requires explicit configuration.
• Singletons are per-DC by default (one per DC, not one cluster-wide).

Design sketch — actor-ts equivalent

Touches several modules. Implementation must be staged.

1. NodeAddress + Member: add DC tag.

export interface NodeAddressData {
  readonly systemName: string;
  readonly host: string;
  readonly port: number;
  readonly dataCenter?: string;   // NEW; optional for back-compat with single-DC deployments
}

Member.dataCenter derived from address. Gossip carries it; tombstones carry it.

2. ClusterSettings: self-DC + multi-DC FD config.

export interface ClusterSettings {
  // ... existing ...
  readonly selfDataCenter?: string;   // default 'default' for single-DC
  readonly multiDataCenter?: {
    readonly crossDcHeartbeatConnections?: number;       // default 3
    readonly crossDcGossipProbability?: number;          // default 0.2
    readonly crossDcFailureDetector?: FailureDetectorSettings;  // separate config
  };
}

3. FailureDetector: per-DC instance.

class MultiDcFailureDetector {
  private readonly intraDc: FailureDetector;
  private readonly crossDc: Map<string, FailureDetector>;  // one per remote DC

  heartbeat(from: NodeAddress): void {
    const dc = from.dataCenter ?? 'default';
    if (dc === this.selfDc) this.intraDc.heartbeat(from);
    else this.crossDc.get(dc)?.heartbeat(from);
  }

  isReachable(member: Member): boolean {
    const dc = member.address.dataCenter ?? 'default';
    return dc === this.selfDc
      ? this.intraDc.isReachable(member.address)
      : this.crossDc.get(dc)?.isReachable(member.address) ?? true;
  }
}

4. Heartbeat sending: DC-aware peer selection.

• Intra-DC: full mesh (every member to every other member in same DC).
• Cross-DC: random subset of crossDcHeartbeatConnections peers per remote DC, refreshed every minute.

5. Gossip: DC-aware propagation.

• Intra-DC: every tick.
• Cross-DC: probability crossDcGossipProbability (default 0.2 = 1-in-5 ticks).

6. Downing: DC-aware decisions.

• Existing DowningProviders (LeaseMajority, KeepMajority) operate per-DC.
• Cross-DC unreachability → emit DataCenterUnreachable event but DON'T trigger downing.
• Application-level "is the other DC up?" check via cluster.dataCenterReachable(name).

7. ShardCoordinator: DC-affine allocation.

• New AllocationStrategy.preferLocalDc() — picks shards from the same DC as the caller.
• Cross-DC failover: when local DC has no available regions, fall back to remote DC.

8. ClusterSingleton: per-DC.

• Singleton is per-DC by default; cross-DC singletons require explicit opt-in (rare; mostly for coordination across DCs).

Integration with existing actor-ts subsystems

• Cluster gossip: extended MemberData carries dataCenter. Backwards-compat: missing field → 'default'.
• FailureDetector: replaced by MultiDcFailureDetector (single-DC mode = intraDc only, no crossDc map).
• DowningProvider: existing impls operate per-DC; new wrapper multiDcDowning orchestrates.
• Sharding: new allocation strategy; existing LeastShardAllocationStrategy continues to work (DC-blind).
• Singleton: per-DC by default.
• Receptionist / DistributedPubSub / DistributedData: per-DC by default (each DC has its own coherent view; cross-DC sync is opt-in).
• Metrics: existing metrics gain dataCenter label.

Out of scope / non-goals

• Single-DC users see no behaviour change. Default dataCenter is 'default'; everything works as today.
• Strong consistency across DCs: this is multi-DC clustering, not multi-DC strong-consistency. CRDTs (DistributedData) replicate across DCs eventually; quorum reads/writes that span DCs are documented as latency-sensitive.
• Geo-routing of HTTP traffic: out of scope — that's a CDN / load-balancer concern, not actor-ts.
• Active-active conflict resolution across DCs: relies on CRDT semantics; no extra magic.

Open design questions

1. What's the unit of failure-domain? Akka uses "data center". GCP / Azure call them "regions" + "zones". We could use failureDomain: string with no built-in semantics. Probably stick with dataCenter for Akka parity.
2. How does a node learn its own DC? Constructor option (selfDataCenter: string) + environment variable fallback (AKKA_TS_SELF_DC). Or k8s downward API (spec.nodeName → topology label).
3. Cross-DC heartbeat-peer selection: random subset is simple; sticky-after-first-pick is more stable. Akka uses sticky.
4. DataCenterUnreachable event: synchronous emit on FD state change, or batched? Affects downing-provider call frequency.
5. Migration path: existing single-DC users with dataCenter = undefined everywhere — how does their cluster behave once one node sets selfDataCenter: 'us-east'? Probably: cluster splits into "default" DC + "us-east" DC; that's intentional but should be documented as a one-time migration.

Test plan

1. Single-DC mode unchanged — full existing cluster test suite passes without setting selfDataCenter.
2. Two-DC cluster forms — 3 nodes in us-east + 3 in eu-west; cluster converges with 6 members, each tagged.
3. Intra-DC partition triggers downing — partition us-east-1 from us-east-2/3; downing-provider acts on us-east-1.
4. Cross-DC partition does NOT trigger downing — partition us-east from eu-west; both DCs continue, DataCenterUnreachable events emitted, no MemberDowned.
5. Cross-DC heartbeat subset — 3 nodes in us-east, 10 in eu-west; each us-east node maintains heartbeat with only 3 eu-west peers.
6. DC-affine sharding — AllocationStrategy.preferLocalDc(); entities started in us-east land on us-east regions.
7. Cross-DC sharding failover — us-east goes down; eu-west takes over the entities.
8. Per-DC singleton — one singleton in us-east, one in eu-west, both active concurrently.
9. Gossip cross-DC probability — observe gossip frequency; verify ~20% of ticks send cross-DC.
10. MultiNodeSpec with 2-DC topology — integration test fixture.

Acceptance criteria

• NodeAddress extended with dataCenter?: string; wire-format back-compat.
• ClusterSettings.selfDataCenter + multiDataCenter config.
• MultiDcFailureDetector per-DC instances.
• DC-aware heartbeat peer selection.
• DC-aware gossip probability.
• DataCenterUnreachable event, not auto-downing.
• AllocationStrategy.preferLocalDc() exported.
• Per-DC singletons.
• Metrics gain dataCenter label.
• Documentation: multi-DC deployment guide + migration from single-DC.
• Test suite covers all 10 cases.
• CHANGELOG entry under "New: multi-DC clustering".

Pre-implementation checklist

Because this is XL and touches many subsystems, the implementation should not start without:

• Joint review of the design sketch.
• Resolve the 5 open design questions.
• Sequencing: NodeAddress + Member changes first (foundational + back-compat-sensitive) → FailureDetector → Heartbeat sender → Gossip → Downing → Sharding → Singleton.
• Multi-week implementation budget agreed.
• Coordinate with [Security] LeaseMajority split-brain at network-latency boundary #142 (LeaseMajority split-brain) since they share downing-provider surface.