Skip to content

[Security] Forged heartbeat.from keeps a dead node "healthy" forever (blocks singleton/shard failover) and makes the node dial an attacker-chosen host:port #572

Description

@pathosDev

Component: src/cluster/Cluster.ts
Severity (assessment): MEDIUM
CWE: CWE-290

onHeartbeat receives the connection's real identity as from but uses it only for the log line inside isPlausibleHeartbeat; the failure-detector bump, the reachability flip and the acknowledgment destination all come from the wire field message.from. A peer can therefore assert liveness on behalf of any address, and can steer an outbound TCP connect to an arbitrary host and port.

Exploit walkthrough

Attacker completes the unauthenticated hello handshake and then sends, every second, {"t":"heartbeat","from":{"systemName":"<cluster>","host":"<dead-node-host>","port":<dead-node-port>},"seq":n,"ts":Date.now()}. FailureDetector.heartbeat(peer) sets lastSeen = now for the victim's key, so decide() returns 'healthy' (FailureDetector.ts:52-58) and failureDetectionTick never marks it unreachable/down. Result: a genuinely crashed node stays up cluster-wide — ClusterSingleton never migrates, ShardRegion never re-allocates its shards, and every message routed to it is silently lost. The same frames also flip an already-unreachable member back to up and emit MemberReachable. Second effect: this.transport.send(peer, …) with a peer never seen before falls through to TcpTransport.openOutbound, which calls backend.connect({ host: to.host, port: to.port }) — the attacker picks any host/port string, so each forged heartbeat makes the cluster node open an outbound TCP connection from inside the trusted network (internal port-scan / SSRF-style probe, and a bandwidth amplifier: ~120 bytes in, one connect + framed hello out).

Evidence — src/cluster/Cluster.ts:547

src/cluster/Cluster.ts:545-558:
```
  private onHeartbeat(from: NodeAddress, message: HeartbeatMessage): void {
    if (!this.isPlausibleHeartbeat(from, message)) return;
    const peer = NodeAddress.fromJSON(message.from);
    this.failureDetector.heartbeat(peer);
    // Reply isn't strictly needed because send() also bumps the detector,
    // but it keeps symmetric latency information.
    this.transport.send(peer, { t: 'heartbeat-ack', from: this.selfAddress.toJSON(), seq: message.seq });

    // If the peer was unreachable and we see traffic again, flip it back.
    const existing = this.members.get(peer.toString());
    if (existing && existing.status === 'unreachable') {
      this.updateMember(existing.withStatus('up'));
```
src/cluster/FailureDetector.ts:36-40  the bump is unconditional and keyed purely on the supplied address:
```
  heartbeat(peer: NodeAddress, now: number = Date.now()): void {
    const key = peer.toString();
    const prev = this.samples.get(key);
    this.samples.set(key, { lastSeen: now, everSeen: prev?.everSeen ?? true });
```
src/cluster/Transport.ts:117 and 165-172  an unknown destination triggers a dial to the supplied host/port:
```
    const connection = this.byPeer.get(to.toString()) ?? this.openOutbound(to);
```
```
        const sock = await backend.connect({
          host: to.host,
          port: to.port,
```

Why the existing guard does not cover it

isPlausibleHeartbeat (Cluster.ts:534-543) is the guard that exists here — I read it in full: it validates only message.seq (Number.isSafeInteger && >= 0) and message.ts (finite, not more than 24 h in the future), and its own JSDoc scopes itself to numeric plausibility (#115). It never compares message.from to from. NodeAddress.fromJSON (NodeAddress.ts:29-31) does no validation of host/port at all. handleWire's own this.failureDetector.heartbeat(from) on line 496 uses the real socket identity, which proves the correct value is in scope — onHeartbeat just prefers the wire field. tests/multi-node/cluster-security.test.ts:479-513 pins the #115 numeric guard, and in both cases the test sets from: peer.toJSON() equal to the injected socket address, so no test exercises a mismatch. Adjacent to but distinct from the tracked #121 (ClusterClient envelope.from, a different file and a reply-routing effect) and #138 (member-map growth via gossip; this path bypasses mergeMember).

Suggested fix

Use the connection identity, not the payload: drop the message.from read and bump/acknowledge against from (this.failureDetector.heartbeat(from), this.transport.send(from, …)), or, to keep the field for diagnostics, extend isPlausibleHeartbeat with NodeAddress.fromJSON(message.from).equals(from) and reject on mismatch with the same possible corruption or forgery warning. Additionally, refuse to open a new outbound connection for an address that is not a known member, so no wire field can drive backend.connect.

Verification status

Found in the whole-framework security audit of 2026-08-01 (v0.12.0), then adjudicated by an independent verifier instructed to refute it.

Verifier note

Cluster.ts:547 const peer = NodeAddress.fromJSON(message.from) is the value used for failureDetector.heartbeat(peer) (line 548), for transport.send(peer, …) (line 551) and for the unreachable→up flip (lines 554-558); from is used only inside isPlausibleHeartbeat, which I read in full (lines 534-543) and which checks only seq (safe integer, >= 0) and ts (finite, <= now+24h) — no equals(from) comparison. FailureDetector.ts:36-39 bumps lastSeen unconditionally for whatever key it is handed, and decide() (line 52-58) then returns 'healthy', so failureDetectionTick (Cluster.ts:716-724) never marks the dead victim unreachable/down. The outbound-dial effect is also real: Transport.ts:117 this.byPeer.get(to.toString()) ?? this.openOutbound(to) reaches Transport.ts:169 backend.connect({ host: to.host, port: to.port }) with no validation (NodeAddress.fromJSON, NodeAddress.ts:29-31, validates nothing). heartbeatTick (Cluster.ts:704-709) always sets from: this.selfAddress, so the origin check is free.

Correction applied: Real defect, but 'high' overstates the marginal capability for the same reason as #1: onGossip (Cluster.ts:562-563) performs the identical wire-field-driven failureDetector.heartbeat(NodeAddress.fromJSON(message.from)), and a forged gossip member with status 'up' lands in reachableMembers() so heartbeatTick dials its attacker-chosen host:port anyway — both claimed effects (liveness forgery and the arbitrary outbound connect) are already reachable through the designed gossip path, which no origin check can close. Attacker must be able to reach the cluster port, which the security docs require to be firewalled and mTLS-protected. Fix is still correct and cheap.

Independently reported by more than one audit surface (cluster-wire, cluster-membership) — merged into this issue.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingpriority: mediumUseful, not urgentsecuritySecurity-relevant — see severity label for impact tierseverity: mediumModerate impact or requires specific conditions

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions