Skip to content

[Bug] Gossip and heartbeats target only reachable members, so a symmetric partition never heals and both halves run as independent clusters until a process restart #930

Description

@pathosDev

Problem

A symmetric network partition never heals. Once a peer is marked unreachable it drops out of reachableMembers(), which is the only source list for both the gossip tick and the heartbeat tick — so both sides immediately stop sending each other anything, and then delete each other at downAfterMs. Nothing ever re-dials: the seed-retry timer cancels itself as soon as self is up, and seed discovery runs exactly once at bootstrap.

Recovery requires restarting a process. The comment at the deletion site claims the opposite — that deleting rather than tombstoning is what makes "a partition followed by a heal" recover the peer — and the repo's own test asserts that it does not.

Evidence

Gossip targets only reachable members:

src/cluster/Cluster.ts:760
    const targets = this.reachableMembers().filter(member => !member.address.equals(this.selfAddress));

So does the heartbeat:

src/cluster/Cluster.ts:785
    for (const member of this.reachableMembers()) {

and reachableMembers() excludes anything marked unreachable (src/cluster/Member.ts:32-37).

The seed-retry timer stops once the node is up (src/cluster/ClusterBootstrap.ts:474-480), and contactSeeds has no other caller — discovery runs once, at bootstrap (src/cluster/ClusterBootstrap.ts:143-151).

The repo's own multi-node test records the outcome:

tests/multi-node/sharding-failover.test.ts:215-222

after heal(), upMembers().length is still 2 and the third node never returns.

Proposal

Keep contacting unreachable peers. Heartbeat and gossip should target members minus tombstones rather than reachableMembers() — an unreachable peer is precisely the one you need to keep probing, since that is the only way to observe that it came back. Separately, re-arm the seed-retry timer whenever the up-member count drops, so a node that loses every peer can rediscover the cluster.

This is a prerequisite for the split-brain resolvers being meaningful at all (see the failure-detector issue in this batch): a resolver that keeps one side alive is only correct if the other side can rejoin afterwards.

Acceptance sketch

  • After partition(a, b) then heal(a, b), both nodes see each other as up again without a restart.
  • Heartbeats continue to an unreachable peer.
  • A node that loses all peers resumes seed discovery.
  • tests/multi-node/sharding-failover.test.ts asserts the healed state rather than recording the unhealed one.
  • The comment at Cluster.ts:804 no longer claims partition+heal recovery that the code does not provide.

Verification status

Found in the ten-lens production-readiness review of 2026-08-05 (v0.13.0) and re-verified before filing: confirmed by reading — both tick sites and reachableMembers() were read in the current tree, and the existing test's assertion was read as corroboration.

Part of the production-readiness review batch — tracked in #913.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingpriority: highTop priority — high impact, plan nextproduction-goalBlocks or defines the path to production readiness

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions