Problem
A symmetric network partition never heals. Once a peer is marked unreachable it drops out of reachableMembers(), which is the only source list for both the gossip tick and the heartbeat tick — so both sides immediately stop sending each other anything, and then delete each other at downAfterMs. Nothing ever re-dials: the seed-retry timer cancels itself as soon as self is up, and seed discovery runs exactly once at bootstrap.
Recovery requires restarting a process. The comment at the deletion site claims the opposite — that deleting rather than tombstoning is what makes "a partition followed by a heal" recover the peer — and the repo's own test asserts that it does not.
Evidence
Gossip targets only reachable members:
src/cluster/Cluster.ts:760
const targets = this.reachableMembers().filter(member => !member.address.equals(this.selfAddress));
So does the heartbeat:
src/cluster/Cluster.ts:785
for (const member of this.reachableMembers()) {
and reachableMembers() excludes anything marked unreachable (src/cluster/Member.ts:32-37).
The seed-retry timer stops once the node is up (src/cluster/ClusterBootstrap.ts:474-480), and contactSeeds has no other caller — discovery runs once, at bootstrap (src/cluster/ClusterBootstrap.ts:143-151).
The repo's own multi-node test records the outcome:
tests/multi-node/sharding-failover.test.ts:215-222
after heal(), upMembers().length is still 2 and the third node never returns.
Proposal
Keep contacting unreachable peers. Heartbeat and gossip should target members minus tombstones rather than reachableMembers() — an unreachable peer is precisely the one you need to keep probing, since that is the only way to observe that it came back. Separately, re-arm the seed-retry timer whenever the up-member count drops, so a node that loses every peer can rediscover the cluster.
This is a prerequisite for the split-brain resolvers being meaningful at all (see the failure-detector issue in this batch): a resolver that keeps one side alive is only correct if the other side can rejoin afterwards.
Acceptance sketch
Verification status
Found in the ten-lens production-readiness review of 2026-08-05 (v0.13.0) and re-verified before filing: confirmed by reading — both tick sites and reachableMembers() were read in the current tree, and the existing test's assertion was read as corroboration.
Part of the production-readiness review batch — tracked in #913.
Problem
A symmetric network partition never heals. Once a peer is marked
unreachableit drops out ofreachableMembers(), which is the only source list for both the gossip tick and the heartbeat tick — so both sides immediately stop sending each other anything, and then delete each other atdownAfterMs. Nothing ever re-dials: the seed-retry timer cancels itself as soon as self isup, and seed discovery runs exactly once at bootstrap.Recovery requires restarting a process. The comment at the deletion site claims the opposite — that deleting rather than tombstoning is what makes "a partition followed by a heal" recover the peer — and the repo's own test asserts that it does not.
Evidence
Gossip targets only reachable members:
So does the heartbeat:
and
reachableMembers()excludes anything marked unreachable (src/cluster/Member.ts:32-37).The seed-retry timer stops once the node is up (
src/cluster/ClusterBootstrap.ts:474-480), andcontactSeedshas no other caller — discovery runs once, at bootstrap (src/cluster/ClusterBootstrap.ts:143-151).The repo's own multi-node test records the outcome:
after
heal(),upMembers().lengthis still 2 and the third node never returns.Proposal
Keep contacting unreachable peers. Heartbeat and gossip should target
membersminus tombstones rather thanreachableMembers()— an unreachable peer is precisely the one you need to keep probing, since that is the only way to observe that it came back. Separately, re-arm the seed-retry timer whenever the up-member count drops, so a node that loses every peer can rediscover the cluster.This is a prerequisite for the split-brain resolvers being meaningful at all (see the failure-detector issue in this batch): a resolver that keeps one side alive is only correct if the other side can rejoin afterwards.
Acceptance sketch
partition(a, b)thenheal(a, b), both nodes see each other asupagain without a restart.tests/multi-node/sharding-failover.test.tsasserts the healed state rather than recording the unhealed one.Cluster.ts:804no longer claims partition+heal recovery that the code does not provide.Verification status
Found in the ten-lens production-readiness review of 2026-08-05 (
v0.13.0) and re-verified before filing: confirmed by reading — both tick sites andreachableMembers()were read in the current tree, and the existing test's assertion was read as corroboration.Part of the production-readiness review batch — tracked in #913.