[Bug] `ClusterRouter` rebuilds only on `MemberUp`/`MemberRemoved` while `upMembers()` drops an unreachable member at once, so 1/N of routed traffic is blackholed for the whole `downAfterMs` window with no dead letter

[pathosDev]

Problem

ClusterRouter rebuilds its routee list on MemberUp and MemberRemoved only. upMembers() — the list it rebuilds from — drops a member the moment its status stops being up. The two disagree, and the gap between them is a blackhole.

When the failure detector marks a peer unreachable it emits MemberUnreachable, which the router's filter ignores. The member is gone from upMembers() immediately, but the router's cached RemoteActorRef for it survives until MemberRemoved fires at downAfterMs. With stock timings that is a 3-second window (unreachableAfterMs: 2_000, downAfterMs: 5_000), measured from the failure the whole downAfterMs — during which 1/N of everything the router routes goes to a node that is not answering. Those messages do not surface anywhere the application can see: RemoteActorRef.tell hands the envelope to Cluster._sendEnvelope → Transport.send, which either writes into a socket the peer is no longer reading or appends to connection.pending and drops the oldest at 1 000 frames. Nothing reaches deadLetters, no onLost-style callback exists, and the sender's ask just times out.

The same filter also ignores MemberLeft. That one is narrower than it looks: onLeave emits MemberLeft and MemberRemoved back to back, so a peer that receives the leave frame directly does rebuild. But a leaving status learned through gossip — mergeMember accepting the node's own record — emits only MemberLeft, and upMembers() drops a leaving member just as it drops an unreachable one. So the router keeps sending to a draining node on that path too.

Evidence

The filter:

src/cluster/router/ClusterRouter.ts:108-118
    this.unsubscribe = this.options.cluster.subscribe((evt) => {
      // Only `up` and `removed` change the routee set.  `joined`,
      // `weakly-up`, `unreachable` are intermediate states we don't
      // route to.  Replay-on-subscribe (Cluster fires every current
      // member as a series of MemberJoined/MemberUp on subscribe) is
      // already handled by the initial rebuild — but firing here too
      // is harmless (rebuild is idempotent).
      if (evt instanceof MemberUp || evt instanceof MemberRemoved) {
        this.rebuildRoutees();
      }
    });

The comment says unreachable members are states "we don't route to". They are exactly the states it does route to, because nothing rebuilds the list when a member enters one:

src/cluster/Cluster.ts:286-291
  /** Members in the `up` state, ordered by address — the "active set". */
  upMembers(): Member[] {
    return Array.from(this.members.values())
      .filter(member => member.status === 'up')
      .sort((a, b) => a.address.compareTo(b.address));
  }

src/cluster/Cluster.ts:795-798
      if (decision === 'unreachable' && member.status === 'up') {
        this.log.debug(`FD: ${member.address} → unreachable (heartbeat timeout)`);
        this.updateMember(member.withStatus('unreachable'));
        this.emit(new MemberUnreachable(this.members.get(member.address.toString())!));

And what happens to the messages in the meantime — a bounded buffer with a once-per-connection warning, and no dead-letter path:

src/cluster/Transport.ts:145-163
  send(to: NodeAddress, message: WireMessage): void {
    if (this.stopped) return;
    const connection = this.byPeer.get(to.toString()) ?? this.openOutbound(to);
    if (connection.peer && connection.socket) {
      connection.socket.write(encodeFrame(message));
    } else {
      // Wait for hello / hello-ack, but never without a bound.
      if (connection.pending.length >= MAX_PENDING_FRAMES) {
        connection.pending.shift();
        if (!connection.pendingOverflowed) {
          connection.pendingOverflowed = true;
          this.log.warn(
            `handshake buffer for ${to} hit ${MAX_PENDING_FRAMES} frames; dropping oldest`,
          );
        }
      }
      connection.pending.push(message);
    }
  }

Proposal

Rebuild on every event that changes what upMembers() (or upMembersWithRole()) returns, which is the actual contract: MemberUnreachable, MemberReachable, MemberLeft and MemberDown alongside the two already handled. Cheapest correct version is to stop enumerating events and rebuild on any ClusterEvent — rebuildRoutees is idempotent and the event rate is a handful per membership change.

While the filter is being touched, #235 asks for the instanceof pair to become a match() — the two changes are the same three lines and should land together, with each arm delegating to a private onXxx handler as the house rule requires.

Independently worth having: messages a RemoteActorRef cannot deliver should reach deadLetters rather than evaporating in connection.pending. That is the difference between "1/N of traffic disappeared for three seconds" and an observable event.

Acceptance sketch

• A member transitioning to unreachable removes it from the router's routee list on the next message, not at downAfterMs.
• A member transitioning to leaving (via gossip, without a direct leave frame) removes it too.
• MemberReachable puts it back.
• A test asserts zero envelopes are addressed to an unreachable member during the unreachable → removed window.
• The subscription arms are a match() with one onXxx handler per arm ([Feature] Replace the ClusterRouter cluster-event instanceof pair with match() #235).

Verification status

Found in the ten-lens production-readiness review of 2026-08-05 (v0.13.0) and re-verified before filing: reproduced by execution. A two-member cluster with a stub transport that records every frame it is asked to send, a round-robin ClusterRouter over /user/worker, and unreachableAfterMs: 300 / downAfterMs: 30_000 to widen the window:

healthy: upMembers=2  envelopes->PEER=5
after crash: PEER status=unreachable  upMembers=sys@10.0.0.1:2552
cluster events seen by a subscriber since the router was built: MemberJoined, MemberUp, SelfUp, LeaderChanged, MemberUnreachable
during the unreachable window: envelopes->PEER = 5 of 10
  (upMembers() no longer contains PEER, but the router's cached routee list does)

Half of the traffic kept going to a member the cluster had already dropped from its active set. The subscriber log shows MemberUnreachable was delivered — the router's filter simply discards it. The MemberLeft half of the claim is stated above in its accurate, weaker form: it was read from emitStatusTransition, not reproduced, and it does not apply to the direct leave-frame path.

Part of the production-readiness review batch — tracked in #913.