[Bug] Every cluster membership transition is logged at `debug` while the default level is `info`, so a partition produces no operator-visible record of which peer went unreachable, when, or why

[pathosDev]

Problem

Every membership transition in Cluster.ts is logged at debug. The default level is info. So on a stock deployment, a node going unreachable, a node being deleted from the membership set, a leader change, and a peer's graceful leave all produce no log output at all — the one class of event an operator is looking for during a partition.

The records themselves are good: they name the peer, the reason, and the previous status. They are simply filtered out. Raising the level to debug to get them is not an option during an incident on a busy cluster — that also switches on per-gossip-round and per-heartbeat chatter from the same file, at gossip cadence, per peer.

The metric does not close the gap either. cluster_members_up is a single unlabelled integer. It tells you the count dropped from 5 to 4; it does not tell you which address left, when, whether it was unreachable or downed or leaving, or who decided.

Evidence

Level census of src/cluster/Cluster.ts on the current tree: 15 log.debug, 9 log.warn, 1 log.info, 0 log.error. The single info is the operator-initiated force-down:

src/cluster/Cluster.ts:403-409
    const removed = downed.withRemoved(Date.now());
    this.members.set(key, removed);
    this.failureDetector.forget(member.address);
    this.emit(new MemberRemoved(removed));
    this.log.info(`operator force-down: ${member.address}`);
    return true;
  }

Everything a partition actually produces is debug:

src/cluster/Cluster.ts:791-802
  private failureDetectionTick(): void {
    for (const member of Array.from(this.members.values())) {
      if (member.address.equals(this.selfAddress)) continue;
      const decision = this.failureDetector.decide(member.address);
      if (decision === 'unreachable' && member.status === 'up') {
        this.log.debug(`FD: ${member.address} → unreachable (heartbeat timeout)`);
        this.updateMember(member.withStatus('unreachable'));
        this.emit(new MemberUnreachable(this.members.get(member.address.toString())!));
      } else if (decision === 'down' && member.status !== 'down' && member.status !== 'removed') {
        this.log.debug(`FD: ${member.address} → down (was ${member.status}); deleting from membership`);
        const downed = member.withStatus('down');
        this.updateMember(downed);

src/cluster/Cluster.ts:1133-1139
    if (changed) {
      this.currentLeader = newLeader;
      const prevStr = prev.fold(() => 'none', (member) => member.address.toString());
      const nextStr = newLeader.fold(() => 'none', (member) => member.address.toString());
      this.log.debug(`leader changed: ${prevStr} → ${nextStr}`);
      this.emit(new LeaderChanged(newLeader));
    }

src/cluster/Cluster.ts:741-744
    this.log.debug(`peer ${peer} sent leave — tombstoning (was ${existing.status} v${existing.version})`);
    const leaving = existing.withStatus('leaving');
    const removed = leaving.withRemoved(Date.now());

src/cluster/Cluster.ts:628-633
    if (this.isLeader()) {
      for (const member of this.members.values()) {
        if (member.status === 'joining' || member.status === 'weakly-up') {
          this.log.debug(`leader-promote: ${member.address} ${member.status}→up`);
          this.updateMember(member.withStatus('up'));
        }

The default level:

src/ActorSystem.ts:459-461
function logLevelFromConfig(config: Config): LogLevel {
  if (!config.hasPath(ConfigKeys.logger.level)) return LogLevel.Info;
  const raw = config.getString(ConfigKeys.logger.level).toLowerCase();

src/config/reference.ts:16-18
  logger {
    level = "info"   # debug | info | warn | error | off
  }

The only membership metric, unlabelled:

src/cluster/Cluster.ts:1102-1105
    metricsOf(this.system).gauge(
      'cluster_members_up', {},
      { help: 'Number of cluster members currently in `up` state.' },
    ).set(this.upMembers().length);

And the runbook shows the operator what to expect during unreachability — as [INFO ] lines that this code can never emit at any level:

docs/src/content/docs/operations/troubleshooting.mdx:74-80
In logs:

```
[INFO ] cluster — node-X marked unreachable
[INFO ] cluster — node-X marked reachable
[INFO ] cluster — node-X marked unreachable
```

Proposal

Membership transitions are low-frequency, operationally decisive events. They belong at info, unconditionally:

• unreachable / reachable transitions (with peer address and the detector's reason)
• down / removed (with previous status and who decided — failure detector, downing provider, or operator)
• graceful leave observed from a peer
• a status transition learned through gossip (Cluster.ts:1083-1086, currently debug)
• leader change
• self joining / self up

Keep at debug what is per-tick or per-round: gossip receipt (:609), the heartbeat path, tombstone pruning (:1173).

The established precedent for this is a cluster log-info switch that defaults to on, with a separate verbose tier for the chatty per-gossip events; #867 proposes the same key (cluster.log-info) as an opt-in toggle. Opt-in is the wrong default here — the events cost nothing at cluster scale and the operator who needs them has already lost the incident by the time they can restart a node with debug on.

Give cluster_members_up a status breakdown at the same time — either a cluster_members{status="up|unreachable|down|leaving"} gauge family (bounded cardinality: one series per status) or a cluster_member_transitions_total{from,to} counter. Deliberately not an address label: that is unbounded across a cluster's lifetime and repeats #658's mistake. The address belongs in the log line, which is where an operator correlates it.

Acceptance sketch

• With the default config, a peer going unreachable produces an info line naming the peer and the reason.
• With the default config, a leader change produces an info line naming both leaders.
• Turning on debug still does not drown the transitions — per-tick gossip and heartbeat records stay at debug.
• A status breakdown metric exists with bounded cardinality, and docs/.../operations/troubleshooting.mdx names the real metric.
• The [INFO ] sample lines in troubleshooting.mdx match what the code emits.

Verification status

Found in the ten-lens production-readiness review of 2026-08-05 (v0.13.0) and re-verified before filing: confirmed by reading. The level census is grep -c "log.debug\|log.info\|log.warn\|log.error" src/cluster/Cluster.ts (15 / 1 / 9 / 0); the default level was also observed at runtime (system.log.level === 1, i.e. LogLevel.Info) on a default ActorSystem. All snippets are verbatim from the current tree.

Related: #867 proposes cluster.log-info as an opt-in diagnostics toggle — this issue argues the transitions should be info by default and the toggle should govern the verbose tier instead. #658 is why the proposed breakdown metric must not carry an address label.

Part of the production-readiness review batch — tracked in #913.