Skip to content

[Docs] The docs say the cluster shutdown phases wire themselves up automatically while the only three addTask sites in src/ hit service-unbind and actor-system-terminate, so a rolling deploy neither leaves the cluster nor hands off shards #993

Description

@pathosDev

Where

  • docs/src/content/docs/fundamentals/coordinated-shutdown.mdx:249 and :345-348
  • docs/src/content/docs/operations/deployment/kubernetes.mdx:301-316
  • and their German mirrors

What is wrong

The docs tell an operator that a SIGTERM to an actor-ts node hands off its shards, leaves the cluster, and waits for the leave to be acknowledged. None of that is wired. CoordinatedShutdown seeds twelve phases; two of them ever receive a task, and neither is a cluster phase.

The docs:

docs/src/content/docs/fundamentals/coordinated-shutdown.mdx:345-348
- **[Cluster overview](/cluster/overview/)** — the
  cluster phases (`cluster-leave`, `cluster-exiting`, …) wire
  themselves up automatically when the cluster extension is
  active.
docs/src/content/docs/fundamentals/coordinated-shutdown.mdx:249
// Cluster downing path is auto-wired by the cluster extension.
docs/src/content/docs/operations/deployment/kubernetes.mdx:301-316
1. K8s marks the pod terminating + starts `preStop`.
2. **10-second LB drain**.
3. SIGTERM lands.
4. Coordinated-shutdown runs:
   - Stop accepting new HTTP requests.
   - Drain in-flight requests.
   - Issue `cluster.leave()`.
   - Wait for cluster to acknowledge leave.
   - Terminate the actor system.
5. Process exits cleanly.
6. K8s starts a new pod from the new image.
7. New pod joins the cluster via the seed provider.

For sharded entities, **rebalancing happens automatically** —
the leaving node's shards are reallocated; new entities re-spawn
on the new pod from the journal.

The source. grep -rn "addTask" src/ finds exactly three registration sites, and one of them is the coordinator registering its own terminator:

src/CoordinatedShutdown.ts:145-149
    if (terminateActorSystem) {
      this.addTask(Phases.ActorSystemTerminate, 'terminate-actor-system', async () => {
        if (!this.system.isTerminated) await this.system.terminate();
      });
    }
src/http/HttpExtension.ts:201-205
        system.extension(CoordinatedShutdownId).addTask(
          Phases.ServiceUnbind,
          shutdownTaskName,
          () => binding.unbind(),
        );
src/devtools/DevToolsExtension.ts:114-118
    this.system.extension(CoordinatedShutdownId).addTask(
      Phases.ServiceUnbind,
      'devtools-detach',
      () => this.detach(),
    );

Populated phases: service-unbind (HTTP unbind, DevTools detach) and actor-system-terminate (system.terminate()). Empty in every deployment: before-service-unbind, service-requests-done, service-stop, before-cluster-shutdown, cluster-sharding-shutdown-region, cluster-leave, cluster-exiting, cluster-exiting-done, cluster-shutdown, before-actor-system-terminate. Nothing under src/cluster/ imports CoordinatedShutdown at all — grep -rn "CoordinatedShutdown" src/cluster/ is empty.

Three consequences the docs get backwards:

1. No shard handoff on shutdown. ShardRegion.postStop cancels timers and nothing else:

src/cluster/sharding/ShardRegion.ts:232-237
  override postStop(): void {
    this.unsubscribe?.();
    this.passivationTimer?.cancel();
    this.registerTimer?.cancel();
    this.asksSweepTimer?.cancel();
  }

The region never tells the coordinator it is going away. The coordinator only learns through onRegionTerminated (src/cluster/sharding/ShardCoordinator.ts:722), which fires off death-watch — i.e. after the failure detector has run its course. Every entity the departing node hosted is unroutable for that whole window, on a shutdown the operator initiated deliberately. That is the opposite of "rebalancing happens automatically".

2. clusterBootstrap bypasses the pipeline entirely. It installs its own signal handlers and its own two-step teardown:

src/cluster/ClusterBootstrap.ts:83-94
  // Wire shutdown.
  let shuttingDown: Promise<void> | null = null;
  const shutdown = async (): Promise<void> => {
    if (shuttingDown) return shuttingDown;
    shuttingDown = (async () => {
      try { await cluster.leave(); } catch { /* best-effort */ }
      await system.terminate();
    })();
    return shuttingDown;
  };

  installSignalHandlers(resolvedOptions.shutdownOnSignals ?? true, shutdown);

system.terminate() does not run CoordinatedShutdown — the framework says so itself at src/devtools/DevToolsExtension.ts:119. So on the recommended clustered entry point, a SIGTERM runs cluster.leave() + terminate() and skips every task the application registered, including the HTTP unbind registered automatically at bind time. The docs' own warning about terminate() (coordinated-shutdown.mdx:320-326, "Direct terminate() skips every task you registered") describes exactly what clusterBootstrap does on SIGTERM.

3. cluster-exiting waits for nothing. There is no task to wait, so the phase's timeout is irrelevant and the node terminates as soon as actor-system-terminate runs — peers discover the departure through gossip or through the failure detector, on their own schedule.

Fix

Two options, and they are not equivalent.

Preferred: implement the wiring. Cluster.join registers cluster-leavethis.leave() and cluster-exiting → await the self member reaching removed (or the phase timeout); ClusterSharding.start registers cluster-sharding-shutdown-region → gracefully hand every hosted shard back to the coordinator and await HandOffComplete; clusterBootstrap's shutdown becomes coordinatedShutdown.run(new ClusterLeavingReason()). This is the substance of #549, which asks for the same wiring from the API-ergonomics side (runUntilTerminated()); the shard-handoff phase and the clusterBootstrap bypass are not in its scope and should be added there or tracked here.

Minimum, if the wiring is not imminent: the documentation must stop asserting behaviour that does not exist. Delete the "wire themselves up automatically" bullet and the "auto-wired by the cluster extension" comment; rewrite the Kubernetes rollout sequence to describe what actually happens on SIGTERM and show the addTask calls a user must write themselves to get the documented sequence. An operator who sizes terminationGracePeriodSeconds against a phase list that is empty is sizing against nothing.

Either way the phase table at coordinated-shutdown.mdx:59-72 should distinguish "phases the framework populates" from "phases reserved for your tasks" — today it reads as a description of shipped behaviour.

Acceptance sketch

  • docs/.../coordinated-shutdown.mdx no longer claims the cluster phases wire themselves up, or a grep -rn "addTask" src/cluster/ returns the tasks that make the claim true.
  • kubernetes.mdx's rollout sequence matches what a SIGTERM to a clusterBootstrap node actually executes.
  • The phase table marks which phases ship with tasks and which are empty by design.
  • A test asserts that after coordinatedShutdown.run() on a clustered system the self member has reached removed — or, if the docs route is taken, no test is needed because no claim is made.
  • A shutdown of a shard-hosting node makes its shards routable on a peer without waiting for failure detection.
  • German mirror updated in the same change.

Verification status

Found in the ten-lens production-readiness review of 2026-08-05 (v0.13.0) and re-verified before filing: confirmed by reading. grep -rn "addTask" src/ --include=*.ts returns three registration sites, all quoted above; grep -rn "CoordinatedShutdown" src/cluster/ returns nothing. ShardRegion.postStop and ClusterBootstrap's shutdown are quoted verbatim from the current tree.

Related: #549 requests the wiring as a feature (HTTP unbind — already landed — plus cluster leave plus runUntilTerminated()); this issue is the docs asserting that wiring already exists, plus the two gaps #549 does not cover (shard handoff, clusterBootstrap bypassing the pipeline). #866 (phase graph from config) and #663 (draining terminate()) are adjacent and orthogonal.

Part of the production-readiness review batch — tracked in #913.

Metadata

Metadata

Assignees

No one assigned

    Labels

    documentationImprovements or additions to documentationpriority: highTop priority — high impact, plan nextproduction-goalBlocks or defines the path to production readiness

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions