Skip to content

[Bug] A terminating actor with a queued user message busy-spins the dispatcher until its last child stops, burning a core through every shutdown under load #915

Description

@pathosDev

Problem

An actor in state terminating that still has a user message queued busy-spins the dispatcher until its last child finishes stopping. Measured: 123 683 dispatcher executions in 300 ms — roughly 412 k/s, a fully saturated core — with exactly one message queued behind a PoisonPill on a parent whose child had a 400 ms postStop.

The trigger is the ordinary case, not an edge case: any producer still sending while a subtree is stopped. That is every system.terminate() and every SIGTERM under load.

Three lines interact:

  • run() only dequeues a user message when state === 'running', so in terminating nothing is consumed.
  • The finally block re-schedules whenever state !== 'terminated' && mailbox.hasMessages() — and the message is still there, because nothing consumed it.
  • onTerminate never suspends the mailbox, so hasMessages() keeps returning true.

Each iteration therefore does no work and immediately queues the next one.

Evidence

src/internal/ActorCell.ts:728-730 — user messages are gated on running:

src/internal/ActorCell.ts:728-730
      if (this.state === 'running') {
        const env = this.mailbox.dequeueUser();
        if (env) {

src/internal/ActorCell.ts:749-754 — the unconditional re-schedule:

src/internal/ActorCell.ts:749-754
    } finally {
      this.processing = false;
      if (this.state !== 'terminated' && this.mailbox.hasMessages()) {
        this.schedule();
      }
    }

src/internal/ActorCell.ts:815-817onTerminate sets the state and stops children, but never suspends:

src/internal/ActorCell.ts:815-817
  private async onTerminate(): Promise<void> {
    if (this.state === 'terminated' || this.state === 'terminating') return;
    this.state = 'terminating';

Reproduced with a counting Dispatcher wrapper:

dispatcher executions during 300ms of teardown: 123683

With MicrotaskDispatcher (src/Dispatcher.ts:29) the same shape is a hard livelock rather than a hot loop, because microtasks run to exhaustion before the event loop can advance the child's postStop timer.

Proposal

Suspend the mailbox in onTerminate, next to the existing state = 'terminating'. hasMessages() then returns false for the user queue while system messages still drain, which is exactly the intended behaviour — and finalizeTermination already routes the remaining user messages to dead letters.

Acceptance sketch

  • A terminating actor with queued user messages performs O(1) dispatcher executions, not O(elapsed time).
  • Queued user messages still reach dead letters on termination.
  • System messages (childTerminated, terminate) still drain while terminating.
  • A regression test asserts an execution-count ceiling during teardown — the invariant is "no dispatcher execution while idle", which no current test covers.

Verification status

Found in the ten-lens production-readiness review of 2026-08-05 (v0.13.0) and re-verified before filing: reproduced by execution with an instrumented dispatcher; the cited lines were read to establish the mechanism.

Part of the production-readiness review batch — tracked in #913.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingpriority: highTop priority — high impact, plan nextproduction-goalBlocks or defines the path to production readiness

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions