Skip to content

[Feature] Dead letters are published to an event stream that nothing subscribes to by default, so an undeliverable message produces zero output while two docs pages claim the system logs it #1000

Description

@pathosDev

Use case

"My message vanished" is the single most common actor-model support question, and on a default actor-ts system it produces no output whatsoever. Dead letters are published onto the event stream, nothing subscribes to that stream by default, and a publish with no matching subscriber is a no-op. The message is gone and the process is silent about it.

Reproduced on the current tree. A default ActorSystem, an actor stopped, then told:

[probe] default log level = 1
[probe] >>> telling a stopped actor on a DEFAULT system <<<
[probe] >>> nothing between the markers = zero output <<<

Zero bytes on stdout and stderr between the markers. Adding a spawned subscriber to the same run shows the DeadLetter was constructed and published correctly — it simply had no audience:

[probe] subscriber saw: DeadLetter(msg=hello?, from=none, to=actor-ts://prod-check-2/user/echo)
[probe] subscriber invocations = 1

The whole mechanism:

src/internal/DeadLetterRef.ts:22-36
  tell(message: unknown, sender: ActorRef | null = null): void {
    // A DeadLetter wrapping another DeadLetter is the signature of a
    // delivery loop: publishing a dead letter reached a subscriber that
    // has terminated without unsubscribing, whose cell then wrapped it
    // again and sent it back here.  Re-publishing would hand it to the
    // same dead subscriber forever, so the nested one is dropped —
    // there is nowhere further to send an undeliverable dead letter.
    // (A single wrap is the NORMAL path: cells wrap before calling.)
    if (message instanceof DeadLetter && message.message instanceof DeadLetter) return;

    const deadLetter = message instanceof DeadLetter
      ? message
      : new DeadLetter(message, sender, this);
    this.eventStream.publish(deadLetter);
  }

publish and out. No logger reference exists in the class, and grep -rn "DeadLetter" src/ finds exactly one subscriber anywhere in the framework — src/devtools/internal/NodeSampler.ts:66-67, which only bumps a counter and only when DevTools is attached.

The documentation states, in two places, that this is logged — with two different levels, both wrong:

docs/src/content/docs/fundamentals/actor-system.mdx:156-159
The `/deadLetters` "actor" is special — messages to a `tell` on a
stopped ref, or to a ref that never existed, route there.  By
default the system logs dead letters at `debug` level; subscribe to
the event stream if you want to react programmatically.
docs/src/content/docs/fundamentals/event-stream.mdx:161-165
  The bus doesn't auto-unsubscribe stopped refs.  Subsequent
  publishes still `tell` the dead ref; the system routes them to
  `/deadLetters`, which logs a warning.  If a subscriber is
  short-lived, `unsubscribe` it in `postStop` — or rely on the
  dead-letter cleanup if you don't mind the noise.

Neither is true at any level. docs/src/content/docs/operations/troubleshooting.mdx offers no dead-letter procedure at all, and operations/overview.mdx:107 sends the operator to "dead-letter" as a cause for "Actor isn't receiving messages" without saying where to look — because there is nowhere.

The remedy the docs point at is also heavier than it sounds. EventStream.subscribe takes an ActorRef, not a callback (src/EventStream.ts:63-66), so observing dead letters means defining an actor class, spawning it, and subscribing it — before anything went wrong. Nobody does that preventively, and by the time the question is asked the messages that vanished are gone.

Proposed shape

Log dead letters at info by default, throttled. The established default for this is a count-based cap — log the first N with full detail, then log a single line saying further ones are suppressed, and resume after a window. That is the right shape — it makes the default system self-explaining without risking a log flood when something starts fanning out dead letters at message rate. The record should carry the recipient path, the sender if known, and the message's class name (not its contents — that is user data).

DeadLetterRef needs a logger, which the ActorSystem constructor already has when it builds the ref (src/ActorSystem.ts:123), so this is a constructor argument and a counter.

Suppress during shutdown by default. The termination path drains every remaining user message to dead letters (src/internal/ActorCell.ts:881-884), so a large system's shutdown would otherwise emit a burst that is expected and uninteresting. Suppressing dead-letter logging during shutdown is the right default here too.

Offer a callback subscription. system.onDeadLetter((dl) => …) — or a general eventStream.subscribeFunction(Class, fn) — removes the spawn-an-actor ceremony for the diagnostic case. The framework already has exactly this helper: subscribeToEventStream(system, channel, handler, name) in src/devtools/internal/EventStreamProbe.ts:38-61 wraps a callback in a spawned ProbeActor and subscribes it. It is devtools-internal and exported from neither public index. Promoting it is most of the work. The actor-ref subscription stays for anything that needs supervision and mailbox semantics.

Give the operations runbook a dead-letter procedure — what the line looks like, how to raise the throttle, how to subscribe programmatically — and fix the two doc claims above, EN + DE.

#867 proposes log-dead-letters among a batch of diagnostics toggles, defaulted off. Off is the wrong default: a framework that silently discards messages, and whose docs claim it does not, has no self-diagnosis story at all. The key should exist — to set the throttle count, and to turn logging off for a system that genuinely expects dead letters — but 10 should be what it ships with.

Acceptance

  • Telling a stopped actor on a default ActorSystem produces exactly one log record naming the recipient path and the message class.
  • After the configured count, a single suppression line is emitted and further dead letters are silent until the window elapses.
  • Dead letters produced by terminate()'s mailbox drain are silent by default.
  • log-dead-letters and log-dead-letters-during-shutdown are wired (tests/unit/config/NoDeadConfigKeys.test.ts passes).
  • A callback-based subscription exists that needs no spawned actor.
  • actor-system.mdx and event-stream.mdx describe the real level and the real throttle; operations/troubleshooting.mdx gains a "my message vanished" entry. EN + DE.

Verification status

Found in the ten-lens production-readiness review of 2026-08-05 (v0.13.0) and re-verified before filing: confirmed by running. A throwaway script on the current tree created a default ActorSystem, spawned an actor, stopped it, and told it — total framework output was zero bytes. The same script with a spawned actor subscribed to DeadLetter on the event stream received the letter, confirming the publish happens and the audience is what is missing. The script was deleted after the run. Both doc claims and DeadLetterRef.tell are quoted verbatim.

Related: #867 tracks log-dead-letters as an opt-in, low-priority diagnostics toggle — this issue argues the default must be on and adds the two false doc claims, which #867 does not mention. #433 (persistent DLQ with inspection and replay) is a much larger feature and does not remove the need for a default log line. #763 is the adjacent case where dead letters are generated in volume by stale event-stream subscriptions — a throttle is required before that becomes loggable at all.

Part of the production-readiness review batch — tracked in #913.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestpriority: highTop priority — high impact, plan nextproduction-goalBlocks or defines the path to production readiness

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions