Skip to content

[Bug] WebsocketConnection.close() is a tell into the same drop-head mailbox as every outbound frame, so a hot broadcast evicts the close command and closeAll() silently does nothing #985

Description

@pathosDev

Problem

WebsocketConnection.close() is not a close. It is connectionRef.tell({ kind: 'close', … }) — an enqueue into the per-connection actor's mailbox, the same queue every outbound frame goes through. That mailbox is the framework default: BoundedMailbox, capacity 10 000, overflow drop-head.

Drop-head discards the oldest queued message. A close command that has been sitting in the queue while the hub keeps producing is, by construction, the oldest thing in it. So the exact situation in which you want to kick a client — a producer flooding faster than the socket drains — is the situation that evicts the kick. closeAll(1008, 'rate limited') returns normally, connection.isOpen still reads true because the socket was never touched, and the hub's broadcast keeps selecting the connection on every subsequent message.

The same mailbox is also a silent drop stage in front of the backpressure policy. WebsocketPolicy.onBackpressure ('drop' or 'close') is evaluated in write(), which only runs for frames that survived the mailbox. Frames evicted before that never reach the check, and the eviction is drop-head, so the frames that are lost are the oldest ones — for a stream protocol that is the worst possible choice: the receiver gets a gap in the middle and the newest frames on top. The only trace is the actor_mailbox_dropped_total counter; there is no log and no dead letter (#773).

Evidence

Every outbound operation on a connection, including close, is a tellsrc/http/websocket/WebsocketConnection.ts:68-78:

src/http/websocket/WebsocketConnection.ts:68-78
  override tell(message: TOut): void {
    this.connectionRef.tell({ kind: 'out', message });
  }

  sendRaw(frame: WebsocketFrame): void {
    this.connectionRef.tell({ kind: 'out-raw', frame });
  }

  close(code = 1000, reason = ''): void {
    this.connectionRef.tell({ kind: 'close', code, reason });
  }

The hub's kick path, src/http/websocket/WebsocketServerActor.ts:94-97:

src/http/websocket/WebsocketServerActor.ts:94-97
  /** Close every connection. */
  protected closeAll(code = 1000, reason = ''): void {
    for (const client of this._clients.values()) client.close(code, reason);
  }

The connection actor is spawned with no mailbox options, so it takes the default — src/http/websocket/WebsocketServerActor.ts:136-138:

src/http/websocket/WebsocketServerActor.ts:136-138
  private onWebsocketAccept(command: WebsocketAcceptCommand): void {
    this.context.spawn(command.actor, command.name);
  }

which is src/internal/ActorCell.ts:187-198:

src/internal/ActorCell.ts:187-198
    this.mailbox = blueprint.mailbox
      ? blueprint.mailbox()
      // #310 — bounded by default.  Unbounded was the pre-#310 default
      // and is still available via `withMailbox(() => new Mailbox())`
      // for use-cases that need it (deterministic replay, test setups,
      // tightly-controlled throughput).  See `DEFAULT_MAILBOX_CAPACITY`
      // + `DEFAULT_MAILBOX_OVERFLOW` for the chosen ceiling + policy.
      : new BoundedMailbox<TMessage>({
        capacity: blueprint.mailboxCapacity ?? DEFAULT_MAILBOX_CAPACITY,
        overflow: DEFAULT_MAILBOX_OVERFLOW,
        onDrop: (reason) => this._onMailboxDrop(reason),
      });

and the eviction itself, src/mailbox/BoundedMailbox.ts:40-55:

src/mailbox/BoundedMailbox.ts:40-55
  override enqueue(env: Envelope<T>): void {
    if (this.size >= this.capacity) {
      match(this.overflow)
        .with('drop-head', () => {
          // `removeOldest` rather than `dequeueUser`: the latter returns
          // undefined while the mailbox is suspended, which used to make this
          // whole arm a no-op — the queue grew past capacity and the drop was
          // reported anyway.  Counting is gated on an actual removal so the
          // metric cannot claim a drop that did not happen.
          const dropped = super.removeOldest();
          if (dropped !== undefined) {
            this.droppedCount++;
            this.onDrop?.('drop-head');
          }
          super.enqueue(env);
        })

The backpressure policy that never sees the evicted frames, src/http/websocket/WebsocketConnectionActor.ts:179-193:

src/http/websocket/WebsocketConnectionActor.ts:179-193
  private write(frame: WebsocketFrame): void {
    if (this.d.socket.readyState !== WebsocketReadyState.OPEN) {
      this.log.debug(`WebsocketConnectionActor ${this.d.id}: write on non-open socket — dropped`);
      return;
    }
    const buffered = this.d.socket.bufferedAmount?.();
    if (buffered !== undefined && buffered > this.d.policy.maxBufferedBytes) {
      if (this.d.policy.onBackpressure === 'close') {
        this.log.warn(`WebsocketConnectionActor ${this.d.id}: send buffer ${buffered} > ${this.d.policy.maxBufferedBytes} — closing`);
        this.closeSocket(1013, 'try again later');
      } else {
        this.log.warn(`WebsocketConnectionActor ${this.d.id}: send buffer ${buffered} > ${this.d.policy.maxBufferedBytes} — dropping frame`);
      }
      return;
    }

Proposal

Acceptance sketch

  • connection.close(code, reason) closes the socket even when the connection actor has a full outbound backlog.
  • closeAll() on a hub whose clients are all backlogged closes every socket.
  • A test fills a connection actor past DEFAULT_MAILBOX_CAPACITY after issuing a close and asserts the socket was closed.
  • Outbound frames dropped by mailbox overflow are logged or dead-lettered, not only counted.

Verification status

Found in the ten-lens production-readiness review of 2026-08-05 (v0.13.0) and re-verified before filing: reproduced by execution. A real WebsocketConnectionActor was spawned over a fake WebsocketSocketAdapter that records close() calls and counts send()s. After preStart, a close(1008, 'rate limited') was told, followed by 10 000 out commands — the shape of a hub that kicks a client and keeps broadcasting:

W6-11: socket.close() calls = []  (frames written: 10000)
W6-11: close command survived the mailbox? false

The socket was never closed. All 10 000 frames the close was meant to stop were written to it instead.

Adjacent issues: #717 is the same mechanism on the hub's mailbox for the websocket-accept command (a flood evicts the spawn and orphans an upgraded socket). This is the per-connection mailbox and the close command — a different actor, a different queue, and a different consequence (a client you decided to disconnect stays connected). #773 covers the general fact that overflow-dropped messages are never dead-lettered. #651 (server-side heartbeat/idle timeout) would not help: it also closes through connection.close().

Part of the production-readiness review batch — tracked in #913.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingpriority: highTop priority — high impact, plan nextproduction-goalBlocks or defines the path to production readiness

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions