Skip to content

socket.io Manager reconnection is unbounded after a server-disconnect retry fails at the transport level #9542

Description

@lstein

Follow-up to #9540 (thanks @JPPhoto for catching this in review — it was not a merge blocker there).

What happens

#9540 added a bounded retry for server-initiated socket disconnects: io server disconnect is
terminal for socket.io (Socket.ondisconnectdestroy()Manager._close() sets
skipReconnect), so the client drives up to five socket.connect() attempts with a doubling
delay before giving up
(invokeai/frontend/web/src/services/events/useSocketIO.ts, MAX_SERVER_DISCONNECT_RECONNECTS).

That bound only governs server-initiated disconnects. socket.connect() calls Manager.open(),
which resets skipReconnect = false, and the socket's options set neither reconnection nor
reconnectionAttempts — so the manager's own defaults apply: reconnection: true,
reconnectionAttempts: Infinity, reconnectionDelay: 1000, reconnectionDelayMax: 5000,
randomizationFactor: 0.5 (socket.io-client/build/esm/manager.js:21-29). If one of our retry
attempts fails at the transport level — server down, port refused — the manager takes over and
retries indefinitely. Our counter never advances, because a failed attempt emits connect_error,
not disconnect.

@JPPhoto measured four reconnect attempts in 180 ms against a closed port with socket.io-client
4.8.3; against a host that black-holes rather than refuses, the documented backoff applies and the
cadence settles at roughly one attempt every 2.5-7.5 s.

Scope

This is socket.io's default behavior for the app's socket everywhere, not something #9540
introduced: any network outage already produces the same unbounded retry, on main as well. What
is new is the interaction — a path that advertises a five-attempt bound can hand off to an
unbounded one.

Why the obvious fix is wrong

Disabling manager reconnection around the custom retry (socket.io.opts.reconnection = false)
would strand the tab in exactly the scenario the retry exists for: the server disconnects the
socket for an authorization change and is then briefly unavailable — a restart, a proxy blip —
and with no manager retry, nothing would ever bring the socket back and the tab would sit with no
events and Invoke disabled until a reload. Bounding reconnectionAttempts globally has the same
problem for ordinary outages, which today self-heal.

Options

  1. Accept and document. Unbounded transport retry is the app's baseline; the only new part is
    that it can be entered from the bounded path. Cheapest, and arguably correct — a client that
    wants events back should keep trying while the server is unreachable.
  2. Bound in time rather than attempts. Stop retrying if no connection has succeeded within
    some window of a server-initiated disconnect (a minute, say), on the grounds that a socket the
    server dropped for authorization reasons and that cannot re-establish is unlikely to be
    welcome. Leaves the tab dead but quiet; needs a visible "reconnect" affordance to be
    defensible.
  3. Rate-limit the failure case. Keep retrying forever, but raise reconnectionDelayMax for
    the post-server-disconnect path so a refusing port cannot produce bursts like the measured
    four-in-180 ms.

My inclination is 1 or 3 — 2 trades a self-healing failure for a permanently dead tab, which is
the failure #9540 set out to remove. Worth deciding deliberately rather than inheriting the
default by accident.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions