Host transport: connections that never complete the WebSocket handshake permanently occupy the current/pending slots — player goes silent until restart

[bbangert]

Standard disclaimer that I investigated this with Claude Fable and validated its findings.

Summary

On the host (IXWebSocket) transport, an incoming connection is registered into the connection manager on TCP accept, before the HTTP/WebSocket handshake. If the handshake never completes (port scanner, TCP health-check/reachability probe, a client whose request is lost on flaky Wi-Fi), no WebSocketMessageType::Close is ever delivered for that socket, so on_connection_lost never runs and the phantom occupies a slot forever.

Since the manager holds at most two connections (current_connection_ + pending_connection_), two such phantoms wedge the player permanently: every subsequent real client is rejected in on_new_connection ("Already have pending connection, rejecting new connection") and observes complete silence — no client/hello, no client/goodbye, not even a
close — until the process restarts.

Field symptom: Music Assistant discovers the player, opens the WebSocket, gets 101 Switching Protocols, waits for client/hello, and times out — indefinitely, on every retry. We root-caused a user report with exactly this signature on a Raspberry Pi Zero player (armv6 Linux, host transport). Note that a diagnosing developer's own "is the port reachable" TCP probe is
itself a phantom-creating touch, so the wedge is easy to trigger while investigating it.

Reproduction

Reproduces with the in-tree example — no external project needed:

cmake -B build -DCMAKE_BUILD_TYPE=Release -DBUILD_EXAMPLES=ON
cmake --build build -j
./build/examples/basic_client/basic_client -l debug
# in another shell (Python 3.8+, stdlib only):
python3 reproduce_phantom_wedge.py            # attached script

Output against main @ bf9e085:

Phase A (sanity): real WebSocket connect on fresh server
  -> client/hello received: opcode=1 '{"type":"client/hello",...'

Phase B (wedge): 2 raw TCP touches that never speak HTTP, then a real WebSocket connect
  raw TCP touch #1 (connect, send nothing, close)
  raw TCP touch #2 (connect, send nothing, close)
  -> NO client/hello: (complete silence — no frames at all)

RESULT: WEDGED. Both connection slots are held by phantom connections; every
future client is silently rejected until the process restarts.

Re-running the script confirms the wedge is persistent (Phase A then fails
too). Server-side debug log for the sequence:

D sendspin.ws_server: New client connection (synthetic sockfd ...)
D sendspin.conn_mgr: No existing connection, accepting as current        <- phantom #1
WebSocketServer::handleConnection() HTTP status: 400 error: Error reading HTTP request line
D sendspin.ws_server: New client connection (synthetic sockfd ...)
D sendspin.conn_mgr: Existing connection present, setting as pending for handoff   <- phantom #2
WebSocketServer::handleConnection() HTTP status: 400 error: Error reading HTTP request line
D sendspin.ws_server: New client connection (synthetic sockfd ...)      <- real client
W sendspin.conn_mgr: Already have pending connection, rejecting new connection

Note there is no close/cleanup line for either phantom — they are never
released.

Versions tested (all reproduce)

ref	result
v0.5.0 (573efd5)	wedged
v0.6.1 (ef1157f)	wedged
v0.6.1 + PR #70's connection_manager.cpp change	wedged (phantoms now log "Cannot send hello - not connected", still never released)
main @ bf9e085	wedged

Environment: Linux x86_64 (also observed on armv6 Raspberry Pi Zero), GCC,
IXWebSocket via FetchContent as configured by the tree.

Root cause

src/host/ws_server.cpp — the IXWebSocket setOnConnectionCallback fires
on TCP accept, before the server-side WebSocket handshake. The
SendspinServerConnection is created and handed to
new_connection_callback_ (→ ConnectionManager::on_new_connection, which
stores it as current/pending) right there (ws_server.cpp ~lines 44–101 on
main).

Cleanup is only wired to ix::WebSocketMessageType::Close
(ws_server.cpp:87-92), which is only delivered for connections that
completed the handshake. A socket that fails the handshake surfaces solely
as IXWebSocket's handleConnection() HTTP 400 log line — no Close, no
Error message reaches the registered message callback — so
connection_closed_callback_ / on_connection_lost never run for it.

Degradation is two-stage:

• 1 phantom: real clients land in the pending slot and are greeted.
  A completed handshake with connection_reason: "playback" wins the
  handoff and evicts the phantom (self-heals). But a "discovery"-reason
  connection loses should_switch_to_new_server against the phantom
  ("Default handoff decision: keep existing") and is disconnected with
  client/goodbye another_server — so a discovery-only server (Music
  Assistant idling) can already be locked out by a single phantom.
• 2 phantoms: hard wedge as above.

Relation to PR #70

PR #70 ("Fix ESP-IDF server-initiated discovery") addresses the same
underlying design weakness on the ESP transport — a connection registered
at accept whose hello is never armed because the GET callback doesn't fire.
Its shared-code change (arm hello from on_new_connection) does not
help here: the hello attempt fires against a phantom, hits
is_connected() == false ("Cannot send hello - not connected"), is dropped
without retry, and the phantom is still never released. Both issues would
be resolved by the same principle: treat a connection as managed only
once its WebSocket session is established, and tear down connections that
die before establishment.

Suggested fix direction

On the host transport, either:

1. Defer registration: create/register the SendspinServerConnection (call
   new_connection_callback_) from the WebSocketMessageType::Open branch
   instead of the accept callback, so only established sessions ever occupy
   a slot; or
2. Keep accept-time registration but add a teardown path for
   never-established connections — e.g. invoke
   connection_closed_callback_(synthetic_sockfd) when handleConnection
   fails the handshake (IXWebSocket reports this), or a
   handshake-completion timeout in ConnectionManager::loop() that drops
   managed connections that never became connected.

(1) mirrors what the ESP httpd path effectively does — its open callback
runs after the upgrade — which is why ESP-based players don't exhibit this.

reproduce_wedge.py