Fix the two races behind the flaky socket integration tests

[ptesavol]

The intermittent macOS CI failures of WebsocketClientServerTest / ConnectionLockingTest (the "flaky socket tests" that have forced retriggers on most PRs, including #34) turn out to be two real defects in WebsocketConnection, not test timing or port assumptions. Both reproduce locally and both are fixed at the library level — no test was modified.

Bug 1 — first frames silently dropped (the hangs/timeouts)

setSocket() attached the rtc message callback immediately, but the application's Data listener is only registered later — inside the Connected event the server emits. A frame arriving in that window was pumped through emit<Data> with zero listeners and vanished:

• WebsocketClientServerTest.TestClientCanTrasmitMessageToServer: the test's promise never resolves → hang until the ctest timeout.
• ConnectionLockingTest.CanLockConnections: the first frame on the wire is the handshake request → connection never materializes → folly::FutureTimeout after 10 s, hasConnection() == false (exactly the CI failure output).

Fix: setSocket() no longer attaches the message callback. A new startReceiving() attaches it — the server calls it right after emitting Connected to the application, the client before open() (its listeners are registered before connect()). This loses nothing: libdatachannel queues incoming frames internally until a message callback is attached and flushes them synchronously at attach (Channel::onMessage → flushPendingMessages()), so the rtc queue bridges the gap.

While reading the rtc source I also closed a sibling hole: rtc does not retro-fire the open callback, so a handshake completing before setSocket() attached onOpen would permanently swallow Connected. setSocket() now checks readyState() after attaching and emits Connected itself in that case (mConnectedEmitted keeps the two paths idempotent).

Bug 2 — teardown deadlock (the "close timeout" hangs)

close()/destroy() called socket->resetCallbacks() while holding mMutex. rtc's synchronized_callback holds its own mutex during callback invocation, and assignment/reset blocks until in-flight callbacks return — and our rtc callbacks lock mMutex. Classic AB-BA inversion:

• main thread: mMutex → waits on callback mutex (inside resetCallbacks())
• rtc thread: callback mutex → waits on mMutex (inside our onClosed)

Observed as tests hanging forever after WebSocket close timeout in the rtc log — this was still reproducible after fixing bug 1 (3 hangs in the first 107 post-fix runs), which is what exposed it as a separate mechanism.

Fix: mark mDestroyed and copy the socket pointer under mMutex, then call resetCallbacks()/close() outside it.

Evidence

Loop	Pre-fix	Post-fix
WebsocketClientServerTest ×N (Debug)	3/200 hung	0/500
ConnectionLockingTest ×N (Debug)	2/100 failed	0/200
Both suites (Release, CI config)	—	0/225
Full streamr-dht suite	—	81/81 (Debug and Release)
trackerless-network suites	—	green (Debug and Release)

The test.sh retry/timeout mitigations (--repeat until-pass:2 --timeout 300) are left in place as general hygiene. MODERNIZATION.md's known-issue record is updated with the resolution.

🤖 Generated with Claude Code

[cursor]

Bugbot is not enabled for this team, so this pull request was not reviewed.

Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs.