tests: robustness gaps - global fd asserts on fixed sleeps, no per-test timeout, dead-port assumptions, bind failure kills the suite

[MDA2AV]

Severity: low — no product code is wrong, but each of these is a future red (or empty) CI run: the fd tests are the likeliest next flake, and two of the gaps turn one failure into zero reported results.

Problem — four related gaps

1. fd-leak assertions are process-global and sleep-synchronized. Fds() counts all of /proc/self/fd (CoreTests.cs#L352, HardeningTests.cs#L144), and the three tests using it settle with fixed sleeps: HardeningTests.cs#L72-L85 (Sleep 200/500, leaked <= 3), CoreTests.cs#L282-L304 and #L309-L337. Test servers "run on a background thread until the process exits" (TestServer.cs#L12-L14), so every earlier test's reactor is still live while baseline and sample are taken — a late connection recycle or any BCL fd activity moves the count. CoreTests.cs#L309 already documents the race it sleeps around ("let earlier tests' recycles settle before the fd baseline").

2. The Runner has no per-test timeout. body() runs synchronously (Runner.cs#L10-L30); one wedged test — a hung Read, an acquire that never completes — hangs the whole suite with only the CI job timeout as backstop, and reports nothing when it fires.

3. Several tests assume literal ports are dead. TCP 5599 (PgTests.cs#L13, dead-backend fail-fast), TCP 9099 (HttpClientTests.cs#L39, refused-connect budget), UDP 9999 (RingHttpClientTests.cs#L73-L87, "a port nothing listens on"). Anything actually listening there breaks — or silently inverts — the test. The a85e389 port-window fix covers ports the harness binds, not ports assumed dead.

4. A bind/listen failure kills the whole suite, not the test. OpenReusePortListener throws a bare InvalidOperationException("bind failed") — no port, no errno (Reactor.Tcp.cs#L382-L430); Reactor.Run has no try/catch (Reactor.Runner.cs#L23-L52) and TestServer starts reactor.Run directly as the thread body (TestServer.cs#L77-L82), so the exception is unhandled on that thread and .NET terminates the process — zero results instead of one FAIL. (Cross-suite collisions don't bind-fail thanks to SO_REUSEPORT — that was the silent-share hazard — so the trigger here is a foreign non-REUSEPORT listener in the window, fd exhaustion, or similar.)

Suggested fix

• fd tests: poll until the count is stable (two identical samples some ms apart) instead of fixed sleeps, and prefer asserting deltas of fds the test itself caused; keep the tolerance as backstop.
• Runner: run body() under a watchdog (Task.Run + Wait(timeoutMs) is enough for synchronous bodies); on expiry print FAIL with a managed thread-stack dump and move on — the suite still exits non-zero.
• Dead ports: derive them instead of hardcoding — bind port 0, note the assigned port, close, use it immediately (the just-freed port is refused/​silent with overwhelming reliability), or reserve from the suite's own TestServer.NextPort() window and never start a server there.
• Bind failure: wrap the harness's thread body in try/catch so startup failures surface through WaitForListen as one failing test; and include port + errno in the core's socket/bind/listen exceptions regardless — "bind failed" alone doesn't say which port or why.

[MDA2AV]

PR #142 covers three of the four bullets and the actionable half of the fourth:

• per-test timeout — Runner runs each body on a worker with a deadline; a wedged test is one FAIL and the suite continues (verified with a deliberately hung probe)
• fd assertions — FdCount.Stable() polls until two samples agree, replacing the fixed sleeps in CoreTests and HardeningTests
• dead ports — TestServer.DeadPort() / DeadUdpPort() bind port 0 and close it; the literals 5599, 9099 and 9999 are gone
• bind failure — the harness now wraps the reactor thread body, so a startup failure surfaces through WaitForListen as one failing test instead of terminating the process with zero results

Left open: the core half of bullet 4 — port and errno in Reactor.Tcp.cs's socket/bind/listen exceptions, which today throw a bare "bind failed". Core changes are frozen at the moment, so that one line is all that remains here.

[MDA2AV]

Three of the four bullets landed in #142, and the fourth's actionable half with them:

• per-test timeout — Runner runs each body on a worker with a deadline
• fd assertions — FdCount.Stable() polls until two samples agree, replacing the fixed sleeps
• dead ports — TestServer.DeadPort() / DeadUdpPort(); the literals 5599, 9099 and 9999 are gone
• bind failure — the harness wraps the reactor thread, so a startup failure surfaces as one failing test rather than terminating the process

Review of that PR then found the guard had made things quieter than the crash it replaced: Reactor.Run opens the listener before InitSharedRingBuffer, OpenWakeFd and OnStart, so a reactor dying in any of those lost the race with WaitForListen's probe and was reported nowhere — and one dying after its test passed left the suite green. Both fixed in the same PR: deaths always reach stderr, and Summary fails the run for any death no test consumed.

Still open: port and errno in the core's socket/bind/listen exceptions — Reactor.Tcp.cs still throws a bare "bind failed" that names neither the port nor the reason. That is the only remaining item here.