stdlib test issue with pasbuild

[Schmitty2005]

When using the command pasbuild test -m blaise-stdlib --compiler compiler/target/blaise, testrunner is using one thread at 100% for over a half hour. Sometimes multiple threads start, then disappear. I ended up stopping (ctrl-z).

There is no output on the terminal during that time.

Is that expected? I could not see these tests needing to take longer than 10 minutes.

Any suggestions?

[graemeg]

Thanks for the clear report — you're not doing anything wrong, and it's genuinely easy to mistake this for a freeze. Two things are combining here, and neither is a real hang in the normal case.

1. The test runner is silent by design until the very end

The default blaise.testing text runner prints nothing per-test — no dots, no progress, no "Running X…". It runs all ~290 stdlib tests quietly and emits a single line at the finish:

OK (290 tests)

So an empty terminal is expected for the whole run — a healthy run and a stuck run look identical right up until that final line appears. On my machine the whole thing finishes in about 8 seconds (OK (290 tests)), so a healthy build+run should never sit there for half an hour.

If you'd like to see it making progress, run the runner directly with --verbose, which prints one line per test:

pasbuild test-compile -m blaise-stdlib --compiler compiler/target/blaise
cd stdlib/target && ./TestRunner --verbose

That immediately tells you whether it's stuck (and on which test) versus just quietly working.

2. The likely culprit: a networking/async test blocking on a socket

The stdlib suite includes a batch of tests that open real loopback sockets and run the async reactor/scheduler: Sockets, NetTcp, HttpServer, WebSocketServer/WebSocketClient, Reactor, Scheduler, and an AsyncStress test. Any of those can legitimately block at run time — a listening port already in use, a loopback connection that never completes (common inside restricted containers, VMs, or with a firewall on lo), or a reactor waiting for an event that never arrives. Because the runner is silent, a single blocked test pins one core and looks like a total freeze.

--verbose will name the exact test it's parked on. If it's one of the networking ones, you can confirm by skipping it and seeing the rest fly through, e.g.:

./TestRunner --suite TListOpsTests   # a fast, non-network suite — should finish instantly

About the "multiple threads that start then disappear"

That part is normal and unrelated to the hang: the compiler compiles each unit in its own worker thread (parallel codegen), so during the build phase you'll briefly see several threads spin up and exit as units finish. That phase is also silent, but it's quick.

One thing worth checking

Since a fresh clone has no compiler/target/blaise (both target/ and releases/ are gitignored), how did you produce the compiler binary you're passing to --compiler? If it came from a hand-rolled bootstrap off an older release, a stage mismatch can make it miscompile a unit — a much rarer cause, but worth ruling out. The safe path is the cold-bootstrap in the README's Building → Bootstrap from a release section. If ./scripts/fixpoint.sh prints FIXPOINT_OK for you, your binary is sound and the hang is the networking-test issue above.

Could you run with --verbose and paste the last line it prints before it stalls? That will pin it down exactly, and if it's a network test hanging in a way it shouldn't, we'll treat it as a bug and fix it. Thanks again for flagging this — the silent-by-default output is a real papercut, and it's a good argument for the runner printing progress (or a per-test timeout) so a stall is obvious.

[graemeg]

Ah wait, I managed to reproduce it. Seems it's related to the C10k-style stress test, where it spawns 2000 client fibers on one worker - each opening it's own TCP connection to 127.0.0.1:29414. Let me dive deeper.

[Schmitty2005]

Too late but, FYI,

THttpFiberTests.TestFiberHttpGet ... OK
THttpFiberTests.TestFiberHttpKeepAlive ... OK
THttpFiberTests.TestFiberHttpConcurrent ... OK
THttpFiberTests.TestFiberHttpHighConcurrency ... 

Stuck on HighConcurrency on my system. Probably on yours as well.

[graemeg]

Got it

It's file-descriptor exhaustion, and the reactor busy-loops instead of coping

Inspecting your stuck process:

• Soft limit Max open files = 1024 (the common Linux default)
• It was sitting at exactly 1024 open fds — completely maxed out
• 258 established loopback connections to 127.0.0.1:29414, plus the listener still open
• A single thread in state R (running in userspace, not blocked on any syscall) — i.e. a busy-loop, which is why one core sits at 100%

The test that hangs, TestFiberHttpHighConcurrency, deliberately fires 2000 concurrent client fibers, each opening its own TCP connection to the fiber HTTP server. Running all 2000 at once needs roughly ~4000 file descriptors (a client fd and a server-accepted fd per connection). That blows straight past a ulimit -n of 1024.

When the process runs out of fds, accept4()/socket() returns EMFILE ("too many open files"). Our fiber accept path (async.io.FiberAcceptT) currently only knows how to handle two cases — "interrupted" (retry) and "would block" (park until readable). It doesn't handle EMFILE, so:

1. The listening socket still has a pending connection → the epoll reactor reports it readable.
2. We call accept4() → EMFILE (no fd available).
3. The socket is still readable → reactor reports it ready again → back to step 2.

That's a tight epoll-ready → accept-EMFILE → repeat spin with no syscall in between — hence 100% on one core and total silence. And because the stalled client fibers never finish, the test's completion counter never reaches 2000, so it never calls Stop(), so the scheduler never returns. Hard hang.

Why it's environment-sensitive: on a machine with a higher ulimit -n, the same test either completes or fails cleanly (I see it fail with http server start, not hang, because my limit is 1M). Your machine's default 1024 is what turns a should-be-clean failure into a spin.

Immediate workaround

Raise the fd limit for that shell before running the tests:

ulimit -n 65536
pasbuild test -m blaise-stdlib --compiler compiler/target/blaise

(That's session-local and safe. You can also skip the stress test while iterating: cd stdlib/target && ./TestRunner --verbose and, if you want, filter with --suite to run everything except that one.)

You can free the currently-stuck process with Ctrl-C (or kill 1277996 if the terminal is wedged) — it's holding port 29414 and ~1000 sockets.

What we'll fix

This is two bugs and both deserve fixing (I've logged them):

1. The reactor must not busy-loop on EMFILE. The accept path should back off (or fail the accept cleanly) when no fd can be allocated, rather than spinning while the listener is readable. A stress test hitting a resource limit should degrade gracefully, never wedge.
2. The test shouldn't silently depend on an unstated host limit. 2000 simultaneous open connections vs a 1024 fd default is an unforced trap. It should cap in-flight connections (a fiber semaphore) so open fds stay well under the limit, or raise/skip based on the actual ulimit -n with a clear message.

And your original observation stands as a third, ergonomic fix: the runner being silent until the end makes any stall indistinguishable from a freeze — a per-test progress line or a watchdog timeout would have told you instantly which test was stuck.

Thanks again for the great report and for leaving the process up — that turned "probably a network test" into a precise root cause.

[graemeg]

Fixed — thanks again for the report and for confirming the ulimit workaround; that made the diagnosis airtight. The fix is committed locally (df556de4) and will land on master in the next batch. It's two parts, because the hang was really two problems:

1. The reactor no longer busy-loops on fd exhaustion

The core bug: when accept4() fails with EMFILE (out of file descriptors), the pending connection keeps the listen socket readable, so the epoll reactor immediately re-reported it ready → accept → EMFILE → repeat, a tight loop with no syscall block (that's your 100%-on-one-core with no output).

I added an FdExhausted errno classifier (EMFILE/ENFILE) across all the per-platform errno profiles, and the fiber accept path now backs off on it — it parks briefly on the timer heap so other fibers can run and close their connections (freeing fds), then retries — instead of spinning. Measured effect on your exact scenario: the process went from 100% CPU, spinning to ~0.5% CPU, cleanly parked in epoll_wait. So even a genuinely fd-starved server degrades gracefully now instead of wedging a core.

2. The stress test no longer demands more fds than exist

TestFiberHttpHighConcurrency opened 2000 simultaneous connections (~4000 fds) — which structurally cannot complete under the default ulimit -n 1024, no matter how good the reactor is (the excess connections just sit in SYN-SENT forever). It now raises RLIMIT_NOFILE toward what it needs and scales the client count to the fd budget the OS actually grants (floor of 200 so it's still a real concurrency test). It runs the full 2000 when the limit allows and quietly scales down otherwise.

Result

The full stdlib suite now completes cleanly under ulimit -n 1024 — your exact command:

$ ( ulimit -n 1024; pasbuild test -m blaise-stdlib --compiler compiler/target/blaise )
...
OK (290 tests)

No more hang, no workaround needed. (The ulimit -n 65536 bump is still a fine thing to do if you're running your own high-concurrency load, but the test suite no longer requires it.)

Verified across the full compiler suite (OK 4754, both build backends) and all four self-hosting fixpoints. The new FdExhausted classifier is covered by unit tests in both the libc and freestanding (--static) profiles.

Your original observation about the silent runner still stands as a real papercut — a stall is indistinguishable from a freeze when nothing prints until the end. That's on the list to improve (per-test progress and/or a watchdog). Thanks for the sharp report; it turned into a genuine reactor-robustness fix.

[Schmitty2005]

You're welcome! And thank you for fixing it and making Blaise!