EUnit fails with 0 failures: a group is cancelled with {shutdown,timeout} and nothing names it

[Taure]

Four runs in a row - #372, #373, main after both, and #374 - ended EUnit like this:

*unexpected termination of test process*
::{shutdown,timeout}
=======================================================
  Failed: 0.  Skipped: 0.  Passed: 1580.
One or more tests were cancelled.
===> Error running tests

Zero failures, a red build, and no test named. It reproduces on CI only; the same commits run 1573-1580 tests with 0 failures locally.

What has been ruled out

• Fixture setup running under eunit's 5s default. Real gap, closed in test: give the property fixtures a hang detector, not a performance budget #374 - four property modules had {timeout, N, ...} on the property but not on the {setup, ...} around it. The run still fails the same way afterwards, with the same count, so this was not the cause.
• prop_reconnect_lifecycle being slow. It is the last module header printed before the abort, but it runs in 1.9s on its own and its property has a 60s timeout.

The lead worth following

The abort lands one second after the last module header. No timer in the tree is one second, so the group that timed out is almost certainly not the one whose name was printed last - eunit reports the cancellation when it reaps, not when the timer starts. Finding the group that actually holds the expiring timer is the whole task.

Until it is found, a red EUnit on this repo means nothing, which is worse than the flake.

[Taure]

Reopening. Three fixes in, and I have been wrong three times — but the evidence is now sharp enough to be worth writing down properly.

It is not ancient, and it is not the property modules

main's CI, most recent first:

Commit	EUnit
e4f87d7 (#374)	fail
755b5c0 (#375)	fail
16ee38d (#373)	fail
57c2726 (#372)	fail
dfc56ed (#371)	success

It starts at #372 - the player-erase hook, the readiness gate and derived queue claims. Everything before is green. That is the first thing to look at, and I did not check it until now.

What the three fixes did and did not do

• test: give the property fixtures a hang detector, not a performance budget #374 raised the property timeout floor from 60s to 300s. Real: prop_reconnect_lifecycle was managing 21 of 25 iterations inside 60s on a CI runner. Did not fix the run.
• test: give the property fixtures a hang detector, not a performance budget #374 also wrapped each fixture in {timeout, 120, ...} to cover setup/0. That pre-empted the 300s inner timeout, because the outer covers the whole fixture including its tests. Fixed in test: three real timeout and fixture-hygiene fixes (does not fix #376) #378 (900s).
• test: three real timeout and fixture-hygiene fixes (does not fix #376) #378 also clears a possibly-leaked no_link meck before creating one. Good hygiene; did not fix the run.

After all three, prop_reconnect_lifecycle passes on CI in 1.23s where it had never passed. So those were real problems. They were not this one.

What the failure actually looks like now

• Always Failed: 0. Skipped: 0. Passed: ~1580. and ::{shutdown,timeout}.
• Always at the very end of the run, ~6.5 minutes in, every time.
• The victim alternates between prop_reconnect_lifecycle and prop_zone_invariants - the two heaviest world-simulation fixtures - and is killed sub-second after its module header, which no timeout in the tree is short enough to explain.
• Reproduces on CI only. Locally the same commits run 1573-1591 tests, 0 failures, including with --cover (which is the only flag CI adds, and only on pull_request).

The next step I would take

Not another timeout. Bisect against #372: run main at dfc56ed and at 57c2726 on a runner, and if it reproduces, work out which of that PR's tests leaves state a later world fixture cannot start against. The readiness marker and the fixture extensions installed into persistent_term are the obvious candidates, and both are new in #372.

Until then a red EUnit on this repo means nothing, which is worse than the flake it is reporting.

[Taure]

Correction, and the actual cause.

The bisect table above is wrong. A probe branch at #371 - the supposed last green - fails too. That green was luck. This is intermittent and always was, which is itself the clue I should have taken from it.

What it is

asobi_match_server:waiting/3 stops with {shutdown, timeout} after ?WAITING_TIMEOUT, sixty seconds. Five test modules start one with start_link and never unlink. asobi_match_server_tests starts 49 and stops 36.

Thirteen linked match servers sit in waiting on every run. A minute later they exit {shutdown, timeout}, and the link carries that into whatever eunit process is running by then - usually a property module, because those run last.

Nothing failed. A group was killed from outside, which is why the reason is exact, the victim alternates, the gap after the module header is sub-second, and the name is undefined.

CI-only because it is a race with total run time: the suite has to still be alive sixty seconds after the leak.

Fix in #381.