test(isolation): 20 reflexive tests fail in the full suite but pass alone — Hypatia.Supervisor is torn down mid-run

[hyperpolymath]

Context

E2E — Elixir Scanner Pipeline is red on main. With #642 making the output visible, the
cause is now measurable rather than mysterious.

Reproduced locally (Elixir 1.18.3 / OTP 27, mix deps.get clean):

Run	Result
mix test (full suite)	1398 tests, 20 failures
mix test test/reflexive_test.exs (alone)	16 tests, 0 failures

Every one of the 20 failures is in test/reflexive_test.exs with the same shape:

** (exit) exited in: GenServer.call(Hypatia.Supervisor, {:terminate_child, Hypatia.SelfDiagnostics}, :infinity)
    ** (EXIT) no process: the process is not alive or there's no process currently
              associated with the given name, possibly because its application isn't started
  test/reflexive_test.exs:29: Hypatia.ReflexiveTest.take_supervised/1

The file passing in isolation is the key datum: this is a test-isolation defect, not a
product bug. Hypatia.Supervisor is alive at the start of an isolated run and dead by the
time reflexive tests run in the full suite.

Likely mechanism (to be confirmed, not assumed)

take_supervised/1 (test/reflexive_test.exs:29) removes a child from the live application
supervisor so start_supervised!/1 can own a clean instance:

Supervisor.terminate_child(Hypatia.Supervisor, module)
Supervisor.delete_child(Hypatia.Supervisor, module)
start_supervised!(module)
on_exit(fn -> ... Supervisor.start_child(Hypatia.Supervisor, module) end)

Two candidate causes, both consistent with the evidence:

1. Restore never happens. If a test crashes before on_exit runs, or start_child
   fails (the spec was deleted), the child stays absent and later take_supervised calls
   compound the damage.
2. max_restarts exceeded. lib/application.ex:90 uses strategy: :one_for_one with
   the default max_restarts: 3, max_seconds: 5. Repeated terminate/restart churn across
   many tests — plus any child that crash-loops — trips the limit and the whole supervisor
   shuts down, taking Hypatia.Supervisor with it. This matches "no process ... its
   application isn't started" exactly.

Hypatia.ReflexiveTest is async: false, so ordering is deterministic per seed — the failure
should be reproducible with a fixed --seed.

Changes

Diagnosis first — do not guess at a fix:

1. Confirm which of the two mechanisms applies: run the full suite with a fixed --seed,
   and assert Process.whereis(Hypatia.Supervisor) in setup to find the first test at
   which it is already nil. Bisect by test file.
2. Then fix at the right level:
   • If restore-failure: make take_supervised/1 restore idempotently and register the
     on_exit before the mutation, so a crash mid-setup still restores.
   • If max_restarts: stop mutating the live application supervisor from tests at all.
     Preferred shape — start these GenServers under the test's own supervisor via
     start_supervised!/1 with an explicit name:, and leave Hypatia.Supervisor
     untouched. A test should not need to dismantle the application to observe it.
3. Consider raising max_restarts in lib/application.ex:90 only if it is genuinely
   warranted in production — not as a way to paper over test churn.

Acceptance criteria

• mix test is 0 failures on a clean checkout, at three different seeds.
• mix test test/reflexive_test.exs still passes in isolation (16 tests).
• No test terminates or deletes a child of Hypatia.Supervisor, or if it must, the
  restore is proven by asserting the child is alive again after the suite.
• E2E — Elixir Scanner Pipeline green on main.

Test plan

• Unit: a regression test asserting Process.whereis(Hypatia.Supervisor) is non-nil at the
  end of reflexive_test.exs.
• Suite: full mix test at seeds 0, 1, and a random one — all green.
• CI: the E2E job green on a PR branch, with fix(tests): e2e.sh discarded all mix test output, making CI undebuggable #642 merged so a regression is legible.

Model tier

Sonnet — bounded, single-repo, no new design; the mechanism is narrowed to two
candidates and the acceptance criteria are mechanical. Escalate to Opus only if the
cause turns out to be a genuine supervision-tree design problem rather than test hygiene.

Related

• fix(tests): e2e.sh discarded all mix test output, making CI undebuggable #642 — makes this failure visible at all (e2e.sh was discarding the output).
• fix(ci): arangodb probe can never pass (401) + stress-test fake/impossible steps #640 — the other two red jobs in tests.yml (arangodb health check); independent cause.
• lib/application.ex:90 — the :one_for_one / max_restarts site.
• test/reflexive_test.exs:29 — take_supervised/1.

[hyperpolymath]

Correction to the original diagnosis, and a partial fix

Two things in the opening post were wrong, and the corrected picture is better.

Wrong: "Hypatia.Supervisor is torn down mid-run"

I instrumented test_helper.exs with a monitor on Hypatia.Supervisor and ran the full
suite. No :DOWN ever fired — the supervisor survives the whole run. The
max_restarts hypothesis is dead.

Wrong: "20 failures, all in reflexive_test"

That was one seed. The failing set changes with the seed:

Seed	Failures	Reflexive involved?
1	9	no
7	57	yes
42	15	yes
0	10	no — instead dashboard, root_hygiene, workflow_audit, metrics, recipe_health

So this is not one bug. It is order-dependent state leakage with several independent
sources, of which reflexive was merely the loudest.

Confirmed: the actual reflexive mechanism

take_supervised/1 registered its restoring on_exit after mutating the live
supervisor. If anything in between raised, the child was deleted and never restored — for
the rest of the suite — so one root failure cascaded into all later tests needing that
module.

Proved by injecting a raise after the delete and asking the supervisor what children it has
(load-independent, unlike suite counts):

[[PROOF]] OLD  -> child present after raise: false     <- lost forever
[[PROOF]] NEW  -> restore registered, runs on_exit

Fixed in #644: register the restore first, make it idempotent, and inspect
Supervisor.start_child/2's return instead of discarding it.

Also found: the suite is load-sensitive

reflexive_test.exs (and others) synchronise with :timer.sleep(30) / :timer.sleep(50).
Under load those flake, which made a whole-suite A/B of the fix inconclusive — counts
moved in both directions across seeds for reasons unrelated to the change. Worth replacing
with deterministic waits (poll-until-condition, or assert_receive on a telemetry event);
until then, suite-count comparisons on a busy machine are not evidence.

Revised remaining work

• reflexive cascade — fix(test): register the supervisor restore BEFORE mutating (#643) #644
• the other order-dependent failures (dashboard, root_hygiene, workflow_audit,
  metrics, recipe_health) — find each leak; these are independent of the above
• replace :timer.sleep/1 synchronisation with deterministic waits
• only then is "mix test green at three seeds" a meaningful acceptance gate

Re-tiering: the cascade was cx/L2 and is done. The remainder is cx/L3 — several
independent leaks across unrelated test files, needing real comprehension of each.

[hyperpolymath]

The failure count is currently unmeasurable — CI does not pin the seed

Two consecutive CI runs on this branch, with zero Elixir source changed between them
(git diff --stat 311b504..1dd31af = .github/workflows/tests.yml only, 0 files matching
\.exs?$|^lib/|^test/):

run	ExUnit seed	result
311b504	20912	1398 tests, 73 failures
1dd31af	406044	1398 tests, 79 failures

Same code, different seed, ±6 failures. The dominant cluster is identical in both
(contract_test 42, blackboard_test 40, reflexive_test 33, kin_test 30), so this is
this issue's order-dependence, not a regression.

The consequence is what matters: while the seed floats, nobody can distinguish a real
regression from reshuffled noise. A fix that genuinely removes 6 failures and a seed that
happens to be kinder look identical. I nearly mis-reported the 73→79 delta as a regression
myself, and only caught it by checking that no Elixir file had changed.

Cheap fix, worth doing before any work on this issue

Pin the seed in CI so the count is comparable run to run:

# test/test_helper.exs
ExUnit.start(seed: String.to_integer(System.get_env("EXUNIT_SEED", "0")))

or simply mix test --seed 0 in the workflow. Then:

• a changed failure count means a real change
• EXUNIT_SEED=<n> mix test still reproduces any specific ordering on demand
• a separate scheduled job can rotate seeds deliberately to find new order-dependence,
  which is the only place randomness actually earns its keep

Without this, the acceptance criterion "failures drop from N" is not checkable.

Related

• test: 7 DETERMINISTIC Elixir failures (fail in isolation) — separate from the #643 order-dependent cluster #645 — the 7 failures that are not order-dependent (identical count in isolation and
  in combination), split out so they can be fixed independently of this cascade