`Module (own workspace)` is flaky on main: two tests race the scheduler gate's process-global state

[CodeGhost21]

What happens

The Module (own workspace) job fails intermittently on main, independent of what a PR changed. In the last twelve main runs it failed four times:

run	job
#129	pass
#128	fail
#127	pass
#125	fail
#124 · #123 · #122 · #121 · #120	pass
#119	fail
#118	fail
#117	pass

I noticed it because it fired on my own merge (#128) and not on the next one (#129, an hour later, containing the same code) — but #118 and #119 predate that work entirely, so this is not new.

The failure is always the same assertion:

thread 'host::test::manual_override_outranks_a_paused_gate_and_is_bounded' panicked at
  crates/tinymemory-module/src/host_test.rs:405:5:
assertion failed: matches!(core_gate::current_policy(), Policy::Paused { .. })
test result: FAILED. 75 passed; 1 failed

Why

tinymemory_core::scheduler_gate keeps its state in two process-global statics:

static GATE: RwLock<Option<Arc<dyn SchedulerGate>>> = RwLock::new(None);
static MANUAL_OVERRIDE_UNTIL: RwLock<Option<std::time::Instant>> = RwLock::new(None);

manual_override_outranks_a_paused_gate_and_is_bounded drives both — clear_manual_override, set_scheduler_gate, set_manual_override, clear_scheduler_gate — and asserts on current_policy() between the writes. store_policy_wakes_sleepers_only_on_resume, in the same file and so the same test binary, drives a gate too.

libtest runs a binary's tests on parallel threads by default, so the two interleave. The assertion that fails is the first read after set_scheduler_gate(gate), which is exactly what a concurrent clear_scheduler_gate from the other test produces. The RwLocks make each individual access safe; they do nothing about two tests wanting different global state at the same time.

Why it is worth fixing rather than re-running

main's CI is the signal everyone uses to decide whether their own branch broke something. At roughly one failure in three, it cannot answer that question — and the failing job is named for the module workspace, so it reads like a real regression rather than a coin toss.

Possible directions

Not proposing a specific fix, but the usual options for global-state tests:

• serialise the tests that touch the gate behind a shared Mutex (or the serial_test pattern) so they cannot interleave;
• give the gate a scoped/injectable handle for tests so no global is involved;
• or, cheapest and least satisfying, move the gate tests into their own integration target so they get their own process.

Happy to do whichever the maintainers prefer — I have no context on why the gate is global, and that probably decides which of these is right.

Environment

Observed on GitHub Actions, main, runs 33671477150 (#128) among others. Not reproduced locally — the interleaving is timing-dependent and CI's parallelism differs from a laptop's.

[CodeGhost21]

Correcting the mechanism I described above — I named the wrong second test.

store_policy_wakes_sleepers_only_on_resume is not a racer. It builds a gate with gate_for_test() and calls gate.store_policy(...) on that local value; it never installs it, so it never touches the globals.

The actual set is three, and the one I missed is the most interesting of them:

test	how it reaches the globals
install_wires_every_seam_this_module_can_supply	indirectly, through install_seams(None) → host.rs → set_scheduler_gate
manual_override_outranks_a_paused_gate_and_is_bounded	directly
override_member_opens_a_window_that_outranks_a_paused_gate (service/test.rs)	directly, plus a manual override through the service member

That third one also widens the diagnosis. It sets a manual override, so the failing assertion has two ways to break rather than one: the gate can be replaced or MANUAL_OVERRIDE_UNTIL can be set by the other test, at which point current_policy() answers Normal while a paused gate is installed. Same root cause, same fix.

The rest of the report stands: process-global statics, parallel test threads, and HostSeamsRestore fixing ordering rather than concurrency.

Fix is up as #131. One thing worth flagging there rather than here: I could not reproduce the race locally — 40 full-suite runs at 16 threads, then 150 runs of just the two gate-installing tests at 8 threads, zero failures. The service test does real async setup before it reaches the gate, so the windows rarely align on a fast machine. The diagnosis is therefore static rather than reproduced, and the PR says so.