Summary
cargo test -p perry-runtime --lib fails a different number of tests on every run, on clean main, with no changes applied. It is a required check (cargo-test), so this is not cosmetic: a suite whose failure count is nondeterministic can hide a real regression indefinitely — a genuine new failure looks exactly like the noise.
Measurement
Ten consecutive runs on 8f05d65f9, unmodified:
3 failed 1 failed 1 failed 3 failed
4 failed 2 failed 2 failed (earlier baseline, same tree)
Seven distinct tests are involved, by frequency across six sampled runs:
| count |
test |
| 5 |
gc::tests::teardown::map_set_owner_records_follow_growth |
| 3 |
gc::tests::teardown::map_set_side_allocations_release_exactly_once |
| 2 |
promise::keyed_table::tests::settling_many_keys_is_not_quadratic |
| 1 |
url::node_compat::tests::path_to_file_url_posix_preserves_relative_trailing_slash |
| 1 |
url::node_compat::tests::path_to_file_url_posix_does_not_add_slash_without_input_slash |
| 1 |
object::prop_plan::tests::read_plan_roundtrip_and_epoch_flush |
| 1 |
gc::tests::teardown::map_set_side_allocations_release_on_thread_exit |
Every one of them passes in isolation. Verified individually:
gc::tests::teardown::map_set_owner_records_follow_growth ok
gc::tests::teardown::map_set_side_allocations_release_exactly_once ok
gc::tests::teardown::map_set_side_allocations_release_on_thread_exit ok
object::prop_plan::tests::read_plan_roundtrip_and_epoch_flush ok
So it is interference under parallel execution, not a defect in any individual test.
Two distinct causes, probably
1. Shared process-global state. The gc::tests::teardown::map_set_* family is 9 of the 14 observed failures. Those tests assert on side-table ownership records and release counts — process-global structures that other tests mutate concurrently. Related prior art: PERRY_GC_FORCE_EVACUATE's test-only mode override is deliberately thread-local "so one test enabling the instrument cannot change the collector's behaviour for any other test" (arena/quarantine.rs), and #7251 found a test made vacuous by libtest ordering, where array::tests built the intrinsic towers first. Same family of problem.
2. A timing assertion. settling_many_keys_is_not_quadratic asserts on complexity, which under a loaded 10-core box running 1600+ tests in parallel is a coin flip.
Why it matters now
cargo-test is a required status check. Today its signal is: "somewhere between 0 and 4 tests failed, and which ones varies." Nobody can distinguish a regression from the baseline noise, which is the same failure mode as a gate that cannot fail — it just fails uselessly instead of passing uselessly.
Suggested fix
- The
map_set_* teardown tests need serialisation against each other and anything else touching the side tables (a shared test lock, or #[serial]), or the global they assert on needs to be per-thread the way the quarantine's mode override already is.
- The quadratic assertion should measure operation counts rather than wall time, or be marked
#[ignore] under parallel runs.
Found while establishing a baseline for #7341 — I nearly filed a false regression against my own change before checking clean main gave the same varying counts.
Summary
cargo test -p perry-runtime --libfails a different number of tests on every run, on cleanmain, with no changes applied. It is a required check (cargo-test), so this is not cosmetic: a suite whose failure count is nondeterministic can hide a real regression indefinitely — a genuine new failure looks exactly like the noise.Measurement
Ten consecutive runs on
8f05d65f9, unmodified:Seven distinct tests are involved, by frequency across six sampled runs:
gc::tests::teardown::map_set_owner_records_follow_growthgc::tests::teardown::map_set_side_allocations_release_exactly_oncepromise::keyed_table::tests::settling_many_keys_is_not_quadraticurl::node_compat::tests::path_to_file_url_posix_preserves_relative_trailing_slashurl::node_compat::tests::path_to_file_url_posix_does_not_add_slash_without_input_slashobject::prop_plan::tests::read_plan_roundtrip_and_epoch_flushgc::tests::teardown::map_set_side_allocations_release_on_thread_exitEvery one of them passes in isolation. Verified individually:
So it is interference under parallel execution, not a defect in any individual test.
Two distinct causes, probably
1. Shared process-global state. The
gc::tests::teardown::map_set_*family is 9 of the 14 observed failures. Those tests assert on side-table ownership records and release counts — process-global structures that other tests mutate concurrently. Related prior art:PERRY_GC_FORCE_EVACUATE's test-only mode override is deliberately thread-local "so one test enabling the instrument cannot change the collector's behaviour for any other test" (arena/quarantine.rs), and #7251 found a test made vacuous by libtest ordering, wherearray::testsbuilt the intrinsic towers first. Same family of problem.2. A timing assertion.
settling_many_keys_is_not_quadraticasserts on complexity, which under a loaded 10-core box running 1600+ tests in parallel is a coin flip.Why it matters now
cargo-testis a required status check. Today its signal is: "somewhere between 0 and 4 tests failed, and which ones varies." Nobody can distinguish a regression from the baseline noise, which is the same failure mode as a gate that cannot fail — it just fails uselessly instead of passing uselessly.Suggested fix
map_set_*teardown tests need serialisation against each other and anything else touching the side tables (a shared test lock, or#[serial]), or the global they assert on needs to be per-thread the way the quarantine's mode override already is.#[ignore]under parallel runs.Found while establishing a baseline for #7341 — I nearly filed a false regression against my own change before checking clean
maingave the same varying counts.