Skip to content

cargo-test: perry-runtime's suite fails a different number of tests on every run #7365

Description

@proggeramlug

Summary

cargo test -p perry-runtime --lib fails a different number of tests on every run, on clean main, with no changes applied. It is a required check (cargo-test), so this is not cosmetic: a suite whose failure count is nondeterministic can hide a real regression indefinitely — a genuine new failure looks exactly like the noise.

Measurement

Ten consecutive runs on 8f05d65f9, unmodified:

3 failed   1 failed   1 failed   3 failed
4 failed   2 failed   2 failed   (earlier baseline, same tree)

Seven distinct tests are involved, by frequency across six sampled runs:

count test
5 gc::tests::teardown::map_set_owner_records_follow_growth
3 gc::tests::teardown::map_set_side_allocations_release_exactly_once
2 promise::keyed_table::tests::settling_many_keys_is_not_quadratic
1 url::node_compat::tests::path_to_file_url_posix_preserves_relative_trailing_slash
1 url::node_compat::tests::path_to_file_url_posix_does_not_add_slash_without_input_slash
1 object::prop_plan::tests::read_plan_roundtrip_and_epoch_flush
1 gc::tests::teardown::map_set_side_allocations_release_on_thread_exit

Every one of them passes in isolation. Verified individually:

gc::tests::teardown::map_set_owner_records_follow_growth              ok
gc::tests::teardown::map_set_side_allocations_release_exactly_once    ok
gc::tests::teardown::map_set_side_allocations_release_on_thread_exit  ok
object::prop_plan::tests::read_plan_roundtrip_and_epoch_flush         ok

So it is interference under parallel execution, not a defect in any individual test.

Two distinct causes, probably

1. Shared process-global state. The gc::tests::teardown::map_set_* family is 9 of the 14 observed failures. Those tests assert on side-table ownership records and release counts — process-global structures that other tests mutate concurrently. Related prior art: PERRY_GC_FORCE_EVACUATE's test-only mode override is deliberately thread-local "so one test enabling the instrument cannot change the collector's behaviour for any other test" (arena/quarantine.rs), and #7251 found a test made vacuous by libtest ordering, where array::tests built the intrinsic towers first. Same family of problem.

2. A timing assertion. settling_many_keys_is_not_quadratic asserts on complexity, which under a loaded 10-core box running 1600+ tests in parallel is a coin flip.

Why it matters now

cargo-test is a required status check. Today its signal is: "somewhere between 0 and 4 tests failed, and which ones varies." Nobody can distinguish a regression from the baseline noise, which is the same failure mode as a gate that cannot fail — it just fails uselessly instead of passing uselessly.

Suggested fix

  • The map_set_* teardown tests need serialisation against each other and anything else touching the side tables (a shared test lock, or #[serial]), or the global they assert on needs to be per-thread the way the quarantine's mode override already is.
  • The quadratic assertion should measure operation counts rather than wall time, or be marked #[ignore] under parallel runs.

Found while establishing a baseline for #7341 — I nearly filed a false regression against my own change before checking clean main gave the same varying counts.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions