test_0855 and test_0873 fail at np=2 on Linux CI, and 0873 hangs the MPI job (first CI run they have ever had)

[lmoresi]

tests/parallel/test_0855_mesh_smoothing_parallel.py and
tests/parallel/test_0873_adapt_collective_stop_mpi.py both fail at np=2 on
the Linux CI runner, and the second one takes the whole MPI job down with it —
76 minutes of silence until GitHub cancelled the job, with mpirun and
hydra_pmi_proxy still alive at cleanup.

Neither has ever run in CI. They are two of the three files #615 identified as
matching no glob in either test script (scripts/test.sh runs test_075* +
test_10*; scripts/test_levels.sh runs test_07*). This is the first time
they have been executed by CI, and they do not pass.

The sharp end of it: test_0873_adapt_collective_stop_mpi was added in #596
specifically to guard against a rank-divergent collective stop. It has been dead
weight since the day it landed, and now that it runs, it fails and then hangs —
which is the exact failure mode it exists to catch.

Evidence

From the test job of #615 (run 33690271100, np=2, batch of six files):

01:11:04.49  tests/parallel/test_0855_mesh_smoothing_parallel.py ......F
01:11:04.82  tests/parallel/test_0873_adapt_collective_stop_mpi.py F
02:27:58     ##[error]The operation was canceled.
             Terminate orphan process: pid (45533) (mpirun)
             Terminate orphan process: pid (45534) (hydra_pmi_proxy)

So: one of seven failures in 0855, then 0873 fails, then nothing for 76
minutes. A live mpirun with no output is a rank blocked in a collective while
another rank has already left it. No traceback was captured — the job was
cancelled before pytest could summarise, so the assertion messages are still
unknown.

What does NOT reproduce

Everything below is macOS (arm64, Open MPI 5.0.10, PETSc via the shared build),
amr-dev, at np=2:

what was run	result
test_0855 alone	4 passed per rank
test_0873 alone	4 passed per rank
the failing batch of six, alone	14 passed, 1 skipped, 6.6 s
batches 1-4 in sequence, same working directory	42 / 19 / 23 / 14 passed

The last row rules out contamination from the batches CI runs first — that was
the obvious suspect and it is not the cause. So this is Linux-specific, or
specific to something else about the runner, and it needs a Linux reproduction
before anyone theorises further.

Why this is filed rather than fixed

I could not reproduce it on the only machine I have. Guessing at a collective
hang from a platform that does not exhibit it is how three wrong theories got
written last time (#638). What the next person needs first is the traceback: run
the batch on Linux without -x and with a per-mpirun timeout so pytest
gets to print its summary instead of being cancelled.

Consequence for #615

#615 is correct and is doing exactly what it was written to do — it made three
invisible files visible and two of them are broken. But it cannot merge until
this is resolved, because it would give every CI run a two-hour hang. Holding it.

[lmoresi]

Root-caused and fixed in #681. It was never a Linux problem — it is an MPI
implementation problem.

What it was

test_0855::test_parallel_matches_serial_bit_identical spawns a serial
reference subprocess from inside an mpirun, scrubbing the launcher's variables
so the child does not join the parent's job. The scrub list named the Open MPI
family only (OMPI_, PMIX_, PRTE_, PRTERUN_, OPAL_PREFIX). MPICH
advertises itself through PMI_*, so on the Linux runner the child inherited
PMI_FD and aborted in MPI_Init:

[cli_0]: PMIU_write error; fd=9 buf=:cmd=init pmi_version=1
system msg for write_line failure : Bad file descriptor
Abort: internal_Init_thread ... PMI_Init returned -1
assert 15 == 0

macOS runs Open MPI, where that list is complete. Hence green here, red there,
and the operating system had nothing to do with it. The complete list already
existed thirty lines away in serial_reference.py; this test hand-rolled a copy
and the copy drifted.

test_0873 was collateral. test_premise_the_metric_splits_the_ranks
passes in isolation. It failed only downstream of the abort above, which is also
what produced the divergence that hung the batch: rank 0 in _from_plexh5(),
rank 1 in barrier().

Corrections to what I filed

Two things in the original report were wrong, and both were guesses dressed as
observations:

• I called it Linux-specific. It is MPICH-specific, and the CI runner happens to
  use MPICH. The same failure would appear on a Mac running MPICH and not on a
  Linux box running Open MPI.
• I said test_0855 fails "one of seven" and test_0873 fails on its own
  terms. test_0855 fails exactly one test for the reason above, and test_0873
  fails nothing of its own.

Verified

where	result
Linux CI runner, MPICH, both files, np=2	8 passed in 3.81 s
Open MPI 5.0.10, test_0855	4 passed
Open MPI 5.0.10, test_1069 (uses serial_reference)	7 passed, 1 skipped

How it was found

With the supervisor from #678, which is what made this tractable: 76 minutes of
silence became a named diagnosis in six, and isolating each test in its own
mpirun produced the assertion that had never survived a run.

That exposed one real limitation, recorded on #678 as a follow-up: killing on
silence costs pytest's end-of-run failure report, so the supervisor tells you
where the ranks are stuck but not what failed just before.

[lmoresi]

Fixed by #681, now merged. Verified on the Linux/MPICH runner: both files, np=2, 8 passed in 3.81 s.

The coverage hole that hid this — three parallel files matching no glob in either test script — is #615's subject and is now unblocked.