Twice in one session (2026-07-27 and 2026-07-28), the test job failed at the Import smoke test step, before any test executed. Both cleared on a plain re-run with no code change.
Symptom
UCX ERROR mana_0: iface ... failed to create UD QP TX wr:256 sge:6 inl:64
resp:0 RX wr:4096 sge:1 resp:0 failed: Operation not supported
UCX ERROR uct_iface_open(ud_verbs/mana_0:1) failed: Input/output error
Abort(807552399): Fatal error in internal_Init_thread: Other MPI error, error stack:
internal_Init_thread(71).: MPI_Init_thread(...) failed
MPIDI_UCX_init_worker(86): ucx function returned with failed status
Process completed with exit code 143
Occurrences:
| date |
PR |
run |
| 2026-07-27 |
#430 |
30237979813 |
| 2026-07-28 |
#460 |
30347218663 |
Both on runner runnervmvrwv9.
What it is
MPICH's UCX layer probes mana_0 — the Azure network adapter exposed as an RDMA-capable device — and tries to open a ud_verbs transport on it. When that fails, MPI_Init_thread aborts rather than falling back to a transport that works. Nothing in the repo triggers it; the failing command is a bare import underworld3.
Why it is worth fixing rather than re-running
It fails before any test runs, so it produces a red X that has nothing to do with the change under review. That trains reviewers to re-run on red, which is exactly the habit that hides a genuine intermittent failure later.
Suggested fix
Constrain the transports UCX will consider in the CI environment, so it does not probe the RDMA device at all. In the workflow env:
env:
UCX_TLS: tcp,self,sm
UCX_NET_DEVICES: lo
or set MPIR_CVAR_CH4_NETMOD=ofi / use the mpich conda package built without UCX. Either removes the probe rather than making the abort survivable.
Worth checking whether the dev (conda-PETSc) environment is the only one affected — the failing step uses pixi run -e dev, and the AMR environments use a custom PETSc build with a platform-default MPI.
Twice in one session (2026-07-27 and 2026-07-28), the
testjob failed at the Import smoke test step, before any test executed. Both cleared on a plain re-run with no code change.Symptom
Occurrences:
Both on runner
runnervmvrwv9.What it is
MPICH's UCX layer probes
mana_0— the Azure network adapter exposed as an RDMA-capable device — and tries to open aud_verbstransport on it. When that fails,MPI_Init_threadaborts rather than falling back to a transport that works. Nothing in the repo triggers it; the failing command is a bareimport underworld3.Why it is worth fixing rather than re-running
It fails before any test runs, so it produces a red X that has nothing to do with the change under review. That trains reviewers to re-run on red, which is exactly the habit that hides a genuine intermittent failure later.
Suggested fix
Constrain the transports UCX will consider in the CI environment, so it does not probe the RDMA device at all. In the workflow env:
or set
MPIR_CVAR_CH4_NETMOD=ofi/ use thempichconda package built without UCX. Either removes the probe rather than making the abort survivable.Worth checking whether the
dev(conda-PETSc) environment is the only one affected — the failing step usespixi run -e dev, and the AMR environments use a custom PETSc build with a platform-default MPI.