Fix MPI on candide (OpenMPI 5 image + latent code bug); containerize & SLURM-ify candide scripts

[cailmdaley]

Update: the user-facing docs that were originally part of this PR (README front door, the candide cluster walkthrough, the MPI run docs) have been moved to the docs-rework PR #739 so this PR stays focused on code/infra. It retains the CLAUDE.md build-loop note, which documents the container build workflow these changes introduce.

---

What

Makes the candide example job scripts actually run ShapePipe through the published container. The container is the supported source of truth (docs/source/container.md), so the example scripts should use it rather than module load intelpython/3 + source activate $HOME/.conda/envs/shapepipe. Verifying the MPI script end-to-end turned up three things in the long-unexercised MPI path that had to be fixed for it to run — a launcher/PMIx mismatch, a code bug, and a stale config — each hidden behind the previous one.

Changes

Job scripts (example/pbs/candide_{smp,mpi}.sh)

• Drop conda entirely. The pipeline runs via apptainer exec against the runtime image, pulled once to a SIF whose path is overridable via $SP_IMAGE.
• SLURM, not PBS. candide moved to SLURM — qsub/#PBS are gone. The scripts now use #SBATCH / sbatch and mpirun -n $SLURM_NTASKS.
• Bind-mount the host clone ($SPDIR) at the same path inside the container so the configs' $SPDIR-relative INPUT_DIR/OUTPUT_DIR resolve identically in- and outside the container.
• Fix a stale config path; propagate the real pipeline exit code instead of always exit 0.
• Un-rot example/pbs/config_mpi.ini (last touched 2020): its MODULE / section names lacked the _runner suffix the loader now requires, so the run died with No module named 'shapepipe.modules.python_example'. Updated to the current names (matching example/config.ini).

Container (Dockerfile)

• Build OpenMPI 5.0.x from source (--with-pmix=internal --with-prrte=internal --disable-dlopen), dropping libopenmpi-dev. The image previously shipped OpenMPI 4.1.4 / PMIx 2, which is wire-incompatible with candide's OpenMPI 5.0.x / PMIx 5 host launcher — so hybrid MPI silently degraded to N independent "rank 0 of 1" processes (wrong science, no error). The stock mpi4py wheel dlopens libmpi.so.40 (ABI-stable across OMPI 4↔5), so uv.lock is untouched.

ShapePipe MPI code (src/shapepipe/pipeline/mpi_run.py, src/shapepipe/run.py)

• Thread module_config_sec through run_mpi → submit_mpi_jobs → worker(). By git history this dates to Multiple Module Runs #415 ("Multiple Module Runs"): worker() gained a module_config_sec parameter, but the MPI submit path wasn't updated in step, so it passes 7 args where 8 are required — worker() missing 1 required positional argument: 'module_runner'. On candide this path wasn't reachable until now (the PMIx mismatch meant MPI never wired up), so fixing the launcher is what surfaced it.

Exit-code propagation (src/shapepipe/shapepipe_run.py)

• main() called run(args) but didn't return it, so exit(main()) was always exit(None) → 0. That means every caught error in ShapePipe — not just MPI — has been exiting 0, invisible to exit $? and CI. Now return run(args), with a regression test. (This is independent of MPI and worth landing on its own.)

CI (.github/workflows/deploy-image.yml)

• Publish images on every branch push, so any PR has a cluster-pullable image to test against before merge.

Verification (on candide)

• ✅ MPI runs end-to-end — the unmodified candide_mpi.sh against the published runtime image. 2-node / 4-task hybrid run: ranks distribute across two nodes (process 0–3 of 4 on n23+n25, not the pre-fix singletons), all three *_example_runner modules produce output, real exit 0 (the script's exit $?), and shapepipe.log records "A total of 0 errors were recorded."
• ✅ candide_smp.sh runs the SMP example pipeline end-to-end through the container, 0 errors.
• ✅ bash -n clean; structural shell-syntax test tier passes.
• ✅ Caught errors now propagate a non-zero exit code (the main() fix), so a failed run no longer looks like success to exit $?.

Question for review — is MPI a used dependency of ShapePipe at all?

We got MPI working on candide (the three fixes above) — but the bigger question this surfaced, @martinkilbinger, is whether it's worth keeping at all. Its entire footprint in the repo is tiny:

• A hard dependency (mpi4py>=4.0 in pyproject.toml) plus the core code (run_mpi in run.py, mpi_run.py).
• Two example job scripts — candide_mpi.sh (candide) and cc_mpi.sh (ccin2p3) — and one active example config (config_mpi.ini).
• Zero production paths: every canfar/candide submission script runs SMP (N_SMP).

And functionally it adds nothing: SMP and MPI call the identical worker() with identical arguments — the same computation behind two dispatchers (worker_handler.py has no MPI; the MPI path just scatters the independent job-list and gathers results). The workload is embarrassingly parallel, so MPI buys only an ergonomic convenience over "many per-node SMP jobs under SLURM."

So: is MPI actually used anywhere, or can the mode (and the mpi4py dependency) be dropped? If it's used, this PR restores it to working order and there's a ready follow-up to harden it against silent desync; if not, removing it (mpi_run.py, run_mpi, the import_mpi branches, mpi4py, the two *_mpi.sh scripts, config_mpi.ini) is clean and contained. Your call — you'd know whether anyone depends on it.

Out of scope (deliberately untouched)

The production submission scripts — scripts/sh/run_scratch_local.sh, init_run_exclusive_canfar.sh, job_sp_canfar*.bash (canfar + modern candide) — and the ccin2p3 scripts. These target workflows we can't fully verify here; note they are still SMP + conda and not yet containerized (a separate future cleanup).

🤖 Generated with Claude Code

— Claude on behalf of Cail

[cailmdaley]

@fabianhervaspeters @martinkilbinger — beyond fixing MPI, this PR doubles as a readiness test for the container itself, and as far as I can tell it's ready for other people to pull it, run it, and try to break it.

Verified end to end on candide today (c03, apptainer 1.4.5, SLURM) against the published :develop-runtime image — observed, not inferred:

• Bundled example via direct apptainer exec → exit 0, "A total of 0 errors were recorded".
• apptainer shell resolves the entry point (/app/.venv/bin/shapepipe_run).
• Real SLURM job — candide_smp.sh → COMPLETED, ExitCode 0:0, 0 errors, bind-mount + exit-code propagation working.
• Branch CI is green.

The user-facing docs for all this — a one-command Quickstart, the candide cluster walkthrough (quota-safe pull → sbatch example/pbs/candide_smp.sh, including the $HOME-quota gotcha), and the MPI run pattern — now live in the docs-rework PR #739, so this PR stays focused on the code/infra. See the README front door and the candide cluster docs there.

So: please pull it and try to break it. The MPI-vs-keep question above still stands separately.

— Claude on behalf of Cail