Skip to content

AL9 Spack for ND Production

Alexander Booth edited this page Nov 6, 2026 · 10 revisions

Available Environments

For environments that have not "been deployed", you'll need to source the respective subspack env script and point at the environment directory with the full path. For example:

source /exp/dune/app/users/abooth/Postdoc/nd-sim-reco/spack_build_instance_261105/share/spack/setup-env.sh
spack env activate -d /exp/dune/app/users/abooth/Postdoc/nd-sim-reco/abooth-nd-run-edep-sim-ndlar012

run-genie:

  • Deployed?: Y
  • Subspack: N/A
  • Environment: dunesw-10_23_00d01-justin-01_06_05-prototype
  • Important details: TODO

run-edep-sim:

  • Deployed?: N
  • Subspack: /exp/dune/app/users/abooth/Postdoc/nd-sim-reco/spack_build_instance_261105
  • Environment: /exp/dune/app/users/abooth/Postdoc/nd-sim-reco/abooth-nd-run-edep-sim-ndlar012
  • Important details: TODO

run-ndlar-flow:

  • Deployed?: N
  • Subspack: /exp/dune/app/users/abooth/Postdoc/nd-sim-reco/spack_build_instance_261106
  • Environment: /exp/dune/app/users/abooth/Postdoc/nd-sim-reco/abooth-nd-run-ndlar-flow-lock
  • Important details: TODO

Transition Status

Coming soon.

Developing with Spack

Adding a Package to the DUNE Spack Stack

  1. Write a recipe. Create a package.py for the package in the dune_spack repo (spack_repo/dune_spack/packages/<name>/), or in your own repo if you're just prototyping.
  2. Test it in a subspack. Make a subspack instance and a from-scratch environment, add the recipe's repo and spack add the package (see the procedure in Creating an Environment from "Scratch" section below for more details).
  3. Open a PR to DUNE/dune_spack. Describe what the recipe adds, how you tested it and any known issues.
  4. To include it in the dunesw release environment, also open a PR to DUNE/dune-release-configs (the included_yaml files). Once it's merged there, the package is built as part of the release instead of by each user.

Creating an Environment from "Scratch"

By from "scratch" I mean a new, empty environment, as opposed to one created from an existing lock file.

Example: Replacing the sim2x2:ndlar012 Container for run-edep-sim

This uses a Spack subspack and a directory environment created from scratch (not seeded from an existing spack.lock; the concretizer chooses the dependency tree). Please note that this example is a mixture of my own informal notes from experimenting (Alex Booth) and Claude. It includes roadblocks I hit for this specific example that might not always be relevant.

1. Create the subspack instance

Source the cvmfs Spack only to get the spack command, then create the instance:

. /cvmfs/dune.opensciencegrid.org/spack/v1.2.2/share/spack/setup-env.sh
cd /exp/dune/app/users/<user>/<workdir>
spack subspack --with-padding $PWD/spack_build_instance_<date>

Note that this assumes that you have already followed these instructions to set up a signing key, which you do once. The instance has its own copy of Spack and its own config. It chains the cvmfs trees (v1.1.1/opt/spack and packages/) as upstreams: packages already on cvmfs are reused, and only new ones are built in the instance.

2. Switch to the instance

Start a fresh shell and source only the instance. The cvmfs setup is not needed again.

. /exp/dune/app/users/<user>/<workdir>/spack_build_instance_<date>/share/spack/setup-env.sh

3. Add the package recipes

Copy the edep_sim and dlpgenerator recipes (package.py in each folder) into the instance's package repo. spack location -p edep-sim shows where. See "Adding a Package to the DUNE Spack Stack" for more details.

Check that they load:

spack info edep-sim
spack info dlpgenerator

4. Create the environment from scratch

Use an absolute path:

ENV=/exp/dune/app/users/<user>/<workdir>/abooth-nd-run-edep-sim-ndlar012
spack env create -d $ENV
spack env activate -d $ENV

5. Add the spec, pinning the ROOT build

spack add edep-sim@4.0.0 +particlebomb target=x86_64_v3 %gcc@12.5.0 ^root/t4p2j3g
  • Put target=... and %gcc@... immediately after the package name.
  • The ^root/t4p2j3g pin is required. The default ROOT 6.28.12 on cvmfs (2rxgsmi) has an incomplete database record, and without the pin CMake fails with Could NOT find Vdt.

6. Concretize and check

spack concretize
spack spec -I | grep -E "root@|vdt|tbb|gsl|fftw"

ROOT should list fftw 3.3.10, gsl 2.8, intel-oneapi-tbb 2022.3.1, unuran and vdt 0.4.6.

Status markers: [^] = reused from cvmfs, [+] = installed in the instance, - = will be built.

7. Install

spack install -j 16 2>&1 | tee ~/edep-install.log

Geant4 10.7.4 and dlpgenerator build first, then edep-sim.

8. Make available for others on the GPVMs as a directory-based environment

Give the environment and the subspack the correct permissions.

chmod -R go+rX /exp/dune/app/users/abooth/Postdoc/nd-sim-reco/abooth-nd-run-edep-sim-ndlar012
chmod -R go+rX /exp/dune/app/users/abooth/Postdoc/nd-sim-reco/spack_build_instance_261105

Troubleshooting notes

  • spack find inside an env lists only that env's specs. Use spack -E find to search everything.
  • Find by hash with a leading slash, e.g. spack -E find -ld /t4p2j3g.
  • A complete ROOT shows its compiler and dependencies in spack -E find -ld. A broken record (2rxgsmi) appears under "no compilers" and triggers Missing dependency not in database warnings.
  • Why the pin is needed: the concretizer reuses installed records. The incomplete ROOT record has no vdt/fftw/gsl/tbb edges, so they never reach CMAKE_PREFIX_PATH. Starting from the cvmfs spack.lock and removing its roots does not help, because the concretizer reuses installed records, not the lock.

Deployment

Coming soon.