Repository navigation
AL9 Spack for ND Production
For environments that have not "been deployed", you'll need to source the respective subspack env script and point at the environment directory with the full path. For example:
source /exp/dune/app/users/abooth/Postdoc/nd-sim-reco/spack_build_instance_261105/share/spack/setup-env.sh
spack env activate -d /exp/dune/app/users/abooth/Postdoc/nd-sim-reco/abooth-nd-run-edep-sim-ndlar012
- Deployed?: Y
- Subspack: N/A
- Environment:
dunesw-10_23_00d01-justin-01_06_05-prototype - Important details: TODO
- Deployed?: N
- Subspack:
/exp/dune/app/users/abooth/Postdoc/nd-sim-reco/spack_build_instance_261105 - Environment:
/exp/dune/app/users/abooth/Postdoc/nd-sim-reco/abooth-nd-run-edep-sim-ndlar012 - Important details: TODO
- Deployed?: N
- Subspack:
/exp/dune/app/users/abooth/Postdoc/nd-sim-reco/spack_build_instance_261106 - Environment:
/exp/dune/app/users/abooth/Postdoc/nd-sim-reco/abooth-nd-run-ndlar-flow-lock - Important details: TODO
Coming soon.
-
Write a recipe. Create a
package.pyfor the package in thedune_spackrepo (spack_repo/dune_spack/packages/<name>/), or in your own repo if you're just prototyping. -
Test it in a subspack. Make a subspack instance and a from-scratch environment, add the recipe's repo and
spack addthe package (see the procedure in Creating an Environment from "Scratch" section below for more details). -
Open a PR to
DUNE/dune_spack. Describe what the recipe adds, how you tested it and any known issues. -
To include it in the dunesw release environment, also open a PR to
DUNE/dune-release-configs(theincluded_yamlfiles). Once it's merged there, the package is built as part of the release instead of by each user.
By from "scratch" I mean a new, empty environment, as opposed to one created from an existing lock file.
This uses a Spack subspack and a directory environment created from scratch (not seeded from an existing spack.lock; the concretizer chooses the dependency tree). Please note that this example is a mixture of my own informal notes from experimenting (Alex Booth) and Claude. It includes roadblocks I hit for this specific example that might not always be relevant.
Source the cvmfs Spack only to get the spack command, then create the instance:
. /cvmfs/dune.opensciencegrid.org/spack/v1.2.2/share/spack/setup-env.sh
cd /exp/dune/app/users/<user>/<workdir>
spack subspack --with-padding $PWD/spack_build_instance_<date>Note that this assumes that you have already followed these instructions to set up a signing key, which you do once.
The instance has its own copy of Spack and its own config. It chains the cvmfs trees (v1.1.1/opt/spack and packages/) as upstreams: packages already on cvmfs are reused, and only new ones are built in the instance.
Start a fresh shell and source only the instance. The cvmfs setup is not needed again.
. /exp/dune/app/users/<user>/<workdir>/spack_build_instance_<date>/share/spack/setup-env.shCopy the edep_sim and dlpgenerator recipes (package.py in each folder) into the instance's package repo. spack location -p edep-sim shows where. See "Adding a Package to the DUNE Spack Stack" for more details.
Check that they load:
spack info edep-sim
spack info dlpgeneratorUse an absolute path:
ENV=/exp/dune/app/users/<user>/<workdir>/abooth-nd-run-edep-sim-ndlar012
spack env create -d $ENV
spack env activate -d $ENVspack add edep-sim@4.0.0 +particlebomb target=x86_64_v3 %gcc@12.5.0 ^root/t4p2j3g- Put
target=...and%gcc@...immediately after the package name. - The
^root/t4p2j3gpin is required. The default ROOT 6.28.12 on cvmfs (2rxgsmi) has an incomplete database record, and without the pin CMake fails withCould NOT find Vdt.
spack concretize
spack spec -I | grep -E "root@|vdt|tbb|gsl|fftw"ROOT should list fftw 3.3.10, gsl 2.8, intel-oneapi-tbb 2022.3.1, unuran and vdt 0.4.6.
Status markers: [^] = reused from cvmfs, [+] = installed in the instance, - = will be built.
spack install -j 16 2>&1 | tee ~/edep-install.logGeant4 10.7.4 and dlpgenerator build first, then edep-sim.
Give the environment and the subspack the correct permissions.
chmod -R go+rX /exp/dune/app/users/abooth/Postdoc/nd-sim-reco/abooth-nd-run-edep-sim-ndlar012
chmod -R go+rX /exp/dune/app/users/abooth/Postdoc/nd-sim-reco/spack_build_instance_261105-
spack findinside an env lists only that env's specs. Usespack -E findto search everything. - Find by hash with a leading slash, e.g.
spack -E find -ld /t4p2j3g. - A complete ROOT shows its compiler and dependencies in
spack -E find -ld. A broken record (2rxgsmi) appears under "no compilers" and triggersMissing dependency not in databasewarnings. - Why the pin is needed: the concretizer reuses installed records. The incomplete ROOT record has no vdt/fftw/gsl/tbb edges, so they never reach
CMAKE_PREFIX_PATH. Starting from the cvmfsspack.lockand removing its roots does not help, because the concretizer reuses installed records, not the lock.
Coming soon.