This repo provides Spack packaging for building the Wire-Cell toolkit (wire-cell-toolkit/package.py) and a few additional Spack packaging to build WCT dependencies not already in Spack. It may be used to build development and production runtime environments.
Other Spack packaging of the Wire-Cell Toolkit exists. Of note is the one that exists as part of the FNALssi Spack suite used to build the software ecosystem found in FNAL’s CVMFS space.
The version name of a release of this package is based on the version name of the most recent release of wire-cell-toolkit that the release of wire-cell-spack builds. A forth digit starting at .0 is appended to the WCT version name and is incremented to indicate new releases of this package that do not update the latest version of WCT itself.
Versions relevant to this package are currently:
- Spack 0.22.0
- Wire-Cell toolkit 0.28.0
Generally, a release of wire-cell-spack can build past released versions of wire-cell-toolkit.
Versions strings used in the examples below may differ.
Here is one way to do it.
$ git clone -c feature.manyFiles=true https://github.com/spack/spack.git /srv/spack/local $ cd /srv/spack/local $ git checkout -b v0.22.0 v0.22.0
See Installation section of Spack docs for details and options.
$ bash --rcfile /srv/spack/local/share/spack/setup-env.sh
See Shell support section of the Spack docs for details and options.
Clone this repo somewhere and register it with Spack.
$ mkdir -p /srv/spack/repos $ cd /srv/spack/repos $ git clone https://github.com/WireCell/wire-cell-spack.git wct-spack $ spack repo add wct-spack
Simply run:
$ spack install wire-cell-toolkit
This will install the “core” WCT code and its dependencies.
Alternatively, you may wish to install a “variant” build that provides additional features requiring more dependencies. Discover what variants are available with:
$ spack info wire-cell-toolkit
For example one may install support for ROOT and HDF5:
$ spack install wire-cell-toolkit+root+hdf
One can build software with more than one compiler as long as it is known to Spack:
$ spack compiler find $ spack compiler list $ spack install wire-cell-toolkit%clang@14.0.6
Spack provides support for “environment modules” that manages the users shell environment. For example, so that PATH like variables are set.
$ spack load wire-cell-toolkit $ wire-cell --help
Note, developers should read this section but skip the example command and see section Developer guide below.
A “Spack view” provides an alternative to shell environment variables for using packages. This method requires far smaller shell environment mangling by providing a single area akin to /usr/local/.
$ spack view add -i myview wire-cell-toolkit $ ls myview/ bin etc include lib libexec man pkg sbin share
To use the software, you may now add the single entry
$(pwd)/myview/bin to $PATH and etc for other PATH-like variables.
## only needed if not using direnv $ myview=$(pwd)/myview $ export PATH=$myview/bin:$PATH $ export MANPATH=$myview/man:$myview/share/man:$MANPATH $ export LD_LIBRARY_PATH=$myview/lib:$LD_LIBRARY_PATH
This section describes one way to provide a developer environment.
After installing at least one instance of wire-cell-toolkit we make a spack view. This provides a single “tidy” directory similar to /usr/local with all the files for a set of packages.
$ cd /path/to/my/dev/area $ spack view -e wire-cell-toolkit add -i local wire-cell-toolkit
This builds the view into a directory called local/ with WCT as the “seed” package but excludes wire-cell-toolkit itself.
The above requires the requested instance of wire-cell-toolkit to have been installed by Spack. It is possible to create an equivalent view without building WCT with Spack. This can be needed in rare cases where Spack fails to build WCT but the developer wants Spack to provide dependencies so WCT source can be corrected so Spack build succeeds.
$ spack install wire-cell-dependencies $ spack view add -i local wire-cell-dependencies
This wire-cell-dependencies “bundle” package takes the same variants as wire-cell-toolkit.
Typically we developers want to form a strong binding between one of our development areas on disk and the shell environment that is required to build and run the software. This can be managed by hand but that is tedious and error prone. Use of direnv overcomes these problems and is well worth the one-time effort to install.
To configure direnv to use the Spack view, first set up direnv for your particular shell and then perform this one time setup:
$ cd /path/to/my/dev/area $ echo 'load_prefix local' > .envrc $ direnv allow
This last command will need repeating if/when you edit the .envrc file. With your shell set up to
use direnv it will automatically warn you when it detects a change.
The wire-cell-python package provides various useful, ancillary facilities but is not (yet)
officially a dependency of WCT. However, developers are strongly recommended to install it as it
is used by some of the tests provided by wire-cell-toolkit.
After the above direnv set up, check what Python we have:
$ which python $ python --version
This should show a modern version of Python provided under the Spack view directory local/. In general, do not use the system Python.
$ cd /path/to/my/dev/area $ git clone https://github.com/WireCell/wire-cell-python.git python $ cd python/ $ pip install -e .
Note, wire-cell-python packaging is still a work in progress and the above will only install core dependencies. Depending on the command line programs and Python modules to be used, the developer may need to install more Python dependencies via pip. This can be done using one or more of the requirements files. For example:
$ pip install -r requirements/base.txt
Future improvements in the wire-cell-python Python packaging will make this unnecessary.
Finally, the developer can get, build and run WCT. Start by getting a copy of the source:
$ cd /path/to/my/dev/area $ git clone https://github.com/WireCell/wire-cell-toolkit.git toolkit $ cd toolkit
We are now set to configure and build. The Spack view and direnv helps us here wcb will still need help to find some dependencies by giving it --with-* command line options:
$ export myview=/path/to/my/dev/area/local # just for brevity here $ ./wcb configure \ --prefix=$myview \ --with-jsonnet=$myview \ --with-jsonnet-libs=$myview/lib/libgojsonnet.so \ --boost-mt --boost-libs=$myview/lib --boost-include=$myview/include $ ./wcb install $ ./wcb --tests # optional, run many tests
Giving --prefix=$myview to wcb will cause WCT to be installed right into the Spack view directory. Since direnv has minimally set up our environment we (almost) ready for full usage:
$ wire-cell --version 0.18.0-5-g8513660
The .envrc file is ideal for capturing the few shell environment variable settings that are needed for developers:
$ cd /path/to/my/dev/area $ cat << EOF >> .envrc # Locate configuration files: path_add WIRECELL_PATH $PWD/toolkit/cfg path_add WIRECELL_PATH $HOME/opt/wire-cell-data # Locate BATS for manual running export BATS_LIB_PATH=$PWD/toolkit/test PATH_add $PWD/toolkit/test/bats/bin EOF
Creating the patterns described above can be automated using the provided ./scripts/wct-make-dev script. This script is particularly useful for developer that frequently create new WCT development areas.
$ ./scripts/wct-make-dev /path/to/work [<spec>] $ ls -a /path/to/work . .. .direnv .envrc local python toolkit
The script will guess the newest version of WCT available unless <spec> is given. It produces:
.direnv/direnv’s area holding a Python virtual env..envrca direnv config file.local/holding the Spack view.python/holding git clone ofwire-cell-pythontoolkit/holding git clone ofwire-cell-toolkit
At the end the script prints a few lines which can be copy-pasted to build and install the latter two. Eg:
cd /path/to/work && direnv allow cd python && pip install -e . && cd - bash -c "$WIRECELL_CONFIG" && bash -c "$WIRECELL_BUILD"
The developer is free to subsequently modify the generated .envrc file as desired.
When using a non-default compiler one must set CC and CXX before configuring wire-cell-toolkit build with wcb.
Spack allows multiple packages of the same name to be installed. Commands like spack load or spack view require a package to be identified uniquely. One way to do that is to specify the version and/or set of variants for the desired package. You can discover what is available with:
$ spack find -v wire-cell-toolkit -- linux-debian11-haswell / gcc@10.2.1 -------------------------- wire-cell-toolkit@0.18.0~cppjsonnet~hdf~root+tbb wire-cell-toolkit@0.18.0~cppjsonnet~hdf+root+tbb wire-cell-toolkit@0.20.0~cppjsonnet+hdf+root+tbb
In this example, adding +root or ~root is enough to make the name unique. You may test uniqueness, eg:
$ spack find -v wire-cell-toolkit~root ==> 1 installed package -- linux-debian11-haswell / gcc@10.2.1 -------------------------- wire-cell-toolkit@0.18.0~cppjsonnet~hdf~root+tbb
If specifying a version and variants are not enough to uniquely determine a package then a “slash-hash” may be used. To discover these you may run:
$ spack find -vl wire-cell-toolkit ==> 2 installed packages -- linux-debian11-haswell / gcc@10.2.1 -------------------------- 7rxgr7w wire-cell-toolkit@0.18.0~cppjsonnet~hdf~root+tbb 55c7krd wire-cell-toolkit@0.18.0~cppjsonnet~hdf+root+tbb urqwjcm wire-cell-toolkit@0.20.0~cppjsonnet+hdf+root+tbb
And again to test a match:
$ spack find -vl /55c7krd ==> 1 installed package -- linux-debian11-haswell / gcc@10.2.1 -------------------------- 55c7krd wire-cell-toolkit@0.18.0~cppjsonnet~hdf+root+tbb
To better keep multiple Spack instances you use separate, a Spack “scope” may be used. Some example ones are provided under scopes/ directory.
debian/- use Debian packages to provide packages otherwise built by Spack. For use, see example scripts/wcspack.
By default, Spack will use preferred versions, usually the newest. You may specify versions of WCT and dependencies at arbitrary detail by spelling out the Spack “spec”.
spack install wire-cell-toolkit@0.20.0+root ^root@6.28.04
See more in the Spack docs.
This package is updated infrequently to rely on new releases of Spack and add support for new releases of WCT and its dependencies at versions supported by that new Spack. This section describes how to update.
Either re-clone the Spack repository as in section Install Spack or fetch updates.
$ cd /srv/spack/local/ $ git tag|tail $ git checkout -b v0.21.2 v0.21.2
Update wire-cell-toolkit versions for recent Releases beyond what are currently captured.
$ spack info wire-cell-toolkit $ spack checksum wire-cell-toolkit 0.25.3 0.26.0 0.27.0 0.27.1
Copy-paste the new version(...) lines into the package.py.
Do a test build.
$ spack install wire-cell-toolkit+root+hdf+tbb
See wire-cell-spack-containers for ideas on how to make and use Docker and Singularity (now Apptainer) containers built via wire-cell-spack methods.