-
Notifications
You must be signed in to change notification settings - Fork 17
Developer Documentation
In this tutorial we will look at how to setup a developer workflow using some helper scripts
provided by spack-manager.
These helper scripts are not really necessary since in the end we are just using the
spack develop feature.
This uses a spack environment
which is a way to orgnaize the software you will be building/working with, similart to other environment concepts
(conda env, pyenv, etc).
The spack develop feature is simply a way of tagging specific software pieces as ones which you want to
be able to modify the source code and have the changes
propagate to its dependencies downstream through the spack install process.
For this tutorial we will take a moderately complicated scenario to illustrate the value of spack develop.
The scenario is you want to make changes in Trilinos, run the unit tests in nalu-wind and ensure the exawind-driver still builds.
The following outline can server as a quick-start guide since it provides the key commands for each step.
-
Setup spack-manager
git clone --recursive https://github.com/psakievich/spack-managerexport SPACK_MANAGER=$(pwd)/spack-managersource $SPACK_MANAGER/start.sh
-
Create an environment
spack manager create-env -d $SPACK_MANAGER/environments/demo
-
Activating an environment
spack env activate -d $SPACK_MANAGER/environments/demo
-
Add root specs
spack add exawind+hypre ^nalu-wind ^trilinos
-
Add develop specs
spack manager develop --repo-branch https://github.com/trilinos/trilinos develop trilinos@developspack manager develop nalu-wind@master
-
Concretize and install
spack concretizespack install
-
Code changes and rebuilding
spack cd -s trilinos # +code changesspack install
-
Running tests
spack cd -b nalu-windspack build-env nalu-wind ./unittestX --gtest_filter=SomeFilter*
-
Advanced topics
- Combinatoric builds
- Parallel builds
First setup spack-manager following the setup instructions
Next we will create a spack environment. We will do this using the spack manager create-env command.
All commands in spack have a ready help section that can be accessed with the -h or --help arguments.
For example spack manager -h provides the following output:
# spack manager -h
usage: spack manager [-h] spack-manager commands ...
commands that are specific to spack-manager
positional arguments:
spack-manager commands
create-env convenience script for setting up a spack environment
develop a more intuitieve interface for spack develop
find-machine get the current machine detected by spack-manager
optional arguments:
-h, --help show this help message and exitAny of the subcommands in spack manager are custom extension commands that have been written to work with the other data embedded in spack-manager.
The spack manager create-env command has several optional arguments, but for the sake of the tutorial we will only use one argument -d to specify a directory where we want to create our environment.
So to create our environment we can run:
spack manager create-env -d $SPACK_MANAGER/environments/demoand a directory will be created at this location that gets populated with all the machine specific data stored in spack-manager.
The $SPACK_MANAGER/environments directory is a holding location for environments, but it is not required for you to create environments there. Feel free to create them any where on the filesystem.
If we navigate to this directory you will see two files:
# ls -lh $SPACK_MANAGER/environments/demo
include.yaml spack.yamlinclude.yaml is a concatenation of all the machine specific information (packages.yaml, 'compilers.yaml, configs.yaml etc.).
This includes the available compilers, package preferences, and configuration settings.
Further details of the config files are outside the scope of this tutorial but can be found
in the spack configuration files documentation.
spack.yaml is what will define our environment and its contents will currently look like this:
spack:
include:
- include.yaml
concretization: together
view: false
specs:We will track changes to this file through out the rest of the tutorial. This file can be modified manually, or through spack commands. In the tutorial we will use the latter of these two options.
Next we will activate the environment. Activating an environment restricts spack's functionality to specifically what is defined in the environment.
To activate this environment run:
spack env activate -d $SPACK_MANAGER/environments/demoor use the shorthand function provided by spack:
spacktivate $SPACK_MANAGER/environments/demoAny directory with a valid spack.yaml can serve as the location for a spack environment, and you do not need to be in this directory when so long as the environment is active. If you want to switch to another environment you can deactivate the current one spack env deactivate (or despacktivate for the shorthand version), and then activate another one.
If you open a new shell and want to work in an environment you've already configured all you have to do is activate it.
You can also query the current environment status with spack env st.
You may notice that at the bottom of the spack.yaml file there is an entry for specs:. These are called root specs.
Root specs are what we tell spack we want in our environment.
Spack solves for all the dependencies to create a full environment at the concretization stage.
Root specs are either added as a list
spack add exawind
spack add nalu-wind
spack add trilinoswhich gives:
#spack.yaml
spack:
include:
- include.yaml
concretization: together
view: false
specs:
- exawind
- nalu-wind
- trilinosOr the dependencies can be chained in a single spec like this:
spack add exawind ^nalu-wind ^trilinoswhich gives:
#spack.yaml
spack:
include:
- include.yaml
concretization: together
view: false
specs:
- exawind ^nalu-wind ^trilinosThese two spack.yaml files are functionally equivalent. Details on specifying specs can be found here, and the scenarios for using one syntax over the other is discussed in the advanced concepts section of this tutorial.
One very helpful command for configuring the correct specs for each package is spack info [package].
For example
# spack info nalu-wind
CMakePackage: nalu-wind
Description:
Nalu-Wind: Wind energy focused variant of Nalu.
Homepage: https://nalu-wind.readthedocs.io
Maintainers: @jrood-nrel
Externally Detectable:
False
Tags:
ecp ecp-apps
Preferred version:
master [git] https://github.com/exawind/nalu-wind.git on branch master
Safe versions:
master [git] https://github.com/exawind/nalu-wind.git on branch master
Deprecated versions:
None
Variants:
Name [Default] Allowed values Description
=========================== ==================== =======================================
abs_tol [1e-15] Absolute tolerance for regression tests
boost [off] on, off Enable Boost integration
build_type [RelWithDebInfo] Debug, Release, CMake build type
RelWithDebInfo,
MinSizeRel
catalyst [off] on, off Compile with Catalyst support
cuda [off] on, off Build with CUDA
cuda_arch [none] none, 60, 75, 35, CUDA architecture
12, 72, 62, 53, 70,
50, 13, 20, 37, 52,
80, 32, 86, 10, 21,
11, 30, 61
fftw [off] on, off Compile with FFTW support
hypre [off] on, off Compile with Hypre support
ipo [off] on, off CMake interprocedural optimization
openfast [off] on, off Compile with OpenFAST support
pic [on] on, off Position independent code
rel_tol [1e-12] Relative tolerance for regression tests
tioga [off] on, off Compile with Tioga support
wind-utils [off] on, off Build wind-utils
Installation Phases:
cmake build install
Build Dependencies:
boost cmake cuda fftw hypre kokkos-nvcc-wrapper mpi nccmp netcdf-c openfast tioga trilinos trilinos-catalyst-ioss-adapter yaml-cpp
Link Dependencies:
boost cuda fftw hypre mpi nccmp netcdf-c openfast tioga trilinos trilinos-catalyst-ioss-adapter yaml-cpp
Run Dependencies:
None
Virtual Packages:
NoneTells us the variants and versions we can choose from when writing our spec.
Next we will create a develop spec. This is how we notify spack that we will be wanting to modify the source code for a given package.
In order for the source of a develop spec there must be an associated root spec. This is why we added 3 packages as root specs in the previous section. If we just wanted to develop the exawind package we wouldn't need to add the others.
Typically all of the packages that spack will install will be cached away inside spec and the contents of the builds are not readily available or modifiable. However, a develop spec is one where you can control the source code location and when you make changes to it spack will do an incremental build.
There are two recommended ways for specifying a develop spec and we will illustrate them both with our two develop packages for this tutorial environment.
The first is to let spack use the package's default repo and branch.
spack manager develop nalu-wind@masterYou will notice that to add a spack develop the spec also has a version.
This is a minimal concrete spec because it has a package name (nalu-wind) and version (master).
It is recommended that you always use the minimal concrete spec for the develop command. If you forget the version you will get an error indicating that you need to specify a concrete spec.
When you run the command above spack will clone the repository to the same directory where the spack.yaml resides and update the spack.yaml file. We will show the updated spack.yaml after showing the second method.
The second method allows you to pick a specific branch and/or fork for your clone operation.
spack manager develop --branch-repo https://github.com/trilinos/trilinos someFeatureBranch trilinos@developIn this case we give the url for fork/repo and the branch in addition to the concrete spec. The mechanics from this point on are virtually identical to method 1.
Now if you look at the spack.yaml you will see a new section and two new entries indicating the develop specs.
#spack.yaml
spack:
include:
- include.yaml
concretization: together
view: false
specs:
- exawind
- nalu-wind
- trilinos
develop:
nalu-wind:
spec: nalu-wind@master
trilinos:
spec: trilinos@developNow that the develop specs are setup you can make any changes you want to the source. You can create new branches, add remotes, etc. From this point on this is just a location where spack is going to look for changes and rebuild the source from.
Now that we've added a develop spec we can concretize.
This is how spack finalizes the environment withe the requirements and constraints communicated through the spack.yaml file.
To concretize run:
spack concretize
Now you can install/build:
spack install
Once the build completes we can look inside the source directory where we will see a series of files
-
spack-build-out.txt: the build output for the development package -
spack-build-env.txt: the build environment used (sourcing this file will allow you to enter the exact environment used to build the software) -
spack-build-[hash].txt: the build directory where the object files, and executables can be found.
If you'd like to change something about the packages in your environment,
say you want to switch from a Release build to a Debug build or add a
new variant to a package, you can modify the spec's in the spack.yaml file.
However, if you do this you must also run
spack concretize -f
This is how you signal to spack that you made a change to an existing spec and need
spack to recreate the dependecy tree.
This will also result in new hashes for affected packages since the specs have been changed.
To make changes all you have to do is edit the source code that has been cloned into $SPACK_MANAGER/environments/demo.
You may also notice that since this is an active environment the $SPACK_ENV variable also points to this location.
Spack also has some convenient ways of getting around. spack cd -e will take you to the active environment.
spack cd -s trilinos will take you to the stage/source directory for Trilinos, and spack cd -b trilinos will take you
to the Trilinos build directory.
You can use an IDE or any other tool you'd like to make your changes.
To run regression or unit tests you need to do two things:
- Get to the build directory
- Call your testing commands in the appropriate environment
For item 1) spack creates a build directory with the format spack-build-[hash] inside
your source code directory.
As noted above, you can get to this directory from anywhere by running the command
spack cd -b [package]
This command is telling spack to go to the build directory of the package you'd like.
It is extra helpful if you've re-concretized your environment and have build directories
from multiple hashes in your environment.
For item 2) it is important to remember that spack is building with a different environment
from the one you used to call the spack install command.
The build shell has a unique environment that should be accessed to run tests.
If you wish to run tests you will need to make sure you have that environment available to
your current shell.
There are two approaches for doing this.
The first is to use a the spack command spack build-env.
This command will let you execute any command in the same environment that a package
was build with.
For example, if you wish to run the regression tests for nalu-wind you can run the following.
spack cd -b nalu-wind
spack build-env nalu-wind ctest -R SomeTest
This has the advantage of keeping your current shell unmodified, but there is some overhead for the command you'd like to execute.
The other option is to source the environment into your current shell.
When spack does a build it creates a spack-build-env.txt file that captures the build
environment.
This is located at the same directory level as the spack-build-[hash] directory.
If you source this file you can run any of the build or test commands and your environment
will match the build environment.
You can also get the output of this file by running the spack build-env command without any arguments.
More information on that command can be found via spack build-env -h.
The main disadvantage of sourcing the build envronment directly into your working shell
is that unexpected changes might occur (python or git version may change).
To avoid this you can also just launch a new sub-shell as follows
spack cd -b nalu-wind
bash -rcfile ../spack-build-env.txtYou can now just run the tests, or do an incremental build of this specific package, and when you are done just type exit to get back to your original shell.
./unittestX
# some code changes
make -j 16
./unittestX
# all done
exitTo do incremental builds you can re-run spack install, or if you've already sourced spack-build-env.txt then you can navigate to the build directory and re-run ninja or make like it was a manual build outside of spack.
Please note that this later option will only rebuild that one package while the former will also rebuild all of its dependencies. So for the use case of modifying Trilinos and then running tests in Nalu-Wind it would make more sense to just run spack install.
This section is for a few advanced topics that are helpful for developing.
Spack gives the ability to build multiple configurations of the same software in the same environment. A prime example of this would be a Cuda and non-Cuda build in the same environment. Setting up an environment like this will allow you to build and test both combinations from the same source code. Let's look at what a spack.yaml would look like for this case when we are developing Exawind, AMR-Wind and Nalu-Wind at the same time.
#spack.yaml
spack:
include:
- include.yaml
concretization: separately # important change
view: false
specs:
- exawind+cuda+amr_wind_gpu~nalu_wind cuda_arch=70 ^nalu-wind ^amr-wind
- exawind~cuda ^nalu-wind ^amr-wind
develop:
exawind:
spec: exawind@master
nalu-wind:
spec: nalu-wind@master
amr-wind:
spec: amr-wind@mainThis will do 2 develop builds of AMR-Wind and Exawind and 1 develop build of Nalu-Wind.
There will be a Cuda version of AMR-Wind and Exawind and a host only version of all 3 packages.
Note that because the concretization flag was set to separately each line in the root spec section will be concretized by itself.
This is a key difference between the styles listed in Add root specs.
Note that if you wanted to run the tests for a package that had a Cuda and non-Cuda build in the same environment you'd need to add that differentiator to the specs when using spack commands i.e (spack cd -b amr-wind+cuda && spack build-env amr-wind+cuda ctest, etc).
Spack also allows for matrix builds if there are a lot of slightly different combinations you'd like to build.
Combinatoric builds are useful, but unfortunately the resources required to build also increase. Luckily spack provides a way to scale out the build using additional parallel processing i.e. beyond make -j N. Install level parallelism can be achieved by launching multiple spack instances. They will then build the DAG in parallel. Further documentation can be found here.