Building Albany using Spack
Spack is a package manager for supercomputers, Linux, and macOS. It makes installing scientific software easy. With Spack, you can build a package with multiple versions, configurations, platforms, and compilers, and all of these builds can coexist on the same machine. The following links provide some documentation on Spack:
- Spack web page https://spack.io
- Spack documentation is at http://spack.readthedocs.io/en/latest/
- Spack Google Group https://groups.google.com/forum/#!forum/spack
- Specific examples of
~/.spack/linux/compilers.yamlfiles are highlighted below - Some slides by Xylar Asay-Davis (@xylar) about the Albany-supported spack build
A working Albany build is supported in the E3SM fork of the spack repo: https://github.com/E3SM-Project/spack.
git clone git@github.com:E3SM-Project/spack.git
cd spack
git checkout develop
. share/spack/setup-env.sh
This approach requires uploading the ssh-keys for your machine to github (under Settings). An alternative to the first line above is: https://github.com/E3SM-Project/spack.git if you prefer cloning using HTTPS instead of SSH, which does not require uploading ssh-keys.
If you do not have access to the repo, please email Irina Tezaur (ikalash@sandia.gov) and/or Xylar Asay-Davis (xylar@lanl.gov).
Note that spack will try to download and install packages from the web. If your machine is behind a proxy, you will need to set proxies (export http_proxy=..., export https_proxy=...) prior to running . share/spack/setup-env.sh above.
spack --insecure install --dirty --keep-stage albany@develop
The --keep-stage option will ensure that the Albany build is kept after the installation of Albany is complete. This is required if you plan to run the Albany regression tests post-build. Note that, by default, the TPLs built by spack will be stages in the /tmp/$user/spack-stage directory. If you wish to change this, you can edit the following line in /path/to/spack/etc/spack/defaults/config.yaml. By specifying the @develop option, you are telling spack to build Albany with the develop branch of Trilinos, which is downloaded/install from its E3SM-Project repository.
build_stage:
- /tmp/$user/spack-stage
If you are interested in building Albany for the purpose of installing MPAS-Albany on top of Albany, please use the following command:
spack --insecure install --dirty --keep-stage albany@develop+mpas
If you are interested in building the PyAlbany capability within an MPAS-Albany-enabled build, the following command should be used:
spack --insecure install --dirty --keep-stage albany@develop+mpas+py
If you are interested in using the PDE-constrained optimization capabilities within Albany, please add the +optimization option, e.g.,
spack --insecure install --dirty --keep-stage albany@develop+mpas+py+optimization
Finally, if you are interested in building Albany with mesh-adaptation capabilities through the Omega_h library, please add the +omega_h option, e.g.,
spack --insecure install --dirty --keep-stage albany@develop+mpas+py+optimization+omegah
The +optimization option will turn in libraries required for optimization within Trilinos, namely SuperLU, ROL and FROSch.
Note that Albany requires one to use one of the following compilers:
Clang 4.0.0 or higher
GCC 5.3.0 or higher
Intel 17.0.0 or higher
NVCC 9.2.88 or higher
Instructions are provided below for how to check what compiler you have, how to build a new compiler using spack, and how to tell spack with what compiler to build Albany.
spack location -i albany
To cd to that location automatically, one can run the following command:
spack cd albany
First, cd to the Albany build directory, which be default will be /tmp/$USER/spack-stage/spack-stage-albany-develop-*/spack-build* (this can also be reached by running spack cd albany. From here, you can execute the command ctest -V to run the tests (or run the tests manually).
If you have built Albany with the --keep-stage option, you can find the source code by default in the following directory: /tmp/$USER/spack-stage/spack-stage-albany-develop-*/spack-src. If you wish to make a change to the source code and recompile the code, you simply make the chance, then cd to the build directory (by default /tmp/$USER/spack-stage/spack-stage-albany-develop-*/spack-build*) and type make -j 16. No module loading / environment specification should be necessary.
-
Building some of the Albany dependencies might result in certificate errors. To ignore certificates, use the command
spack --insecure install albany@developNote that ignoring ssh certificate issues can compromise your system's security. Note that
spack --insecure install albanyis equivalent tospack -k install albany. The--insecureoption is in general required on machines behind firewalls (with proxies). -
To build Albany using your current shell environment, instead of the Spack environment
spack install --dirty albany@develop -
To build Albany using 16 processors with verbose build output
spack install -j 16 -v albany@develop -
To run the Albany regression tests post-build, change into the build stage directory and run them with CTest
spack cd albany cd ../spack-build ctest
To run just the PyAlbany tests, a user can execute the following command:
ctest -R PyAlbany
- Uninstalling a spackage: spack unistall albany
The following is an example compilers.yaml file. It is typically found in the following location on Linux: ~/.spack/$arch/compilers.yaml.
compilers:
- compiler:
environment: {}
extra_rpaths: []
flags: {}
modules: []
operating_system: rhel6
paths:
cc: /sierra/sntools/SDK/compilers/clang/6.00-RHEL6/bin/clang
cxx: /sierra/sntools/SDK/compilers/clang/6.00-RHEL6/bin/clang++
f77: null
fc: null
spec: clang@6.0.0
target: x86_64
- compiler:
environment:
set:
LD_LIBRARY_PATH : /sierra/sntools/SDK/compilers/gcc/5.4.0-RHEL6/lib:/sierra/sntools/SDK/compilers/gcc/5.4.0-RHEL6/lib64
extra_rpaths: []
flags: {}
modules: []
operating_system: rhel6
paths:
cc: /sierra/sntools/SDK/compilers/gcc/5.4.0-RHEL6/bin/gcc
cxx: /sierra/sntools/SDK/compilers/gcc/5.4.0-RHEL6/bin/g++
f77: /sierra/sntools/SDK/compilers/gcc/5.4.0-RHEL6/bin/gfortran
fc: /sierra/sntools/SDK/compilers/gcc/5.4.0-RHEL6/bin/gfortran
spec: gcc@5.4.0
target: x86_64
To generate a default compilers.yaml file, please run spack compiler find.
To see what compilers you have on your machine, you can run spack compilers. Lets say this command returns the following:
==> Available compilers
-- gcc rhel7-x86_64 ---------------------------------------------
gcc@7.2.0 gcc@6.1.0 gcc@4.8.5 gcc@4.4.7
and we wish to build Albany for MPAS with gcc-7.2.0. This can be done as follows:
spack --insecure install --dirty --keep-stage albany@develop%gcc@7.2.0+mpas
Lets say the compiler you wish to use is not available on your system, e.g., it is gcc@8.4.0. One first needs to build the compiler using spack:
spack --insecure install --dirty --keep-stage gcc@8.4.0
Once that is done, one needs to make a note of the location in which the compiler was installed, and update one's compilers.yaml file to have an entry for the new compiler. Next, one can run spack compilers again and verify that gcc@8.4.0 shows up. Finally, to build Albany with gcc-8.4.0, one runs the following command:
spack --insecure install --dirty --keep-stage albany@develop%gcc@7.2.0+mpas
If you wish to use a different pre-built compiler for the spack build, you can add an entry to it in the ~/.spack/$arch/compilers.yaml file, following the example above.
By default, spack will stage its builds in the /tmp/$USER/spack-stage directory. To change this, edit the spack/etc/spack/default/config.yaml, line 62 to specify the new path, e.g.,
build_stage:
- /scratch/albany/nightlySpackBuild/spack-stage
In some cases, a user may not want spack to build every single TPL on which Albany depends; rather, one may want to have SPACK use pre-built TPLs. This can be accomplished by creating a ~/.spack/packages.yaml file that points to the pre-installed TPLs. This file will look as follows:
packages:
openmpi:
paths:
openmpi@1.10.7: /projects/sems/install/rhel7-x86_64/sems/v2/tpl/openmpi/1.10.7/gcc/10.1.0/base/7jgrwmo
buildable: False
cmake:
paths:
cmake@3.23.1: /projects/sems/install/rhel7-x86_64/sems/v2/utility/cmake/3.23.1/gcc/8.3.0/base/frlm6uo
buildable: False
boost:
paths:
boost@1.74.0: /projects/sems/install/rhel7-x86_64/sems/v2/tpl/boost/1.74.0/gcc/10.1.0/base/ua5ge5q
buildable: False
Instructions for building the MALI code on top of an Albany install with the "+mpas" option can be found here. The main steps are as follows:
- Build and install pnetcdf, netcdf and netcdf-fortran (these are installed automatically when building Albany using spack).
- Build and install Scorpio with PNetCDF enabled. This can be done using spack as follows:
spack --insecure install --dirty --keep-stage scorpio%gcc@10.1.0+pnetcdf
Alternatively, the code can be installed from its repository found here. Please use head of master. The following is an example cmake script to use:
rm -fr CMake*
INSTALLDIR="./install"
CC=mpicc
CXX=mpicxx
FC=mpif90
cmake \
-D CMAKE_INSTALL_PREFIX:PATH=$INSTALLDIR \
-D PnetCDF_PATH=${PARALLEL_NETCDF_ROOT} \
-D NetCDF_C_PATH=${NETCDF_C_ROOT} \
-D NetCDF_Fortran_PATH=${NETCDF_FORTRAN_ROOT} \
-D PIO_ENABLE_TIMING=OFF \
../scorpio
- Clone MALI as follows:
git clone git@github.com:MALI-Dev/E3SM.git
git checkout develop
git submodule update --init --recursive
cd components/mpas-albany-landice
- Build MALI by modifying and sourcing the following build script, which will build an executable called
landice_model.
#!/bin/bash
SCORPIO= path-to-scorpio-install
ALBANY= path-to-albany-install (from spack build)
NETCDF_FORTRAN_ROOT= path-to-netcdf-fortran (from spack build)
source ${ALBANY}/export_albany.in
echo $ALBANY_LINK_LIBS #make sure the albany libs are correct
echo $SCORPIO #make sure the scorpio path is correct
make -j 12 gfortran ALBANY=true USE_PIO2=true CORE=landice PIO=$SCORPIO NETCDF=$NETCDF_FORTRAN_ROOT MPAS_EXTERNAL_LIBS="$ALBANY_LINK_LIBS" DEBUG=false EXE_NAME=landice_model
- To test that the build worked OK, you can download a smoke test here, and follow instructions in the README file for how to run the test. Note that you will likely have to export the appropriate LD_LIBRARY_PATH to avoid errors when running.