Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
LCM with Kokkos CUDA support on Shannon
Building Albany/LCM on Shannon with CUDA
The purpose of this guide is to try and explain and make sense of the build process for Albany/LCM for CUDA on the Shannon GPU cluster. The process is quite complicated and many modules are not updated to compile for GPUs.
The first thing we will do is set up our environment. All of the following commands can easily be added to a shell script or just executed before compiling. In my scripts I sourced an sh file called "build_common.sh" that I used for environment settings.
First, we need to load the required modules:
module purge module add cmake/22.214.171.124 module add openmpi/1.8.4/gnu/4.7.2/cuda/7.0.28 module add libmpc/1.0.1 module add nvcc-wrapper/gnu
Trilinos will need Cmake version 2.8.11 in order to compile. Additionally, we want to use OpenMPI here with GCC version 4.7.2. While Shannon has newer versions of GCC including 5.10, the version of CUDA installed on the cluster does not necessarily support those versions of GCC. MPC is a library required by Trilinos that is available on the cluster. Finally,
nvcc-wrapper is a utility that wraps compiler calls to forward properly to either GCC or NVCC.
Next we set our install prefix variable:
mkdir -p install PREFIX=$PWD/install
Next we need to export the compiler environment variables that are needed for building the various dependencies of Albany/LCM. First the compiler search paths:
export CPATH=$CPATH:$PREFIX/include export LIBRARY_PATH=$LIBRARY_PATH:$PREFIX/lib
Now we need to set which compilers should be used. We want to compile everything with mpicc except for the CUDA code. To do this we will set the
NVCC_WRAPPER_DEFAULT_COMPILER environment variable for the
nvcc-wrapper script. This determines what compiler will be used for C and C++ files rather than Cuda files.
export CC=mpicc export CXX=mpicxx export FC=mpif90 export NVCC_WRAPPER_DEFAULT_COMPILER=mpicc
The Trilinos packages we need for Albany require a few dependencies. This document will only show how to obtain the dependencies specifically required to build LCM.
Note: On Shannon you may not be able to use wget or git clone; if this is the case then you should download these packages and transfer them over using
Boost is used throughout Trilinos. It's mostly a header only library so very little will need to be compiled. I've tested this with the latest version of boost at the time of writing, 1.58.
wget http://downloads.sourceforge.net/project/boost/boost/1.58.0/boost_1_58_0.tar.bz2 tar -xjf boost_1_58_0.tar.bz2
Once you have it downloaded,
cd into the directory and run
bootstrap.sh with the following arguments:
./boostrap.sh --with-libraries=system,program_options --prefix=$PREFIX
This configures boost to only build the specified libraries. You can omit the argument if you want but you will then build the entirety of boost which takes a long time.
--prefix sets the installation directory. So if you set
install/ earlier then when you install boost it will place the header files in
install/include/, the library files in
install/lib/, and so on.
Once configuring is done we can build boost and install it:
./b2 -j 8 ./b2 install
This will run boost's
bjam build system on 8 processors and install it to
Next we will obtain and build zLib:
wget http://zlib.net/zlib-1.2.8.tar.gz tar -xzf zlib-1.2.8.tar.gz pushd zlib-1.2.8 ./configure --prefix=$PREFIX make -j8 make install popd
HDF5 is built similarly:
wget http://www.hdfgroup.org/ftp/HDF5/current/src/hdf5-1.8.15-patch1.tar.bz2 tar -xjf hdf5-1.8.15-patch1.tar.bz2 pushd hdf5-1.8.15-patch1 ./configure --prefix=$PREFIX --enable-parallel make -j8 make install popd
You know the drill...
wget ftp://ftp.unidata.ucar.edu/pub/netcdf/netcdf-126.96.36.199.tar.gz tar -xzf netcdf-188.8.131.52.tar.gz pushd netcdf-184.108.40.206 ./configure --prefix=$PREFIX make -j8 make install popd
Shannon has the Intel MKL package installed, so you can use that for BLAS and LAPACK if you want. I'll show you how to build the default LAPACK implementation from source though. It won't be as fast or as optimized as MKL however.
wget http://www.netlib.org/lapack/lapack-3.5.0.tgz tar -xzf lapack-3.5.0.tgz pushd lapack-2.5.0 mkdir -p build cd build cmake -DCMAKE_INSTALL_PREFIX=$PREFIX .. make -j8 install popd
We are now finally ready to build Trilinos.
First, clone the git repository for the latest version of Trilinos. In my script I also run
git pull in order to make sure the current version is the latest.
git clone https://software.sandia.gov/trilinos/repositories/publicTrilinos pushd publicTrilinos mkdir -p build cd build
Now we run cmake on Trilinos similarly to how we did it for LAPACK. Trilinos has a lot of CMake cache variables so it is recommended to run this with a shell script. I will be breaking up the cmake command into several parts. Each line of the command needs the continuation symbol
\ at the end, so don't forget that.
Note: I'm separating out the command into multiple parts for readability. The entire cmake command should be run at once.
cmake \ -DTPL_ENABLE_MPI=ON \ -DTPL_ENABLE_CUDA=ON \ -DTPL_ENABLE_Thrust=On \ \
This part of the command enables specific third party libraries, MPI, CUDA, and Thrust that we want to build with. Thrust is necessary for Kokkos if CUDA is enabled. On Shannon, Thrust comes with CUDA so you don't need to install it.
-DTrilinos_ENABLE_ALL_PACKAGES=OFF \ -DKokkos_ENABLE_Cuda_UVM=ON \ -DTrilinos_ENABLE_EXPLICIT_INSTANTIATION=ON \ -DTrilinos_ENABLE_DEVELOPMENT_MODE=OFF \ -DTrilinos_ASSERT_MISSING_PACKAGES=OFF \ \
First, we should not enable all packages. This will dramatically reduce compile times. Furthermore, some Trilinos packages conflict or do not work at all so this will probably also break your build if you do not set it.
We also need to enable UVM for CUDA. Kokkos and some Trilinos libraries like Phalanx use UVM to assign data in Kokkos views.
In order to reduce memory use while compiling, avoiding annoying other people on the cluster, we enable explicit template instantiation.
Disabling Trilinos development mode will disable pedantic compiling and (I think) warnings as errors. Trilinos has a lot of warnings when building so I would not recommend enabling this flag since it may break your build.
Finally, it's necessary to disable asserting on missing packages. Trilinos contains references to packages that don't actually exist, so enabling this flag would break your build.
-DBoost_INCLUDE_DIRS=$PREFIX/include \ -DBoost_LIBRARY_DIRS=$PREFIX/lib \ -DBoostLib_INCLUDE_DIRS=$PREFIX/include \ -DBoostLib_LIBRARY_DIRS=$PREFIX/lib \ \
These flags just set the location of boost. Shannon already has boost installed but it may be the wrong version. I recommend installing the version of boost I specify in this guide.
-DTrilinos_ENABLE_DEBUG=OFF \ -DTrilinos_ENABLE_Piro=ON \ -DTrilinos_ENABLE_Phalanx=ON \ -DTrilinos_ENABLE_ThyraTpetraAdapters=On \ -DTrilinos_ENABLE_STKIO=On \ -DTrilinos_ENABLE_STKMesh=On \ -DTrilinos_ENABLE_STKClassic=Off \ -DTrilinos_ENABLE_STKDoc_tests=Off \ -DTrilinos_ENABLE_Teko=Off \ \
First, we want to disable debug mode. If you want to enable it to get asserts you can do that and then also set -DCMAKE_BUILD_TYPE=Debug.
Next we enable and disable specific libraries that Albany/LCM require. Piro, Phalanx, and ThyraTpetraAdapters are all required. The Albany makefile claims STKIO and STKMesh are not required but they actually are. Do not include all of STK. This will break your build. Additionally, explicitly disable STKClassic and STKDoc_tests as they will break your build. Finally, for some reason Teko doesn't compile on Shannon. Disable it.
-DCMAKE_INSTALL_PREFIX=$PREFIX \ .. #Done with CMake command make -j8 install popd
We finally set our install location and our source location. I like having my install location in one place; however, this can cause issues if you update Trilinos. If you are pulling from Trilinos make sure you delete the prefix folder. The reason for this is Trilinos copies all its headers in a flat directory. So Trilinos will use the headers in the prefix location when building instead of the headers in the source directory. If you update Trilinos it will use outdated headers and fail to compile unless you delete the install directory.
Finally, we get to Albany.
Albany also uses CMake in a similar manner to Trilinos. First, we clone the repository:
git clone https://github.com/gahansen/Albany.git pushd Albany mkdir -p build cd build
Now for the CMake command.
Note: I'm separating out the command into multiple parts again.
cmake \ -DTrilinos_DIR=$PREFIX \ -DALBANY_TRILINOS_DIR=$PREFIX \ -DCMAKE_CXX_FLAGS="-std=c++11" \ -DENABLE_DEBUGGING=OFF \ \
This sets the trilinos directory for Albany. Also it sets the CXX flags. You can add optimization flags and so on if you would like. However, you need to at least set something because if not Albany copies the Trilinos flags. This has some weird consequences such as NVCC complaining about duplicate flags and failing to compile your code. We also disable debugging but you can turn it on.
-DENABLE_LCM=ON \ -DENABLE_BGL=OFF \ -DENABLE_QCAD=OFF \ -DENABLE_LCM_SPECULATIVE=OFF \ -DENABLE_LAME=OFF \ -DENABLE_HYDRIDE=OFF \ -DENABLE_SCOREC=OFF \ -DENABLE_AMP=OFF \ -DENABLE_SEE=OFF \ -DENABLE_FELIX=OFF \ -DENABLE_MOR=OFF \ -DENABLE_ALBANY_CI=OFF \ -DENABLE_ASCR=OFF \ -DENABLE_SG_MP=OFF \ \
Next we enable LCM and disable other parts of Albany to speed up compile time. You can enable these if you want.
-ENABLE_DEMO_PDES=OFF \ -DENABLE_64BIT_INT=OFF \ -DENABLE_CHECK_FPE=ON \ .. #End of CMake command make -j8 popd
Make sure you disable Demo PDES. There are some issues compiling those with CUDA on Shannon. I'm not exactly sure if 64 bit ints are used at all, so it may not be necessary to disable this. Finally, if you want floating point exception checking, set ENABLE_CHECK_FPE.
I hope this guide was helpful in getting LCM copmpiled with Kokkos support on Shannon.