Shannon

Detailed build and run instructions with Serial, OpenMP and CUDA node (10/13/2016)

Please refer to SAND report written by Jerry Watkins, which can be found here.

These are the build instructions for Shannon (11/18/2015)

Please email gahanse@sandia.gov with comments or questions.

Shannon is integrated by Appro

Current configuration installed 9/2014
22 compute nodes
Each node has two 8 core Sandy Bridge Xeon E5-2670 processors @ 2.6 GHz
Each node has 128 GB - DDR3 memory 1600 MHz
6 nodes have 2x NVIDIA K20x (Kepler), 12 have 4x K20x, 4 have 4x K40m, and 9 have 2x K80s
2688 732MHz cores
Mellanox QDR Infiniband

Set up your build environment

Here is the module load sequence used

	module purge
	module load openmpi/1.10.1/gnu/4.7.2/cuda/7.5.7
	export CUDA_LAUNCH_BLOCKING=1

Install the required Third Party Libraries (TPLs) for Trilinos

These install directions assume that a base install directory of /home/gahanse/gcc-4.7.2/ompi-1.10.1 is used

Note that "shannon" has prebuilt versions of zlib, hfd5, and netcdf. These are actually used, but we show build instructions on how one would build these libraries if they do not exist. We mark these as (optional).

Building ZLib (optional)

Version 1.2.8

  cd /home/gahanse/src
  wget http://zlib.net/zlib-1.2.8.tar.gz
  tar -xvf zlib-1.2.8.tar.gz
  cd zlib-1.2.8
  CC=mpicc CFLAGS="-O3" ./configure --prefix=/home/gahanse/gcc-4.7.2/ompi-1.10.1
  make install

Building HWLOC (optional)

Version 1.9

  ./configure --prefix=/home/gahanse/gcc-4.7.2/ompi-1.10.1 CC=mpicc CXX=mpicxx

Building HDF5 (optional)

Version 1.8.13

  cd /home/gahanse/src/
  wget http://www.hdfgroup.org/ftp/HDF5/current/src/hdf5-1.8.13.tar.bz2
  tar -xvf hdf5-1.8.13.tar.gz
  cd hdf5-1.8.13
  ./configure CC=mpicc FC=mpif90 CXX=mpicxx CXXFLAGS="-fPIC -O3" CFLAGS="-fPIC -O3" FCFLAGS="-fPIC -O3" \
  --enable-parallel --with-zlib=/home/gahanse/gcc-4.7.2/ompi-1.10.1 --prefix=/home/gahanse/gcc-4.7.2/ompi-1.10.1
  make -j 8
  make install

Building Netcdf (optional)

Version 4.3.3.1

  cd /home/gahanse/src/
  wget ftp://ftp.unidata.ucar.edu/pub/netcdf/netcdf-4.3.3.1.tar.gz
  tar -xvf netcdf-4.3.3.1.tar.gz
  cd netcdf-4.3.3.1

Edit the file include/netcdf.h. Make the following edits near line 228:

  #define NC_MAX_DIMS    65536    /* max dimensions per file */
  #define NC_MAX_ATTRS    8192    
  #define NC_MAX_VARS   524288    /* max variables per file */
  #define NC_MAX_NAME   256 
  #define NC_MAX_VAR_DIMS      8     /* max per variable dimensions */

Now, configure build:

  ./configure --prefix=/home/gahanse/gcc-4.7.2/ompi-1.10.1 CC=mpicc FC=mpifort CXX=mpicxx \
  CFLAGS="-I/home/gahanse/gcc-4.7.2/ompi-1.10.1/include -fpic -O3" \
  CXXFLAGS="-I/home/gahanse/gcc-4.7.2/ompi-1.10.1/include -fPIC -O3" \
  LDFLAGS=-L/home/gahanse/gcc-4.7.2/ompi-1.10.1/lib \
  FCFLAGS="-fPIC -O3" --disable-fsync --disable-cdmremote --disable-dap --disable-doxygen --enable-netcdf-4

  make -j 8 
  make install

Building boost

Version 1.59.0

  cd /home/gahanse/src/
  wget http://sourceforge.net/projects/boost/files/boost/1.59.0/boost_1_59_0.tar.gz/download
  tar -xvf boost_1_59_0.tar.gz
  cd boost_1_59_0

Build boost with the commands

  echo "using gcc :  4.7.2 : /home/projects/gcc/4.7.2/bin/g++ ;" > ~/user-config.jam
  echo "using mpi : /home/projects/x86-64-sandybridge-nvidia/openmpi/1.10.1/gnu/4.7.2/cuda/7.5.7/bin/mpicxx ;" >> ~/user-config.jam
  ./bootstrap.sh \
  --with-libraries=signals,regex,filesystem,system,mpi,serialization,thread,program_options,exception \
  --prefix=/home/gahanse/gcc-4.7.2/ompi-1.10.1

  ./b2 -j 8
  ./b2 -j 8 install

Building ParMetis

Version 4.0.3

  cd /home/gahanse/src/
  wget http://glaros.dtc.umn.edu/gkhome/fetch/sw/parmetis/parmetis-4.0.3.tar.gz
  tar -xvf parmetis-4.0.3.tar.gz
  cd parmetis-4.0.3

Edit metis.h to use an IDXTYPEWIDTH of 64

  cd metis/include
  vi metis.h
  #define IDXTYPEWIDTH 64
  #define REALTYPEWIDTH 64

Run CMake to build parmetis

  cd /home/gahanse/src/parmetis-4.0.3
  cd build
  export metis=/home/gahanse/src/parmetis-4.0.3/metis

  cmake \
  -DCMAKE_INSTALL_PREFIX=/home/gahanse/gcc-4.7.2/ompi-1.10.1 \
  -DMETIS_PATH=$metis \
  -DGKLIB_PATH=$metis/GKlib \
  -DCMAKE_C_COMPILER=mpicc \
  -DCMAKE_CXX_COMPILER=mpicxx \
  -DMPI_INCLUDE_PATH=/home/gahanse/gcc-4.7.2/ompi-1.10.1/include \
  -DCMAKE_C_FLAGS="-fPIC -O3" \
  -DCMAKE_CXX_FLAGS="-fPIC -O3" \
  ..

  make -j 8
  make install

  cp libmetis/libmetis.a /home/gahanse/gcc-4.7.2/ompi-1.10.1/lib
  cp $metis/include/metis.h /home/gahanse/gcc-4.7.2/ompi-1.10.1/include

Build Trilinos

Download Trilinos

  cd /home/gahanse/src
  git clone https://github.com/trilinos/Trilinos.git

Download SCOREC

  cd /home/gahanse/src/Trilinos
  git clone git@github.com:SCOREC/core.git SCOREC

Configure the Trilinos build

  mkdir build
  cd build
  ./do-configure

Example do-configure script executed above

  #!/bin/sh
  #

  export TRILINOS_HOME=/home/gahanse/Codes/Trilinos
  BUILD_DIR=`pwd`
  INSTALL_DIR=/home/gahanse/trilinos/MPI_NGCC
  BOOST_DIR=/home/gahanse/gcc-4.9.0/mpich-3.1.4
  MPI_BASE_DIR=/home/gahanse/gcc-4.9.0/mpich-3.1.4
  NETCDF=/home/gahanse/gcc-4.9.0/mpich-3.1.4
  HDFDIR=/home/gahanse/gcc-4.9.0/mpich-3.1.4
  PARMETISDIR=/home/gahanse/gcc-4.9.0/mpich-3.1.4
  MKL_PATH=/opt/intel/mkl/lib/intel64
  HWLOC_PATH=/home/gahanse/gcc-4.9.0/mpich-3.1.4
  LABLAS_LIBRARIES="-L$MKL_PATH/lib/intel64 -Wl,--start-group $MKL_PATH/mkl/lib/intel64/libmkl_intel_lp64.a $MKL_PATH/mkl/lib/intel64/
  
  cmake \
      -Wno-dev \
      -D Trilinos_CONFIGURE_OPTIONS_FILE:FILEPATH=$TRILINOS_HOME/sampleScripts/AlbanySettings.cmake \
      -D Trilinos_ENABLE_SCOREC:BOOL=ON \
      -D SCOREC_DISABLE_STRONG_WARNINGS:BOOL=ON \
      -D CMAKE_BUILD_TYPE:STRING=NONE \
      -D CMAKE_CXX_COMPILER:FILEPATH="/home/gahanse/Codes/Trilinos/build/nvcc_wrapper_gh" \
      -D CMAKE_C_COMPILER:FILEPATH="mpicc" \
      -D CMAKE_Fortran_COMPILER:FILEPATH="mpifort" \
      -D CMAKE_CXX_FLAGS:STRING="-DNDEBUG" \
      -D Trilinos_CXX11_FLAGS:STRING="-std=c++11 --expt-extended-lambda --expt-relaxed-constexpr -Wno-unused-local-typedefs -Wno-sign-
      -D CMAKE_C_FLAGS:STRING="-O3 -DNDEBUG" \
      -D CMAKE_Fortran_FLAGS:STRING="-O3 -DNDEBUG" \
      -D Trilinos_ENABLE_EXPLICIT_INSTANTIATION:BOOL=ON \
      -D Tpetra_INST_INT_LONG_LONG:BOOL=OFF \
      -D Tpetra_INST_INT_INT:BOOL=ON \
      -D Tpetra_INST_DOUBLE:BOOL=ON \
      -D Tpetra_INST_FLOAT:BOOL=OFF \
      -D Tpetra_INST_COMPLEX_FLOAT:BOOL=OFF \
      -D Tpetra_INST_COMPLEX_DOUBLE:BOOL=OFF \
      -D Tpetra_INST_INT_LONG:BOOL=OFF \
      -D Tpetra_INST_INT_UNSIGNED:BOOL=OFF \
      -D Zoltan_ENABLE_ULONG_IDS:BOOL=OFF \
      -D Teuchos_ENABLE_LONG_LONG_INT:BOOL=ON \
      -D Rythmos_ENABLE_DEBUG:BOOL=ON \
      -D Trilinos_ENABLE_Kokkos:BOOL=ON \
      -D Phalanx_KOKKOS_DEVICE_TYPE:STRING="CUDA" \
      -D Phalanx_INDEX_SIZE_TYPE:STRING="INT" \
      -D Phalanx_SHOW_DEPRECATED_WARNINGS:BOOL=OFF \
      -D Kokkos_ENABLE_Serial:BOOL=ON \
      -D Kokkos_ENABLE_OpenMP:BOOL=OFF \
      -D Kokkos_ENABLE_Pthread:BOOL=OFF \
      -D Kokkos_ENABLE_Cuda:BOOL=ON \
      -D TPL_ENABLE_CUDA:BOOL=ON \
      -D Kokkos_ENABLE_Cuda_UVM:BOOL=ON \
      -D Kokkos_ENABLE_Cuda_LAMBDA:BOOL=ON \
      -D TPL_ENABLE_CUSPARSE:BOOL=ON \
  \
      -D TPL_ENABLE_MPI:BOOL=ON \
      -D MPI_BASE_DIR:PATH=$MPI_BASE_DIR \
  \
      -D TPL_ENABLE_Pthread:BOOL=OFF \
  \
      -D TPL_ENABLE_Boost:BOOL=ON \
      -D TPL_ENABLE_BoostLib:BOOL=ON \
      -D TPL_ENABLE_BoostAlbLib:BOOL=ON \
      -D Boost_INCLUDE_DIRS:PATH=$BOOST_DIR/include \
      -D Boost_LIBRARY_DIRS:PATH=$BOOST_DIR/lib \
      -D BoostLib_INCLUDE_DIRS:PATH=$BOOST_DIR/include \
      -D BoostLib_LIBRARY_DIRS:PATH=$BOOST_DIR/lib \
      -D BoostAlbLib_INCLUDE_DIRS:PATH=$BOOST_DIR/include \
      -D BoostAlbLib_LIBRARY_DIRS:PATH=$BOOST_DIR/lib \
  \
      -D TPL_ENABLE_Netcdf:STRING=ON \
      -D Netcdf_INCLUDE_DIRS:PATH=$NETCDF/include \
      -D Netcdf_LIBRARY_DIRS:PATH=$NETCDF/lib \
  \
      -D TPL_ENABLE_HDF5:STRING=ON \
      -D HDF5_INCLUDE_DIRS:PATH=$HDFDIR/include \
      -D HDF5_LIBRARY_DIRS:PATH=$HDFDIR/lib \
  \
      -D TPL_ENABLE_Zlib:STRING=ON \
      -D Zlib_INCLUDE_DIRS:PATH=$HDFDIR/include \
      -D Zlib_LIBRARY_DIRS:PATH=$HDFDIR/lib \
  \
      -D TPL_ENABLE_BLAS:STRING=ON \
      -D TPL_ENABLE_LAPACK:STRING=ON \
      -D BLAS_LIBRARY_DIRS:FILEPATH=${MKL_PATH} \
      -D BLAS_LIBRARY_NAMES="mkl_intel_lp64;mkl_sequential;mkl_core;m" \
      -D LAPACK_LIBRARY_NAMES="" \
  \
      -D TPL_ENABLE_ParMETIS:STRING=ON \
      -D ParMETIS_INCLUDE_DIRS:PATH="${PARMETISDIR}/include" \
      -D ParMETIS_LIBRARY_DIRS:PATH="${PARMETISDIR}/lib" \
  \
      -D CMAKE_INSTALL_PREFIX:PATH=$INSTALL_DIR \
      -D Trilinos_EXTRA_LINK_FLAGS:STRING="-L${HDFDIR}/lib -lnetcdf -lhdf5_hl -lhdf5 -lz" \
  \
      -D Trilinos_ENABLE_Moertel:BOOL=OFF \
      -D Trilinos_ENABLE_TriKota:BOOL=OFF \
      -D Trilinos_ENABLE_SEACASSVDI:BOOL=OFF \
      -D Trilinos_ENABLE_SEACASPLT:BOOL=OFF \
      -D Trilinos_ENABLE_SEACASBlot:BOOL=OFF \
      -D Trilinos_ENABLE_STK:BOOL=ON \
      -D Trilinos_ENABLE_STKUtil:BOOL=ON \
      -D Trilinos_ENABLE_STKTopology:BOOL=ON \
      -D Trilinos_ENABLE_STKMesh:BOOL=ON \
      -D Trilinos_ENABLE_STKIO:BOOL=ON \
      -D Trilinos_ENABLE_STKTransfer:BOOL=ON \
      -D Trilinos_ENABLE_STKUnit_tests:BOOL=OFF \
      -D Trilinos_ENABLE_STKDoc_tests:BOOL=OFF \
      -D TPL_ENABLE_X11:BOOL=OFF \
      -D Trilinos_ENABLE_ThreadPool:BOOL=OFF \
      -D Trilinos_ENABLE_Teko:BOOL=OFF \
      -D Trilinos_ENABLE_MueLu:BOOL=ON \
      -D Trilinos_ENABLE_Amesos2:BOOL=ON \
      -D Amesos2_ENABLE_KLU2:BOOL=ON \
      -D BUILD_SHARED_LIBS:BOOL=OFF \
      -D Trilinos_ENABLE_OptiPack:BOOL=OFF \
      -D Trilinos_ENABLE_GlobiPack:BOOL=OFF \
  \
  $EXTRA_ARGS \
  ${TRILINOS_HOME}
  
  exit 0

Build Trilinos
```
  make -j 8
  make install
```

Build Albany

Clone the repo

  cd /home/gahanse/src
  git clone git@github.com:SNLComputation/Albany.git
  cd Albany

Configure the Albany build

  mkdir build
  cd build
  ./do-configure

CMake configure script ./do-configure:

  #!/bin/sh
  #
  
  export TRILINOS_HOME=/home/gahanse/trilinos/MPI_CUVM
  
  cmake \
    -DALBANY_TRILINOS_DIR:PATH=${TRILINOS_HOME} \
    -DENABLE_LCM:BOOL=ON \
    -DENABLE_LCM_SPECULATIVE:BOOL=OFF \
    -DENABLE_HYDRIDE:BOOL=OFF \
    -DENABLE_SCOREC:BOOL=ON \
    -DENABLE_SG_MP:BOOL=OFF \
    -DENABLE_FELIX:BOOL=ON \
    -DENABLE_AERAS:BOOL=ON \
    -DENABLE_QCAD:BOOL=OFF \
    -DENABLE_MOR:BOOL=OFF \
    -DENABLE_ATO:BOOL=OFF \
    -DENABLE_ASCR:BOOL=OFF \
    -DENABLE_CHECK_FPE:BOOL=OFF \
    -DENABLE_ALBANY_EPETRA_EXE:BOOL=ON \
    -DENABLE_KOKKOS_UNDER_DEVELOPMENT:BOOL=ON \
  \
  ..
  
  exit 0

Build Albany
```
  make -j 8
  make install
```

Run Albany tests

  salloc -n 4 -N 4 -p stella
  srun ctest
          (we need "srun" in purpose to get tests run on the compute node)

Run Albany executables salloc -N 1 -p stella srun -n 1 ./Alabny.exe input.xml (we need "srun" in purpose to get tests run on the compute node)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Shannon

Detailed build and run instructions with Serial, OpenMP and CUDA node (10/13/2016)

These are the build instructions for Shannon (11/18/2015)

Set up your build environment

Install the required Third Party Libraries (TPLs) for Trilinos

Building ZLib (optional)

Building HWLOC (optional)

Building HDF5 (optional)

Building Netcdf (optional)

Building boost

Building ParMetis

Build Trilinos

Build Albany

Docker Image

SPACK Package

ToDo List

Albany FAQs

Capabilities

Sub-Projects

Compiling

Nightly Tests

Code HowTo

User Group Meetings

PAALS Tutorials

Wiki HowTo

Visualization Utilities

Good Commits

Python and Albany

Clone this wiki locally