Rhel6 using Gcc

This set of build instructions is for the Albany suite on Rhel6 linux.

The compilers are gcc 5.2.0, OpenMPI 1.8.5, and RHEL 6 linux.

Modules on CEE:

	module load sierra-devel/gcc-5.2.0-openmpi-1.8.8
	module load sierra-mkl/16.0-2016.2.181

Build supporting packages

Trilinos requires a set of "third party libraries" (TPLs) to support Albany. The actual libraries needed depends on the final Albany configuration desired. These instructions build a superset, all the libraries needed to support any Albany configuration.

Notes: In this example

• The TPLs are installed in /projects/albany

Building ZLib

• Version 1.2.8

    CC=mpicc CFLAGS=-O3 ./configure --64 --archs="-arch x86_64" --prefix=/projects/albany
    make -j 4
    make install
  

Building HDF5

• Version 1.8.17

    ./configure CC=mpicc FC=mpifort CXX=mpicxx CXXFLAGS="-O3 -fPIC -march=native" \
     CFLAGS="-O3 -fPIC -march=native" \
     FCFLAGS="-O3 -fPIC -march=native" --enable-parallel \
     --with-zlib=/projects/albany --prefix=/projects/albany
    make -j 4
    make install
  

Building parallel-netcdf

• Version 1.7.0

• Configure and build:

    ./configure --prefix=/projects/albany MPICC=mpicc MPIF77=mpifort MPIF90=mpifort MPICXX=mpicxx \
    CXXFLAGS="-fPIC -O3 -march=native" CFLAGS="-fPIC -O3 -march=native" \
    F77FLAGS="-fPIC -O3 -march=native" F90FLAGS="-fPIC -O3 -march=native"
    make -j 8 
    make install
  

Building Netcdf

• Version 4.4.0

• Edit the file include/netcdf.h. Make the following edits near line 265:

    #define NC_MAX_DIMS    65536    /* max dimensions per file */
    #define NC_MAX_ATTRS    8192
    #define NC_MAX_VARS   524288    /* max variables per file */
    #define NC_MAX_NAME   256
    #define NC_MAX_VAR_DIMS      8     /* max per variable dimensions */
  

• Now, configure using CMake and build:

    ./configure CC=mpicc FC=mpifort CXX=mpicxx \
        CXXFLAGS="-fPIC -I/projects/albany/include -O3 -march=native" \
        CFLAGS="-fPIC -I/projects/albany/include -O3 -march=native" \
        LDFLAGS="-fPIC -L/projects/albany/lib -O3 -march=native" \
        FCFLAGS="-fPIC -I/projects/albany/include -O3 -march=native" \
        --prefix=/projects/albany --disable-doxygen --enable-netcdf4 --enable-pnetcdf
    make -j 8
    make install
  

Building boost

• Version 1.55.0

• Build boost with the commands

    echo "using mpi : /sierra/sntools/SDK/mpi/openmpi/1.8.5-gcc-5.2.0-RHEL6/bin/mpicxx ;" >> ./tools/build/v2/user-config.jam
    echo "using gcc : /sierra/sntools/SDK/compilers/gcc/5.2.0-RHEL6/bin/g++ ;" >> ./tools/build/v2/user-config.jam
    ./bootstrap.sh --with-libraries=signals,regex,filesystem,system,mpi,serialization,thread,program_options,exception --prefix=/projects/albany
    ./b2 -j 4
    ./b2 -j 4 install
  

Building ParMetis

• Version 4.0.3

    wget http://glaros.dtc.umn.edu/gkhome/fetch/sw/parmetis/parmetis-4.0.3.tar.gz
    tar -xvf parmetis-4.0.3.tar.gz
  

• Edit metis.h to use an IDXTYPEWIDTH of 64

    cd parmetis-4.0.3/metis/include
    vi metis.h
    #define IDXTYPEWIDTH 64
    #define REALTYPEWIDTH 64
  

• Run CMake to build parmetis

    cd build
    export metis=/projects/albany/src/parmetis-4.0.3/metis/  
    cmake \
    -DCMAKE_INSTALL_PREFIX=/projects/albany/parmetis-4.0.3 \
    -DMETIS_PATH=$metis \
    -DGKLIB_PATH=$metis/GKlib \
    -DCMAKE_C_COMPILER=mpicc \
    -DCMAKE_CXX_COMPILER=mpicxx \
    -DCMAKE_C_FLAGS=-O3  \
    -DCMAKE_CXX_FLAGS=-O3  \
    ..
    make -j 4
    make install
  
    cp libmetis/libmetis.a /projects/albany/parmetis-4.0.3/lib
    cp $metis/include/metis.h /projects/albany/parmetis-4.0.3/include
  

SuperLU (optional)

• Version 4.3

• Build paths

    mkdir /projects/albany/SuperLU_4.3
    mkdir /projects/albany/SuperLU_4.3/include
    mkdir /projects/albany/SuperLU_4.3/lib
  

• Edit make.inc for your machine and environment

    PLAT = _linux
    SuperLUroot = /projects/albany/SuperLU_4.3
    SUPERLULIB      = $(SuperLUroot)/lib/libsuperlu_4.3.a
    TMGLIB          = libtmglib.a
    BLASDEF     = -DUSE_VENDOR_BLAS
    BLASLIB     = -L/sierra/sntools/SDK/compilers/intel/composer_xe_2016.3.210/mkl/lib/intel64 -lmkl_intel_lp64 -lmkl_sequential -lmkl_core
    LIBS        = $(SUPERLULIB) $(BLASLIB)
    ARCH         = ar
    ARCHFLAGS    = cr
    RANLIB       = ranlib
    CC           = gcc
    CFLAGS       = -O3
    NOOPTS       =
    FORTRAN      = gfortran
    FFLAGS       = -O3
    LOADER       = $(CC)
    LOADOPTS     =
    CDEFS        = -DAdd_
    MATLAB       = /usr/sww/matlab
  

• Build SuperLU

    make -j 4
    cd SRC
    cp *.h /projects/albany/SuperLU_4.3/include		
  

Build HWLOC

• HWLOC is an optional package that can be used for thread pinning by Kokkos. If you are not building Kokkos, or if you don't care about thread pinning, please skip.

• Version 1.8

    ./configure CC=mpicc CXX=mpicxx --prefix=/projects/albany
    make -j 4
    make install
  

Build Trilinos

Notes: In this example

• The Trilinos source is installed in /home/gahanse/Codes/Trilinos, using a command like

    cd /home/gahanse/Codes
    git clone git@github.com:trilinos/Trilinos.git
  

• The SCOREC tools are installed as a subdirectory of Trilinos, and are build automatically by the Trilinos build process. The commands to do this are

    cd /home/gahanse/Codes/Trilinos
    git clone https://github.com/SCOREC/core.git SCOREC
  

• do-configure script

    #!/bin/sh
    #
    export TRILINOS_HOME=/projects/albany/src/Trilinos
    BUILD_DIR=`pwd`
    INSTALL_DIR=/projects/albany/trilinos/MPI_REL
    BOOST_DIR=/projects/albany
    LIB_DIR=/projects/albany
    MPI_BASE_DIR=/sierra/sntools/SDK/mpi/openmpi/1.8.8-gcc-5.2.0-RHEL6
    NETCDF=/projects/albany
    HDFDIR=/projects/albany
    PARMETISDIR=/projects/albany
    MKL_PATH=/sierra/sntools/SDK/compilers/intel/composer_xe_2016.3.210
    HWLOC_PATH=/projects/albany
    LABLAS_LIBRARIES="-L$MKL_PATH/lib/intel64 -Wl,--start-group $MKL_PATH/mkl/lib/intel64/libmkl_intel_lp64.a $MKL_PATH/mkl/lib/intel64/libmkl_core.a $MKL_PATH/mkl/lib/intel64/libmkl_sequential.a -Wl,--end-group"
    
    cmake \
        -Wno-dev \
        -D Trilinos_CONFIGURE_OPTIONS_FILE:FILEPATH=$TRILINOS_HOME/sampleScripts/AlbanySettings.cmake \
        -D Trilinos_ENABLE_SCOREC:BOOL=ON \
        -D SCOREC_DISABLE_STRONG_WARNINGS:BOOL=ON \
        -D CMAKE_BUILD_TYPE:STRING=NONE \
        -D CMAKE_CXX_FLAGS:STRING="-O3 -march=native -DNDEBUG" \
        -D CMAKE_C_FLAGS:STRING="-O3 -march=native -DNDEBUG" \
        -D CMAKE_Fortran_FLAGS:STRING="-O3 -march=native -DNDEBUG" \
        -D Rythmos_ENABLE_DEBUG:BOOL=ON \
        -D KokkosClassic_DefaultNode:STRING="Kokkos::Compat::KokkosSerialWrapperNode" \
        -D Trilinos_ENABLE_EXPLICIT_INSTANTIATION:BOOL=ON \
        -D Tpetra_INST_INT_LONG_LONG:BOOL=ON \
        -D Tpetra_INST_INT_INT:BOOL=ON \
        -D Tpetra_INST_DOUBLE:BOOL=ON \
        -D Tpetra_INST_FLOAT:BOOL=OFF \
        -D Tpetra_INST_COMPLEX_FLOAT:BOOL=OFF \
        -D Tpetra_INST_COMPLEX_DOUBLE:BOOL=OFF \
        -D Tpetra_INST_INT_LONG:BOOL=OFF \
        -D Tpetra_INST_INT_UNSIGNED:BOOL=OFF \
        -D Trilinos_ENABLE_Kokkos:BOOL=ON \
        -D Trilinos_ENABLE_KokkosCore:BOOL=ON \
        -D Phalanx_KOKKOS_DEVICE_TYPE:STRING="SERIAL" \
        -D Phalanx_INDEX_SIZE_TYPE:STRING="INT" \
        -D Phalanx_SHOW_DEPRECATED_WARNINGS:BOOL=OFF \
        -D Kokkos_ENABLE_Serial:BOOL=ON \
        -D Kokkos_ENABLE_OpenMP:BOOL=OFF \
        -D Kokkos_ENABLE_Pthread:BOOL=OFF \
    \
        -D TPL_ENABLE_MPI:BOOL=ON \
        -D MPI_BASE_DIR:PATH=$MPI_BASE_DIR \
    \
    	-D TPL_ENABLE_Pthread:BOOL=OFF \
    \
        -D TPL_ENABLE_Boost:BOOL=ON \
        -D TPL_ENABLE_BoostLib:BOOL=ON \
        -D TPL_ENABLE_BoostAlbLib:BOOL=ON \
    	-D Boost_INCLUDE_DIRS:PATH=$BOOST_DIR/include \
    	-D Boost_LIBRARY_DIRS:PATH=$BOOST_DIR/lib \
    	-D BoostLib_INCLUDE_DIRS:PATH=$BOOST_DIR/include \
    	-D BoostLib_LIBRARY_DIRS:PATH=$BOOST_DIR/lib \
    	-D BoostAlbLib_INCLUDE_DIRS:PATH=$BOOST_DIR/include \
    	-D BoostAlbLib_LIBRARY_DIRS:PATH=$BOOST_DIR/lib \
    \
        -D TPL_ENABLE_Netcdf:STRING=ON \
    	-D Netcdf_INCLUDE_DIRS:PATH=$NETCDF/include \
    	-D Netcdf_LIBRARY_DIRS:PATH=$NETCDF/lib \
    \
        -D TPL_ENABLE_HDF5:STRING=ON \
        -D HDF5_INCLUDE_DIRS:PATH=$HDFDIR/include \
        -D HDF5_LIBRARY_DIRS:PATH=$HDFDIR/lib \
    \
        -D TPL_ENABLE_Zlib:STRING=ON \
        -D Zlib_INCLUDE_DIRS:PATH=$HDFDIR/include \
        -D Zlib_LIBRARY_DIRS:PATH=$HDFDIR/lib \
    \
        -D TPL_ENABLE_BLAS:STRING=ON \
        -D TPL_ENABLE_LAPACK:STRING=ON \
        -D TPL_BLAS_LIBRARIES:STRING="$LABLAS_LIBRARIES" \
        -D TPL_LAPACK_LIBRARIES:STRING="$LABLAS_LIBRARIES" \
    \
        -D TPL_ENABLE_ParMETIS:STRING=ON \
        -D ParMETIS_INCLUDE_DIRS:PATH="${PARMETISDIR}/include" \
        -D ParMETIS_LIBRARY_DIRS:PATH="${PARMETISDIR}/lib" \
    \
    	-D CMAKE_INSTALL_PREFIX:PATH=$INSTALL_DIR \
        -D Trilinos_EXTRA_LINK_FLAGS:STRING="-L${HDFDIR}/lib -lnetcdf -lhdf5_hl -lhdf5 -lz -lgfortran -Wl,-rpath,/projects/albany/lib" \
    \
        -D Trilinos_ENABLE_Moertel:BOOL=ON \
        -D Moertel_ENABLE_EXAMPLES:BOOL=ON \
        -D Moertel_ENABLE_Morkon:BOOL=ON \
        -D Trilinos_ENABLE_TriKota:BOOL=OFF \
        -D HAVE_INTREPID_KOKKOSCORE:BOOL=ON \
        -D Trilinos_ENABLE_SECONDARY_TESTED_CODE=OFF \
        -D Trilinos_ENABLE_Amesos2:BOOL=ON \
        -D Amesos2_ENABLE_KLU2:BOOL=ON \
    \
    $EXTRA_ARGS \
    ${TRILINOS_HOME}
  

To build Trilinos, use an out-of-source build. I usually make a build directory that matches the configuration of Trilinos that I am building in the Trilinos root directory:

	cd /home/gahanse/Codes/Trilinos
    mkdir build
	cd build
	mkdir MPI_REL

• Place the above do-configure file into the MPI_REL directory, change directories into MPI_REL, and perform the configure, build, and install:

    cd MPI_REL
    ./do-configure
    make -j 4
    make -j 4 install
  

Build Albany

• CMake configure script:

    #!/bin/sh
    
    export ALB_64BIT_INT=OFF
    export ALB_EPETRA=ON
    export ALB_ENABLE_SCOREC=ON
    export ALB_ENABLE_LCM=ON
    export ALB_ENABLE_LAME=OFF
    export ALB_ENABLE_HYDRIDE=OFF
    export ALB_ENABLE_QCAD=OFF
    export ALB_ENABLE_MOR=OFF
    export ALB_ENABLE_ASCR=OFF
    export ALB_ENABLE_AERAS=ON
    export ALB_ENABLE_FELIX=ON
    export ALB_ENABLE_AlbanyCI=OFF
    export ALB_DEBUG=OFF
    export ALB_ENABLE_CHECK_FPE=OFF
    export ALB_ENABLE_SPECULATIVE=OFF
    export ALB_ENABLE_SG_MP=OFF
    export ALB_ENABLE_PERF_TESTS=ON
    export TRILINOS_INSTALL_DIR=/projects/albany/trilinos/MPI_REL
    export ALB_VERBOSE=OFF
    
    cmake \
          -D ALBANY_TRILINOS_DIR:FILEPATH="$TRILINOS_INSTALL_DIR" \
          -D ENABLE_LCM:BOOL=${ALB_ENABLE_LCM} \
          -D ENABLE_AERAS:BOOL=${ALB_ENABLE_AERAS} \
          -D ENABLE_QCAD:BOOL=${ALB_ENABLE_QCAD} \
          -D ENABLE_HYDRIDE:BOOL=${ALB_ENABLE_HYDRIDE} \
          -D ENABLE_LCM_SPECULATIVE:BOOL=${ALB_ENABLE_SPECULATIVE} \
          -D CMAKE_VERBOSE_MAKEFILE:BOOL=${ALB_VERBOSE} \
          -D ENABLE_DEBUGGING:BOOL=${ALB_DEBUG} \
          -D ENABLE_CHECK_FPE:BOOL=${ALB_ENABLE_CHECK_FPE} \
          -D ENABLE_SCOREC:BOOL=${ALB_ENABLE_SCOREC} \
          -D ENABLE_FELIX:BOOL=${ALB_ENABLE_FELIX} \
          -D ENABLE_MOR:BOOL=${ALB_ENABLE_MOR} \
          -D ENABLE_ALBANY_CI:BOOL=${ALB_ENABLE_AlbanyCI} \
          -D ENABLE_ASCR:BOOL=${ALB_ENABLE_ASCR} \
          -D ENABLE_SG_MP:BOOL=${ALB_ENABLE_SG_MP} \
          -D ENABLE_PERFORMANCE_TESTS:BOOL=${ALB_ENABLE_PERF_TESTS} \
          -D ALBANY_CTEST_TIMEOUT:INTEGER=70 \
          -D ENABLE_64BIT_INT:BOOL=${ALB_64BIT_INT} \
          -D ENABLE_ALBANY_EPETRA_EXE:BOOL=${ALB_EPETRA} \
    \
    ..
  

Clone the Albany repo in a suitable place:

	cd /home/ghansen/Codes
	git clone https://github.com/SNLComputation/Albany.git

• Create an out-of-source build directory

    cd Albany
    mkdir build
  

• Place the above configure script into the build directory, then configure, build, and test

    cd build
    ./buildAlbany
    make -j 4
    ctest