Rhel6 using Gcc

Glen Hansen edited this page Jun 11, 2016 · 5 revisions

This set of build instructions is for the Albany suite on Rhel6 linux.

The compilers are gcc 5.2.0, OpenMPI 1.8.5, and RHEL 6 linux.

Modules on CEE:

	module load sierra-devel/gcc-5.2.0-openmpi-1.8.8
	module load sierra-mkl/16.0-2016.2.181

Build supporting packages

Trilinos requires a set of "third party libraries" (TPLs) to support Albany. The actual libraries needed depends on the final Albany configuration desired. These instructions build a superset, all the libraries needed to support any Albany configuration.

Notes: In this example

  • The TPLs are installed in /projects/albany

Building ZLib

  • Version 1.2.8

      CC=mpicc CFLAGS=-O3 ./configure --64 --archs="-arch x86_64" --prefix=/projects/albany
      make -j 4
      make install
    

Building HDF5

  • Version 1.8.17

      ./configure CC=mpicc FC=mpifort CXX=mpicxx CXXFLAGS="-O3 -fPIC -march=native" \
       CFLAGS="-O3 -fPIC -march=native" \
       FCFLAGS="-O3 -fPIC -march=native" --enable-parallel \
       --with-zlib=/projects/albany --prefix=/projects/albany
      make -j 4
      make install
    

Building parallel-netcdf

  • Version 1.7.0

  • Configure and build:

      ./configure --prefix=/projects/albany MPICC=mpicc MPIF77=mpifort MPIF90=mpifort MPICXX=mpicxx \
      CXXFLAGS="-fPIC -O3 -march=native" CFLAGS="-fPIC -O3 -march=native" \
      F77FLAGS="-fPIC -O3 -march=native" F90FLAGS="-fPIC -O3 -march=native"
      make -j 8 
      make install
    

Building Netcdf

  • Version 4.4.0

  • Edit the file include/netcdf.h. Make the following edits near line 265:

      #define NC_MAX_DIMS    65536    /* max dimensions per file */
      #define NC_MAX_ATTRS    8192
      #define NC_MAX_VARS   524288    /* max variables per file */
      #define NC_MAX_NAME   256
      #define NC_MAX_VAR_DIMS      8     /* max per variable dimensions */
    
  • Now, configure using CMake and build:

      ./configure CC=mpicc FC=mpifort CXX=mpicxx \
          CXXFLAGS="-fPIC -I/projects/albany/include -O3 -march=native" \
          CFLAGS="-fPIC -I/projects/albany/include -O3 -march=native" \
          LDFLAGS="-fPIC -L/projects/albany/lib -O3 -march=native" \
          FCFLAGS="-fPIC -I/projects/albany/include -O3 -march=native" \
          --prefix=/projects/albany --disable-doxygen --enable-netcdf4 --enable-pnetcdf
      make -j 8
      make install
    

Building boost

  • Version 1.55.0

  • Build boost with the commands

      echo "using mpi : /sierra/sntools/SDK/mpi/openmpi/1.8.5-gcc-5.2.0-RHEL6/bin/mpicxx ;" >> ./tools/build/v2/user-config.jam
      echo "using gcc : /sierra/sntools/SDK/compilers/gcc/5.2.0-RHEL6/bin/g++ ;" >> ./tools/build/v2/user-config.jam
      ./bootstrap.sh --with-libraries=signals,regex,filesystem,system,mpi,serialization,thread,program_options,exception --prefix=/projects/albany
      ./b2 -j 4
      ./b2 -j 4 install
    

Building ParMetis

  • Version 4.0.3

      wget http://glaros.dtc.umn.edu/gkhome/fetch/sw/parmetis/parmetis-4.0.3.tar.gz
      tar -xvf parmetis-4.0.3.tar.gz
    
  • Edit metis.h to use an IDXTYPEWIDTH of 64

      cd parmetis-4.0.3/metis/include
      vi metis.h
      #define IDXTYPEWIDTH 64
      #define REALTYPEWIDTH 64
    
  • Run CMake to build parmetis

      cd build
      export metis=/projects/albany/src/parmetis-4.0.3/metis/  
      cmake \
      -DCMAKE_INSTALL_PREFIX=/projects/albany/parmetis-4.0.3 \
      -DMETIS_PATH=$metis \
      -DGKLIB_PATH=$metis/GKlib \
      -DCMAKE_C_COMPILER=mpicc \
      -DCMAKE_CXX_COMPILER=mpicxx \
      -DCMAKE_C_FLAGS=-O3  \
      -DCMAKE_CXX_FLAGS=-O3  \
      ..
      make -j 4
      make install
    
      cp libmetis/libmetis.a /projects/albany/parmetis-4.0.3/lib
      cp $metis/include/metis.h /projects/albany/parmetis-4.0.3/include
    

SuperLU (optional)

  • Version 4.3

  • Build paths

      mkdir /projects/albany/SuperLU_4.3
      mkdir /projects/albany/SuperLU_4.3/include
      mkdir /projects/albany/SuperLU_4.3/lib
    
  • Edit make.inc for your machine and environment

      PLAT = _linux
      SuperLUroot = /projects/albany/SuperLU_4.3
      SUPERLULIB      = $(SuperLUroot)/lib/libsuperlu_4.3.a
      TMGLIB          = libtmglib.a
      BLASDEF     = -DUSE_VENDOR_BLAS
      BLASLIB     = -L/sierra/sntools/SDK/compilers/intel/composer_xe_2016.3.210/mkl/lib/intel64 -lmkl_intel_lp64 -lmkl_sequential -lmkl_core
      LIBS        = $(SUPERLULIB) $(BLASLIB)
      ARCH         = ar
      ARCHFLAGS    = cr
      RANLIB       = ranlib
      CC           = gcc
      CFLAGS       = -O3
      NOOPTS       =
      FORTRAN      = gfortran
      FFLAGS       = -O3
      LOADER       = $(CC)
      LOADOPTS     =
      CDEFS        = -DAdd_
      MATLAB       = /usr/sww/matlab
    
  • Build SuperLU

      make -j 4
      cd SRC
      cp *.h /projects/albany/SuperLU_4.3/include		
    

Build HWLOC

  • HWLOC is an optional package that can be used for thread pinning by Kokkos. If you are not building Kokkos, or if you don't care about thread pinning, please skip.

  • Version 1.8

      ./configure CC=mpicc CXX=mpicxx --prefix=/projects/albany
      make -j 4
      make install
    

Build Trilinos

Notes: In this example

  • The Trilinos source is installed in /home/gahanse/Codes/Trilinos, using a command like

      cd /home/gahanse/Codes
      git clone git@github.com:trilinos/Trilinos.git
    
  • The SCOREC tools are installed as a subdirectory of Trilinos, and are build automatically by the Trilinos build process. The commands to do this are

      cd /home/gahanse/Codes/Trilinos
      git clone https://github.com/SCOREC/core.git SCOREC
    
  • do-configure script

      #!/bin/sh
      #
      export TRILINOS_HOME=/projects/albany/src/Trilinos
      BUILD_DIR=`pwd`
      INSTALL_DIR=/projects/albany/trilinos/MPI_REL
      BOOST_DIR=/projects/albany
      LIB_DIR=/projects/albany
      MPI_BASE_DIR=/sierra/sntools/SDK/mpi/openmpi/1.8.8-gcc-5.2.0-RHEL6
      NETCDF=/projects/albany
      HDFDIR=/projects/albany
      PARMETISDIR=/projects/albany
      MKL_PATH=/sierra/sntools/SDK/compilers/intel/composer_xe_2016.3.210
      HWLOC_PATH=/projects/albany
      LABLAS_LIBRARIES="-L$MKL_PATH/lib/intel64 -Wl,--start-group $MKL_PATH/mkl/lib/intel64/libmkl_intel_lp64.a $MKL_PATH/mkl/lib/intel64/libmkl_core.a $MKL_PATH/mkl/lib/intel64/libmkl_sequential.a -Wl,--end-group"
      
      cmake \
          -Wno-dev \
          -D Trilinos_CONFIGURE_OPTIONS_FILE:FILEPATH=$TRILINOS_HOME/sampleScripts/AlbanySettings.cmake \
          -D Trilinos_ENABLE_SCOREC:BOOL=ON \
          -D SCOREC_DISABLE_STRONG_WARNINGS:BOOL=ON \
          -D CMAKE_BUILD_TYPE:STRING=NONE \
          -D CMAKE_CXX_FLAGS:STRING="-O3 -march=native -DNDEBUG" \
          -D CMAKE_C_FLAGS:STRING="-O3 -march=native -DNDEBUG" \
          -D CMAKE_Fortran_FLAGS:STRING="-O3 -march=native -DNDEBUG" \
          -D Rythmos_ENABLE_DEBUG:BOOL=ON \
          -D KokkosClassic_DefaultNode:STRING="Kokkos::Compat::KokkosSerialWrapperNode" \
          -D Trilinos_ENABLE_EXPLICIT_INSTANTIATION:BOOL=ON \
          -D Tpetra_INST_INT_LONG_LONG:BOOL=ON \
          -D Tpetra_INST_INT_INT:BOOL=ON \
          -D Tpetra_INST_DOUBLE:BOOL=ON \
          -D Tpetra_INST_FLOAT:BOOL=OFF \
          -D Tpetra_INST_COMPLEX_FLOAT:BOOL=OFF \
          -D Tpetra_INST_COMPLEX_DOUBLE:BOOL=OFF \
          -D Tpetra_INST_INT_LONG:BOOL=OFF \
          -D Tpetra_INST_INT_UNSIGNED:BOOL=OFF \
          -D Trilinos_ENABLE_Kokkos:BOOL=ON \
          -D Trilinos_ENABLE_KokkosCore:BOOL=ON \
          -D Phalanx_KOKKOS_DEVICE_TYPE:STRING="SERIAL" \
          -D Phalanx_INDEX_SIZE_TYPE:STRING="INT" \
          -D Phalanx_SHOW_DEPRECATED_WARNINGS:BOOL=OFF \
          -D Kokkos_ENABLE_Serial:BOOL=ON \
          -D Kokkos_ENABLE_OpenMP:BOOL=OFF \
          -D Kokkos_ENABLE_Pthread:BOOL=OFF \
      \
          -D TPL_ENABLE_MPI:BOOL=ON \
          -D MPI_BASE_DIR:PATH=$MPI_BASE_DIR \
      \
      	-D TPL_ENABLE_Pthread:BOOL=OFF \
      \
          -D TPL_ENABLE_Boost:BOOL=ON \
          -D TPL_ENABLE_BoostLib:BOOL=ON \
          -D TPL_ENABLE_BoostAlbLib:BOOL=ON \
      	-D Boost_INCLUDE_DIRS:PATH=$BOOST_DIR/include \
      	-D Boost_LIBRARY_DIRS:PATH=$BOOST_DIR/lib \
      	-D BoostLib_INCLUDE_DIRS:PATH=$BOOST_DIR/include \
      	-D BoostLib_LIBRARY_DIRS:PATH=$BOOST_DIR/lib \
      	-D BoostAlbLib_INCLUDE_DIRS:PATH=$BOOST_DIR/include \
      	-D BoostAlbLib_LIBRARY_DIRS:PATH=$BOOST_DIR/lib \
      \
          -D TPL_ENABLE_Netcdf:STRING=ON \
      	-D Netcdf_INCLUDE_DIRS:PATH=$NETCDF/include \
      	-D Netcdf_LIBRARY_DIRS:PATH=$NETCDF/lib \
      \
          -D TPL_ENABLE_HDF5:STRING=ON \
          -D HDF5_INCLUDE_DIRS:PATH=$HDFDIR/include \
          -D HDF5_LIBRARY_DIRS:PATH=$HDFDIR/lib \
      \
          -D TPL_ENABLE_Zlib:STRING=ON \
          -D Zlib_INCLUDE_DIRS:PATH=$HDFDIR/include \
          -D Zlib_LIBRARY_DIRS:PATH=$HDFDIR/lib \
      \
          -D TPL_ENABLE_BLAS:STRING=ON \
          -D TPL_ENABLE_LAPACK:STRING=ON \
          -D TPL_BLAS_LIBRARIES:STRING="$LABLAS_LIBRARIES" \
          -D TPL_LAPACK_LIBRARIES:STRING="$LABLAS_LIBRARIES" \
      \
          -D TPL_ENABLE_ParMETIS:STRING=ON \
          -D ParMETIS_INCLUDE_DIRS:PATH="${PARMETISDIR}/include" \
          -D ParMETIS_LIBRARY_DIRS:PATH="${PARMETISDIR}/lib" \
      \
      	-D CMAKE_INSTALL_PREFIX:PATH=$INSTALL_DIR \
          -D Trilinos_EXTRA_LINK_FLAGS:STRING="-L${HDFDIR}/lib -lnetcdf -lhdf5_hl -lhdf5 -lz -lgfortran -Wl,-rpath,/projects/albany/lib" \
      \
          -D Trilinos_ENABLE_Moertel:BOOL=ON \
          -D Moertel_ENABLE_EXAMPLES:BOOL=ON \
          -D Moertel_ENABLE_Morkon:BOOL=ON \
          -D Trilinos_ENABLE_TriKota:BOOL=OFF \
          -D HAVE_INTREPID_KOKKOSCORE:BOOL=ON \
          -D Trilinos_ENABLE_SECONDARY_TESTED_CODE=OFF \
          -D Trilinos_ENABLE_Amesos2:BOOL=ON \
          -D Amesos2_ENABLE_KLU2:BOOL=ON \
      \
      $EXTRA_ARGS \
      ${TRILINOS_HOME}
    

To build Trilinos, use an out-of-source build. I usually make a build directory that matches the configuration of Trilinos that I am building in the Trilinos root directory:

	cd /home/gahanse/Codes/Trilinos
    mkdir build
	cd build
	mkdir MPI_REL
  • Place the above do-configure file into the MPI_REL directory, change directories into MPI_REL, and perform the configure, build, and install:

      cd MPI_REL
      ./do-configure
      make -j 4
      make -j 4 install
    

Build Albany

  • CMake configure script:

      #!/bin/sh
      
      export ALB_64BIT_INT=OFF
      export ALB_EPETRA=ON
      export ALB_ENABLE_SCOREC=ON
      export ALB_ENABLE_LCM=ON
      export ALB_ENABLE_LAME=OFF
      export ALB_ENABLE_HYDRIDE=OFF
      export ALB_ENABLE_QCAD=OFF
      export ALB_ENABLE_MOR=OFF
      export ALB_ENABLE_ASCR=OFF
      export ALB_ENABLE_AERAS=ON
      export ALB_ENABLE_FELIX=ON
      export ALB_ENABLE_AlbanyCI=OFF
      export ALB_DEBUG=OFF
      export ALB_ENABLE_CHECK_FPE=OFF
      export ALB_ENABLE_SPECULATIVE=OFF
      export ALB_ENABLE_SG_MP=OFF
      export ALB_ENABLE_PERF_TESTS=ON
      export TRILINOS_INSTALL_DIR=/projects/albany/trilinos/MPI_REL
      export ALB_VERBOSE=OFF
      
      cmake \
            -D ALBANY_TRILINOS_DIR:FILEPATH="$TRILINOS_INSTALL_DIR" \
            -D ENABLE_LCM:BOOL=${ALB_ENABLE_LCM} \
            -D ENABLE_AERAS:BOOL=${ALB_ENABLE_AERAS} \
            -D ENABLE_QCAD:BOOL=${ALB_ENABLE_QCAD} \
            -D ENABLE_HYDRIDE:BOOL=${ALB_ENABLE_HYDRIDE} \
            -D ENABLE_LCM_SPECULATIVE:BOOL=${ALB_ENABLE_SPECULATIVE} \
            -D CMAKE_VERBOSE_MAKEFILE:BOOL=${ALB_VERBOSE} \
            -D ENABLE_DEBUGGING:BOOL=${ALB_DEBUG} \
            -D ENABLE_CHECK_FPE:BOOL=${ALB_ENABLE_CHECK_FPE} \
            -D ENABLE_SCOREC:BOOL=${ALB_ENABLE_SCOREC} \
            -D ENABLE_FELIX:BOOL=${ALB_ENABLE_FELIX} \
            -D ENABLE_MOR:BOOL=${ALB_ENABLE_MOR} \
            -D ENABLE_ALBANY_CI:BOOL=${ALB_ENABLE_AlbanyCI} \
            -D ENABLE_ASCR:BOOL=${ALB_ENABLE_ASCR} \
            -D ENABLE_SG_MP:BOOL=${ALB_ENABLE_SG_MP} \
            -D ENABLE_PERFORMANCE_TESTS:BOOL=${ALB_ENABLE_PERF_TESTS} \
            -D ALBANY_CTEST_TIMEOUT:INTEGER=70 \
            -D ENABLE_64BIT_INT:BOOL=${ALB_64BIT_INT} \
            -D ENABLE_ALBANY_EPETRA_EXE:BOOL=${ALB_EPETRA} \
      \
      ..
    

Clone the Albany repo in a suitable place:

	cd /home/ghansen/Codes
	git clone https://github.com/gahansen/Albany.git
  • Create an out-of-source build directory

      cd Albany
      mkdir build
    
  • Place the above configure script into the build directory, then configure, build, and test

      cd build
      ./buildAlbany
      make -j 4
      ctest
    
Clone this wiki locally
You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.
Press h to open a hovercard with more details.