Rhel6 using Intel

Glen Hansen edited this page Apr 19, 2017 · 9 revisions

This set of build instructions are for the Albany suite on Rhel6 linux.

The compilers are Intel 17.0, Intel MPI 5.1, and RHEL 6 linux.

Modules on CEE:

	module load sierra-devel/intel-17.0.1-intelmpi-5.1
	module load sierra-mkl/17.0-2017.2.174

Build supporting packages

Trilinos requires a set of "third party libraries" (TPLs) to support Albany. The actual libraries needed depends on the final Albany configuration desired. These instructions build a superset, all the libraries needed to support any Albany configuration.

Notes: In this example

  • The TPLs are installed in /projects/albany

Building ZLib

  • Version 1.2.11

      CC=mpiicc CFLAGS="-O3 -DNDEBUG -diag-disable=cpu-dispatch -mkl=sequential" ./configure --64 --prefix=/projects/albany/intel5.1
      make -j 4
      make install

Building Yaml

  • git clone git@github.com:jbeder/yaml-cpp.git

      cmake \
      	-DCMAKE_CXX_FLAGS:STRING='-O3 -DNDEBUG -diag-disable=cpu-dispatch -mkl=sequential' \
      	-DCMAKE_C_FLAGS:STRING='-O3 -DNDEBUG -diag-disable=cpu-dispatch -mkl=sequential' \
      	-DCMAKE_INSTALL_PREFIX:PATH=/projects/albany/intel5.1 \
      make -j 4
      make install

Building HDF5

  • Version 1.10.0 patch 1

      ./configure CC=mpiicc FC=mpiifort CXX=mpiicpc CXXFLAGS="-O3 -fPIC -DNDEBUG -diag-disable=cpu-dispatch -mkl=sequential" \
       CFLAGS="-O3 -fPIC -DNDEBUG -diag-disable=cpu-dispatch -mkl=sequential" \
       FCFLAGS="-O3 -fPIC -DNDEBUG -diag-disable=cpu-dispatch -mkl=sequential" --enable-parallel \
       --with-zlib=/projects/albany/intel5.1 --prefix=/projects/albany/intel5.1
      make -j 4
      make install

Building parallel-netcdf

  • Version 1.8.1

  • Configure and build:

      ./configure --prefix=/projects/albany/intel5.1 CC=mpiicc F77=mpiifort F90=mpiifort CXX=mpiicpc \
      CXXFLAGS="-O3 -fPIC -DNDEBUG -diag-disable=cpu-dispatch -mkl=sequential" \
      CFLAGS="-O3 -fPIC -DNDEBUG -diag-disable=cpu-dispatch -mkl=sequential" \
      FFLAGS="-O3 -fPIC -DNDEBUG -diag-disable=cpu-dispatch -mkl=sequential" \
      FCFLAGS="-O3 -fPIC -DNDEBUG -diag-disable=cpu-dispatch -mkl=sequential" \
      make -j 8 
      make install

Building Netcdf

  • Version

  • Edit the file include/netcdf.h. Make the following edits near line 265:

      #define NC_MAX_DIMS    65536    /* max dimensions per file */
      #define NC_MAX_ATTRS    8192
      #define NC_MAX_VARS   524288    /* max variables per file */
      #define NC_MAX_NAME   256
      #define NC_MAX_VAR_DIMS      8     /* max per variable dimensions */
  • Add the install directory of the hdf5 shared libs and the libpnetcdf.so file to your LD_LIBRARY_PATH or the config below will fail:

      export LD_LIBRARY_PATH=/projects/albany/intel5.1/lib:$LD_LIBRARY_PATH	
  • Now, configure using CMake and build:

      ./configure CC=mpiicc FC=mpiifort CXX=mpiicpc \
          CXXFLAGS="-fPIC -I/projects/albany/intel5.1/include -O3 -DNDEBUG -diag-disable=cpu-dispatch -mkl=sequential" \
          CFLAGS="-fPIC -I/projects/albany/intel5.1/include -O3 -DNDEBUG -diag-disable=cpu-dispatch -mkl=sequential" \
          LDFLAGS="-fPIC -L/projects/albany/intel5.1/lib -O3 -DNDEBUG -diag-disable=cpu-dispatch -mkl=sequential" \
          FCFLAGS="-fPIC -I/projects/albany/intel5.1/include -O3 -DNDEBUG -diag-disable=cpu-dispatch -mkl=sequential" \
          --prefix=/projects/albany/intel5.1 --enable-netcdf-4 --enable-pnetcdf
      make -j 8
      make install

Building boost

  • Version 1.63.0

  • Build boost with the commands

      echo 'using intel-linux : : /sierra/sntools/SDK/compilers/intel/composer_xe_2017.2.174/compilers_and_libraries_2017.2.174/linux/bin/intel64/icpc : <cxxflags>"-O3 -DNDEBUG -diag-disable=cpu-dispatch" ;' > ~/user-config.jam
      echo 'using mpi : /sierra/sntools/SDK/compilers/intel/composer_xe_2017.2.174/compilers_and_libraries_2017.2.174/linux/mpi/intel64/bin/mpiicpc : <cxxflags>"-I/sierra/sntools/SDK/compilers/intel/composer_xe_2017.2.174/compilers_and_libraries_2017.2.174/linux/mpi/intel64/include -O3 -DNDEBUG -diag-disable=cpu-dispatch" ;' >> ~/user-config.jam
      ./bootstrap.sh --with-toolset=intel-linux --with-libraries=signals,regex,filesystem,system,mpi,serialization,thread,program_options,exception --prefix=/projects/albany/intel5.1
      ./b2 toolset=intel-linux -j 4
      ./b2 toolset=intel-linux -j 4 install
  • The above bjam command gave me an error, so I removed the following lines from project-config.jam:

      if ! intel-linux in [ feature.values <toolset> ]
          using intel-linux ;

Building ParMetis

  • Version 4.0.3

      wget http://glaros.dtc.umn.edu/gkhome/fetch/sw/parmetis/parmetis-4.0.3.tar.gz
      tar -xvf parmetis-4.0.3.tar.gz
  • Edit metis.h to use an IDXTYPEWIDTH of 64

      cd parmetis-4.0.3/metis/include
      vi metis.h
      #define IDXTYPEWIDTH 64
      #define REALTYPEWIDTH 64
  • Run CMake to build parmetis

      cd /projects/albany/src/parmetis-4.0.3 
      cd build
      export metis=/projects/albany/src/parmetis-4.0.3/metis/  
      cmake \
      -DCMAKE_INSTALL_PREFIX=/projects/albany/intel5.1 \
      -DMETIS_PATH=$metis \
      -DGKLIB_PATH=$metis/GKlib \
      -DCMAKE_C_COMPILER=mpiicc \
      -DCMAKE_CXX_COMPILER=mpiicpc \
      -DCMAKE_C_FLAGS="-O3 -DNDEBUG -diag-disable=cpu-dispatch -fPIC" \
      -DCMAKE_CXX_FLAGS="-O3 -DNDEBUG -diag-disable=cpu-dispatch -fPIC" \
      make -j 4
      make install
      cp libmetis/libmetis.a /projects/albany/intel5.1/lib
      cp $metis/include/metis.h /projects/albany/intel5.1/include

SuperLU (optional)

  • Version 4.3

  • Build paths

      mkdir /projects/albany/intel5.1/SuperLU_4.3
      mkdir /projects/albany/intel5.1/SuperLU_4.3/include
      mkdir /projects/albany/intel5.1/SuperLU_4.3/lib
  • Edit make.inc for your machine and environment

      PLAT = _linux
      SuperLUroot = /projects/albany/intel5.1/SuperLU_4.3
      SUPERLULIB      = $(SuperLUroot)/lib/libsuperlu_4.3.a
      TMGLIB          = libtmglib.a
      MKLHOME         = /sierra/sntools/SDK/compilers/intel/composer_xe_2017.2.174/compilers_and_libraries_2017.2.174/linux/mkl/lib/intel64
      BLASLIB     = -L${MKLHOME} -lmkl_intel_lp64 -lmkl_sequential -lmkl_core
      LIBS        = $(SUPERLULIB) $(BLASLIB)
      ARCH         = xiar
      ARCHFLAGS    = cr
      RANLIB       = ranlib
      CC           = mpiicc 
      CFLAGS       = -O3 -DNDEBUG -DPRNTlevel=0 -DAdd_ -mkl=sequential
      NOOPTS       =
      FORTRAN      = mpiifort
      FFLAGS       = -O3 -mkl=sequential
      LOADER       = $(CC)
      LOADOPTS     =
      MATLAB       = /usr/sww/matlab
  • Build SuperLU

      cd SRC
      cp *.h /projects/albany/intel5.1/SuperLU_4.3/include		
  • Note: Do not make SuperLU in parallel (-j * make argument) as the build will be incomplete.


  • HWLOC is an optional package that can be used for thread pinning by Kokkos. If you are not building Kokkos, or if you don't care about thread pinning, please skip.

  • Version 1.11.6

      ./configure CC=mpiicc CXX=mpiicpc --prefix=/projects/albany/intel5.1
      make -j 4
      make install
Clone this wiki locally
You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.
Press h to open a hovercard with more details.