Westley

Glen Hansen edited this page Jun 11, 2016 · 5 revisions

Set up your build environment

This build is based on the 2016 Intel compilers

	. /opt/intel/compilers_and_libraries/linux/bin/compilervars.sh intel64
	export PATH=/usr/local/trilinos/MPI_REL/bin:/usr/local/gcc-5.1.0/bin:/opt/intel/bin:/usr/local/bin:/usr/lib64/qt-3.3/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:$HOME/bin
	. /opt/intel/impi/5.1.2.150/bin64/mpivars.sh
	export LD_LIBRARY_PATH=/opt/intel/composer_xe_2015.3.187/compiler/lib/mic:/usr/local/gcc-5.1.0/lib64:$LD_LIBRARY_PATH
	export I_MPI_MIC=1

These install directions assume that a base install directory of /usr/local/mic is used

Install the required Third Party Libraries (TPLs) for Trilinos

Build HWLOC

  • HWLOC is an optional package that can be used for thread pinning by Kokkos. If you are not building Kokkos, or if you don't care about thread pinning, please skip.

  • Version 1.10

      cd /usr/local/src
      wget http://www.open-mpi.org/software/hwloc/v1.10/downloads/hwloc-1.10.0.tar.bz2
      tar -xvf hwloc-1.10.0.tar.bz2
      cd hwloc-1.10.0
      ./configure CC=mpiicc CFLAGS="-O3 -mmic -fopenmp" CXX=mpiicpc \
      	CXXFLAGS="-O3 -mmic -fopenmp" LDFLAGS=-static \
      	--host=x86_64-k1om-linux --enable-shared=no \
      	--enable-static=yes --prefix=/usr/local/mic
      make -j 8
      make install
    

Building ZLib

  • Version 1.2.8

      cd /usr/local/src
      wget http://zlib.net/zlib-1.2.8.tar.gz
      tar -xvf zlib-1.2.8.tar.gz
      cd zlib-1.2.8
      CC=mpiicc CXX=mpiicpc CFLAGS="-O3 -mmic -fopenmp" CXXFLAGS="-O3 -mmic -fopenmp" \
      	./configure --64 --static --prefix=/usr/local/mic
      make install
    

Building HDF5

  • Version 1.8.17

Cross compiling HDF5 for the MIC can be a challenge. Please see the instructions at:

	http://linux4hippos.blogspot.com/2013/05/cross-compile-hdf5-for-intel-xeon-phi.html

Build Instructions: Note

  • The above website is based on HDF5-1.8.9 but they work with version HDF5-1.8.17

  • The above website, step (1), instructs one to set TEST_SCRIPT = in several files. These instructions were followed here.

  • as_fn_exit was commented out in three places in the configure script as described in step (2) above

      cd hdf5-1.8.17
      ./configure CC=mpiicc FC=mpiifort CXX=mpiicpc CXXFLAGS="-O3 -mmic -fopenmp" \
      	CFLAGS="-O3 -mmic -fopenmp" FCFLAGS="-O3 -mmic -fopenmp" -host=x86_64-k1om-linux \
      	--enable-parallel --enable-shared=no --with-zlib=/usr/local/mic --prefix=/usr/local/mic
      make -j 8
    

Note that the build fails as expected (above website, step (6)). Follow the instructions given to proceed.

Building parallel-netcdf

  • Version 1.7.0

  • Configure and build:

      ./configure --prefix=/usr/local/mic MPICC=mpiicc MPIF77=mpiifort MPIF90=mpiifort MPICXX=mpiicpc \
      CXXFLAGS="-O3 -mmic -fopenmp" CFLAGS="-O3 -mmic -fopenmp" \
      F77FLAGS="-O3 -mmic -fopenmp" F90FLAGS="-O3 -mmic -fopenmp" -host=x86_64-k1om-linux
      make -j 8 
      make install
    

Building Netcdf

  • Version 4.4.0

  • Edit the file include/netcdf.h. Make the following edits near line 265:

      #define NC_MAX_DIMS    65536    /* max dimensions per file */
      #define NC_MAX_ATTRS    8192    
      #define NC_MAX_VARS   524288    /* max variables per file */
      #define NC_MAX_NAME   256 
      #define NC_MAX_VAR_DIMS      8     /* max per variable dimensions */
    
  • Now, configure using CMake and build:

      cd hdf5-1.8.17
      ./configure CC=mpiicc FC=mpiifort CXX=mpiicpc CXXFLAGS="-O3 -mmic -fopenmp -I/usr/local/mic/include" \
      	CFLAGS="-O3 -mmic -fopenmp -I/usr/local/mic/include" \
          LDFLAGS="-O3 -mmic -fopenmp -L/usr/local/mic/lib" \
          FCFLAGS="-O3 -mmic -fopenmp -I/usr/local/mic/include" -host=x86_64-k1om-linux \
      	--prefix=/usr/local/mic --enable-netcdf-4 --disable-dap --enable-shared=no \
          --disable-utilities --disable-testsets --enable-pnetcdf
      make -j 8
      make install
    

Building boost

  • Version 1.58.0

      cd /usr/local/src/
      wget http://sourceforge.net/projects/boost/files/boost/1.58.0/boost_1_58_0.tar.gz/download boost_1_58_0.tar.gz
      tar -xvf boost_1_58_0.tar.gz
      cd boost_1_58_0
    
  • Build boost with the commands

      echo "using intel : mic : : <cxxflags>\"-O3 -mmic -fopenmp\" ;" \
      	>> ~/user-config.jam
      echo "using mpi : /opt/intel/compilers_and_libraries_2016.1.150/linux/mpi/intel64/bin/mpiicpc : <cxxflags>\"-I/opt/intel/compilers_and_libraries_2016.1.150/linux/mpi/intel64/include -mmic -fopenmp\" ;" \
      	>> ~/user-config.jam
      ./bootstrap.sh --with-toolset=intel \
      --with-libraries=signals,regex,filesystem,system,mpi,serialization,thread,program_options,exception \
      --prefix=/usr/local/mic
      ./b2 -j 8
      ./b2 -j 8 install
    
  • The above bjam command gave me an error, so I removed the following lines from project-config.jam:

      if ! intel-darwin in [ feature.values <toolset> ]
      {
          using intel-darwin ;
      }
    

Building ParMetis

  • Version 4.0.3

      cd /usr/local/src/
      wget http://glaros.dtc.umn.edu/gkhome/fetch/sw/parmetis/parmetis-4.0.3.tar.gz
      tar -xvf parmetis-4.0.3.tar.gz
      cd parmetis-4.0.3
    
  • Edit metis.h to use an IDXTYPEWIDTH of 64

      cd metis/include
      vi metis.h
      #define IDXTYPEWIDTH 64
      #define REALTYPEWIDTH 64
    
  • Run CMake to build parmetis

      cd /usr/local/src/parmetis-4.0.3
      cd build
      export metis=/usr/local/src/parmetis-4.0.3/metis
    
      cmake \
      -DCMAKE_INSTALL_PREFIX=/usr/local/mic \
      -DMETIS_PATH=$metis \
      -DGKLIB_PATH=$metis/GKlib \
      -DCMAKE_C_COMPILER=mpiicc \
      -DCMAKE_C_FLAGS="-O3 -mmic -fopenmp" \
      -DCMAKE_CXX_COMPILER=mpiicpc \
      -DCMAKE_CXX_FLAGS="-O3 -mmic -fopenmp" \
      ..
    
      make -j 8
      make install
    
      cp libmetis/libmetis.a /usr/local/mic/lib
      cp $metis/include/metis.h /usr/local/mic/include
    

Build Trilinos

  • Download Trilinos

      cd /usr/local/src
      git clone git@github.com:trilinos/Trilinos.git
    
  • Download SCOREC

      cd /usr/local/src/Trilinos
      git clone git@github.com:SCOREC/core.git SCOREC
    
  • Configure the Trilinos build

      mkdir build
      cd build
      ./do-configure
    
  • Example do-configure script executed above

      #!/bin/sh
      #
      
      export TRILINOS_HOME=/usr/local/src/Trilinos
      BUILD_DIR=`pwd`
      INSTALL_DIR=/usr/local/mic/trilinos/MPI_MIC
      BOOST_DIR=/usr/local/mic/boost-1.58.0
      MPI_BASE_DIR=/opt/intel/compilers_and_libraries_2016.1.150/linux/mpi/intel64
      NETCDF=/usr/local/mic
      HDFDIR=/usr/local/mic
      ZLIB_DIR=/usr/local/mic
      PARMETISDIR=/usr/local/mic
      HWLOC_PATH=/usr/local/mic
      INTEL_DIR=/opt/intel/compilers_and_libraries_2016.1.150/linux/mkl/lib/mic
      
      cmake \
          -Wno-dev \
          -D Trilinos_CONFIGURE_OPTIONS_FILE:FILEPATH=$TRILINOS_HOME/sampleScripts/AlbanySettings.cmake \
          -D Trilinos_ENABLE_SCOREC:BOOL=ON \
          -D SCOREC_DISABLE_STRONG_WARNINGS:BOOL=ON \
          -D CMAKE_BUILD_TYPE:STRING=RELEASE \
          -D TPL_FIND_SHARED_LIBS:BOOL=OFF \
          -D BUILD_SHARED_LIBS:BOOL=OFF \
          -D TPL_ENABLE_MPI:BOOL=ON \
          -D MPI_BASE_DIR:PATH="$MPI_BASE_DIR"\
          -D CMAKE_CXX_COMPILER:FILEPATH="${MPI_BASE_DIR}/bin/mpiicpc" \
          -D CMAKE_C_COMPILER:FILEPATH="${MPI_BASE_DIR}/bin/mpiicc" \
          -D CMAKE_Fortran_COMPILER:FILEPATH="${MPI_BASE_DIR}/bin/mpiifort" \
          -D CMAKE_AR:FILEPATH=/opt/intel/compilers_and_libraries_2016.1.150/linux/bin/intel64_mic/xiar \
          -D CMAKE_LINKER:FILEPATH=/opt/intel/compilers_and_libraries_2016.1.150/linux/bin/intel64_mic/xild \
          -D CMAKE_CXX_FLAGS:STRING="-O3 -mmic -DMPICH_IGNORE_CXX_SEEK -DMPICH_SKIP_MPICXX -DPREC_TIMER -restrict -fasm-blocks -DDEVICE=1wq  -fopenmp" \
          -D CMAKE_C_FLAGS:STRING="-O3 -mmic -DMPICH_IGNORE_CXX_SEEK -DMPICH_SKIP_MPICXX -DPREC_TIMER -restrict -fasm-blocks -DDEVICE=1wq  -fopenmp" \
          -D CMAKE_Fortran_FLAGS:STRING="-O3 -mmic -DPREC_TIMER -fopenmp" \
          -D CMAKE_VERBOSE_MAKEFILE:BOOL=OFF \
          -D Trilinos_ENABLE_ALL_PACKAGES:BOOL=OFF \
          -D Trilinos_WARNINGS_AS_ERRORS_FLAGS:STRING="" \
          -D Trilinos_ENABLE_Teuchos:BOOL=ON \
         \
          -D Trilinos_WARNINGS_AS_ERRORS_FLAGS:STRING="" \
          -D Trilinos_ENABLE_ALL_PACKAGES:BOOL=OFF \
          -D Trilinos_ENABLE_ALL_OPTIONAL_PACKAGES:BOOL=OFF \
         \
          -D Trilinos_ENABLE_SEACASNemesis:BOOL=ON \
          -D Trilinos_ENABLE_Zoltan:BOOL=ON \
          -D Zoltan_ENABLE_METIS:BOOL=OFF \
          -D Trilinos_ENABLE_Kokkos:BOOL=ON \
          -D Trilinos_ENABLE_KokkosCore:BOOL=ON \
          -D Trilinos_ENABLE_KokkosContainers:BOOL=ON \
          -D Trilinos_ENABLE_KokkosExample:BOOL=OFF \
          -D Trilinos_EXTRA_LINK_FLAGS:STRING="$NETCDF/lib/libnetcdf.a;$HDFDIR/lib/libhdf5_hl.a;$HDFDIR/lib/libhdf5.a;$ZLIB_DIR/lib/libz.a" \
         \
          -D Kokkos_ENABLE_Serial:BOOL=OFF \
          -D Kokkos_ENABLE_OpenMP:BOOL=ON \
          -D Kokkos_ENABLE_Pthread:BOOL=OFF \
          -D Kokkos_ENABLE_Cuda:BOOL=OFF \
          -D Trilinos_ENABLE_OpenMP:BOOL=ON \
          -D TPL_ENABLE_CUDA:BOOL=OFF \
          -D TPL_ENABLE_CUSPARSE:BOOL=OFF \
      \
          -D TPL_ENABLE_Pthread:BOOL=OFF \
          -D TPL_ENABLE_HWLOC:BOOL=OFF \
      \
          -D TPL_ENABLE_Boost:BOOL=ON \
          -D TPL_ENABLE_BoostLib:BOOL=ON \
          -D TPL_ENABLE_BoostAlbLib:BOOL=ON \
          -D Boost_INCLUDE_DIRS:PATH=$BOOST_DIR/include \
          -D Boost_LIBRARY_DIRS:PATH=$BOOST_DIR/lib \
          -D BoostLib_INCLUDE_DIRS:PATH=$BOOST_DIR/include \
          -D BoostLib_LIBRARY_DIRS:PATH=$BOOST_DIR/lib \
          -D BoostAlbLib_INCLUDE_DIRS:PATH=$BOOST_DIR/include \
          -D BoostAlbLib_LIBRARY_DIRS:PATH=$BOOST_DIR/lib \
      \
          -D TPL_ENABLE_Netcdf:STRING=ON \
          -D TPL_Netcdf_PARALLEL:BOOL=ON \
          -D Netcdf_INCLUDE_DIRS:PATH=$NETCDF/include \
          -D TPL_Netcdf_LIBRARIES:FILEPATH="$NETCDF/lib/libnetcdf.a;$HDFDIR/lib/libhdf5_hl.a;$HDFDIR/lib/libhdf5.a;$ZLIB_DIR/lib/libz.a" \
          -D TPL_ENABLE_Pnetcdf:STRING=ON \
          -D Pnetcdf_INCLUDE_DIRS:PATH=$NETCDF/include \
          -D Pnetcdf_LIBRARY_DIRS:PATH=$NETCDF/lib \
      \
          -D TPL_ENABLE_HDF5:STRING=ON \
          -D HDF5_INCLUDE_DIRS:PATH=$HDFDIR/include \
          -D HDF5_LIBRARY_DIRS:PATH="$HDFDIR/lib;$ZLIB_DIR/lib" \
      \
          -D TPL_ENABLE_Zlib:STRING=ON \
          -D Zlib_INCLUDE_DIRS:PATH=$ZLIB_DIR/include \
          -D Zlib_LIBRARY_DIRS:PATH=$ZLIB_DIR/lib \
      \
          -D TPL_ENABLE_BLAS:BOOL=ON \
          -D TPL_ENABLE_LAPACK:BOOL=ON \
          -D BLAS_LIBRARY_DIRS:FILEPATH=${INTEL_DIR} \
          -D TPL_BLAS_LIBRARIES:STRING="${INTEL_DIR}/libmkl_intel_lp64.a;${INTEL_DIR}/libmkl_sequential.a;${INTEL_DIR}/libmkl_core.a" \
          -D LAPACK_LIBRARY_NAMES:STRING= \
      \
          -D CMAKE_INSTALL_PREFIX:PATH=$INSTALL_DIR \
          -D TPL_ENABLE_X11=OFF \
          -D FC_FN_CASE=LOWER \
          -D FC_FN_UNDERSCORE=UNDER  \
          -D Trilinos_ENABLE_MueLu:BOOL=OFF \
      	-D TPL_FIND_SHARED_LIBS:BOOL=OFF \
      \
      $EXTRA_ARGS \
      ${TRILINOS_HOME}
      
      exit 0
    
  • Build Trilinos

      make -j 8
      make install
    

Build Albany

  • Clone the repo

      cd /usr/local/src
      git clone git@github.com:gahansen/Albany.git
      cd Albany
    
  • Configure the Albany build

      mkdir build
      cd build
      ./do-configure
    
  • CMake configure script ./do-configure:

      #!/bin/sh
      
      export ALB_64BIT_INT=OFF
      export ALB_EPETRA=ON
      export ALB_SEE=OFF
      export ALB_ENABLE_SCOREC=ON
      export ALB_ENABLE_LCM=ON
      export ALB_ENABLE_LAME=ON
      export ALB_ENABLE_HYDRIDE=OFF
      export ALB_ENABLE_QCAD=OFF
      export ALB_ENABLE_MOR=OFF
      export ALB_ENABLE_ASCR=OFF
      export ALB_ENABLE_AERAS=OFF
      export ALB_ENABLE_FELIX=OFF
      export ALB_ENABLE_AlbanyCI=OFF
      export ALB_DEBUG=OFF
      export ALB_ENABLE_CHECK_FPE=OFF
      export ALB_ENABLE_SPECULATIVE=OFF
      export ALB_ENABLE_SG_MP=OFF
      export ALB_ENABLE_PERF_TESTS=ON
      export TRILINOS_INSTALL_DIR=/usr/local/host/trilinos/MPI_MIC
      export ALB_VERBOSE=OFF
      
      cmake \
            -D ALBANY_TRILINOS_DIR:FILEPATH="$TRILINOS_INSTALL_DIR" \
            -D ENABLE_LCM:BOOL=${ALB_ENABLE_LCM} \
            -D ENABLE_AERAS:BOOL=${ALB_ENABLE_AERAS} \
            -D ENABLE_QCAD:BOOL=${ALB_ENABLE_QCAD} \
            -D ENABLE_HYDRIDE:BOOL=${ALB_ENABLE_HYDRIDE} \
            -D ENABLE_LCM_SPECULATIVE:BOOL=${ALB_ENABLE_SPECULATIVE} \
            -D ENABLE_LAME:BOOL=${ALB_ENABLE_LAME} \
            -D CMAKE_VERBOSE_MAKEFILE:BOOL=${ALB_VERBOSE} \
            -D ENABLE_DEBUGGING:BOOL=${ALB_DEBUG} \
            -D ENABLE_CHECK_FPE:BOOL=${ALB_ENABLE_CHECK_FPE} \
            -D ENABLE_SCOREC:BOOL=${ALB_ENABLE_SCOREC} \
            -D ENABLE_FELIX:BOOL=${ALB_ENABLE_FELIX} \
            -D ENABLE_MOR:BOOL=${ALB_ENABLE_MOR} \
            -D ENABLE_ALBANY_CI:BOOL=${ALB_ENABLE_AlbanyCI} \
            -D ENABLE_ASCR:BOOL=${ALB_ENABLE_ASCR} \
            -D ENABLE_SG_MP:BOOL=${ALB_ENABLE_SG_MP} \
            -D ENABLE_PERFORMANCE_TESTS:BOOL=${ALB_ENABLE_PERF_TESTS} \
            -D ALBANY_CTEST_TIMEOUT:INTEGER=70 \
            -D ENABLE_64BIT_INT:BOOL=${ALB_64BIT_INT} \
            -D ENABLE_ALBANY_EPETRA_EXE:BOOL=${ALB_EPETRA} \
            -D ENABLE_SEE:BOOL=${ALB_SEE} \
            -D ALBANY_CXX_FLAGS:STRING=${ALB_FLAGS} \
      	  -D ENABLE_KOKKOS_UNDER_DEVELOPMENT:BOOL=ON \
      	  -D ENABLE_CROSS_COMPILE:BOOL=ON \
      	  -D ALBANY_MPI_OPTIONS:BOOL=ON \
      	  -D ALBANY_MPI_EXEC:STRING=${MPI_BASE_DIR}/bin/mpiexec.hydra \
      	  -D ALBANY_MPI_EXEC_NUMPROCS_FLAG:STRING=-n \
      	  -D ALBANY_MPI_EXEC_MAX_NUMPROCS:STRING=4 \
      	  -D ALBANY_MPI_TRAILING_OPTIONS:STRING='-hosts mic1 -ppn 4 -env OMP_NUM_THREADS 56 -env KMP_AFFINITY balanced -binding domain=omp -env LD_LIBRARY_PATH ${INTEL_DIR}' \
      \
      ..
    
  • Build Albany

      make -j 8
      make install
    
  • Test Albany

      ctest
    
Clone this wiki locally
You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.
Press h to open a hovercard with more details.