T64 Debian using Gcc

Glen Hansen edited this page Apr 16, 2016 · 7 revisions

This set of build instructions builds the tpetra branch of Albany in 64 bit mode, in other words with support of a > 2.1 billion word address space to handle problems with very large numbers of degrees of freedom.

This is the avatar server at RPI:

Linux avatar 2.6.32-5-amd64 #1 SMP Mon Feb 29 00:33:51 UTC 2016 x86_64 GNU/Linux

The compilers are gcc 4.8.4, OpenMPI 1.8.3, and Debian linux.

Build supporting packages

Trilinos requires a set of "third party libraries" (TPLs) to support Albany. The actual libraries needed depends on the final Albany configuration desired. These instructions build a superset, all the libraries needed to support any Albany configuration.

Notes: In this example

  • The TPLs are installed in /users/ghansen

Building ZLib

  • Version 1.2.8

      cd /fasttmp/ghansen
      wget http://zlib.net/zlib-1.2.8.tar.gz
      tar -xvf zlib-1.2.8.tar.gz
      cd zlib-1.2.8
      mkdir build
      cd build
      cmake \
      -DCMAKE_C_COMPILER=mpicc \
      -DCMAKE_C_FLAGS="-O3" \
      -DCMAKE_INSTALL_PREFIX=/users/ghansen \
      ..
    
      make -j 8
      make install
    

Building HDF5

  • Version 1.8.16

      tar -xvf hdf5-1.8.16.tar.gz
      cd hdf5-1.8.16
      ./configure --prefix=/users/ghansen CC=/users/ghansen/ompi-gcc/bin/mpicc \
      CXX=/users/ghansen/ompi-gcc/bin/mpicxx FC=/users/ghansen/ompi-gcc/bin/mpif90 \
      CXXFLAGS="-fPIC -O3 -march=native" CFLAGS="-fPIC -O3 -march=native" FFLAGS="-fPIC -O3 -march=native" \
      --enable-parallel --enable-shared --with-zlib=/users/ghansen
      make -j 4
      make install
    

Building parallel-netcdf

  • Version 1.7.0

  • Configure and build:

      ./configure --prefix=/users/ghansen CC=/users/ghansen/ompi-gcc/bin/mpicc \
      FC=/users/ghansen/ompi-gcc/bin/mpif90 CXX=/users/ghansen/ompi-gcc/bin/mpicxx \
      CXXFLAGS="-O3 -march=native -fPIC" CFLAGS="-O3 -march=native -fPIC" FFLAGS="-O3 -march=native -fPIC"
      make -j 8 
      make install
    

Building Netcdf

  • Version 4.4.0

  • Edit the file include/netcdf.h. Make the following edits near line 265:

      #define NC_MAX_DIMS    65536    /* max dimensions per file */
      #define NC_MAX_ATTRS    8192    
      #define NC_MAX_VARS   524288    /* max variables per file */
      #define NC_MAX_NAME   256 
      #define NC_MAX_VAR_DIMS      8     /* max per variable dimensions */
    
  • Now, configure and build:

      ./configure --prefix=/users/ghansen CC=/users/ghansen/ompi-gcc/bin/mpicc \
      FC=/users/ghansen/ompi-gcc/bin/mpif90 CXX=/users/ghansen/ompi-gcc/bin/mpicxx \
      CXXFLAGS="-I/users/ghansen/include -O3 -march=native -fPIC" \
      CFLAGS="-I/users/ghansen/include -O3 -march=native -fPIC" LDFLAGS="-L/users/ghansen/lib -O3 -march=native -fPIC" \
      --disable-fsync --enable-shared --disable-doxygen --enable-netcdf-4 --enable-pnetcdf
      make -j 4 
      make install
    

Building boost

  • Version 1.60.0

  • Build boost with the commands

      rm ~/user-config.jam
      echo "using gcc : 4.8.4 : /users/ghansen/bin/g++ ;" >> ~/user-config.jam
      echo "using mpi : /users/ghansen/ompi-gcc/bin/mpicxx ;" >> ~/user-config.jam
      ./bootstrap.sh --with-libraries=signals,regex,filesystem,system,mpi,serialization,thread,program_options,exception --prefix=/users/ghansen
      ./b2 -j 4
      ./b2 -j 4 install
    

Building ParMetis

  • Version 4.0.3

      cd /fasttmp/ghansen
      wget http://glaros.dtc.umn.edu/gkhome/fetch/sw/parmetis/parmetis-4.0.3.tar.gz
      tar -xvf parmetis-4.0.3.tar.gz
      cd parmetis-4.0.3
    
  • Edit metis.h to use an IDXTYPEWIDTH of 64

      cd metis/include
      vi metis.h
      #define IDXTYPEWIDTH 64
    
  • Run CMake to build parmetis

      cd /fasttmp/ghansen/parmetis-4.0.3
      mkdir build2
      cd build2
      export metis=/fasttmp/ghansen/parmetis-4.0.3/metis
      cmake \
      -DCMAKE_INSTALL_PREFIX=/users/ghansen \
      -DMETIS_PATH=$metis \
      -DGKLIB_PATH=$metis/GKlib \
      -DCMAKE_C_COMPILER=mpicc \
      -DCMAKE_CXX_COMPILER=mpicxx \
      ..
      make -j 4
      make install
    
      cp libmetis/libmetis.a /users/ghansen/lib
      cp $metis/include/metis.h /users/ghansen/include
    

SuperLU

  • Version 4.3

      wget http://crd-legacy.lbl.gov/~xiaoye/SuperLU/superlu_4.3.tar.gz
      tar -xvf superlu_4.3.tar.gz
      cd SuperLU_4.3
    
  • Build paths

      mkdir /users/ghansen/SuperLU_4.3
      mkdir /users/ghansen/SuperLU_4.3/include
      mkdir /users/ghansen/SuperLU_4.3/lib
    
  • Edit make.inc for your machine and environment

      PLAT = _linux
      SuperLUroot = /users/ghansen/SuperLU_4.3
      SUPERLULIB      = $(SuperLUroot)/lib/libsuperlu_4.3.a
      TMGLIB          = libtmglib.a
      BLASDEF     = -DUSE_VENDOR_BLAS
      BLASLIB     = -L/usr/lib -lblas
      LIBS        = $(SUPERLULIB) $(BLASLIB)
      ARCH         = ar
      ARCHFLAGS    = cr
      RANLIB       = ranlib
      CC           = gcc
      CFLAGS       = -O3
      NOOPTS       =
      FORTRAN      = gfortran
      FFLAGS       = -O3
      LOADER       = $(CC)
      LOADOPTS     =
      CDEFS        = -DAdd_
      MATLAB       = /usr/sww/matlab
    
  • Build SuperLU

      make -j 4
      cd SRC
      cp *.h /users/ghansen/SuperLU_4.3/include		
    

Build HWLOC

  • HWLOC is an optional package that can be used for thread pinning by Kokkos. If you are not building Kokkos, or if you don't care about thread pinning, please skip.

  • Version 1.9

      wget https://www.open-mpi.org/software/hwloc/v1.9/downloads/hwloc-1.9.tar.gz
      tar -xvf hwloc-1.9.tar.gz
      cd hwloc-1.9
      ./configure CC=mpicc CXX=mpicxx --prefix=/users/ghansen
      make -j 4
      make install
    

Build Trilinos

  • do-configure script

      #!/bin/sh
      #
      export TRILINOS_HOME=/lore/ghansen/Trilinos/publicTrilinos
      BUILD_DIR=`pwd`
      INSTALL_DIR=/users/ghansen/Trilinos/MPI_REL
      LIB_DIR=/users/ghansen
      MPI_BASE_DIR=/users/ghansen
      SUPERLUDIR=${LIB_DIR}/SuperLU_4.3
      
      export BOOST_ROOT=$BOOST_DIR
      EXTRA_ARGS=$@
      cmake \
          -D Trilinos_CONFIGURE_OPTIONS_FILE:FILEPATH=$TRILINOS_HOME/sampleScripts/AlbanySettings.cmake \
          -D CMAKE_BUILD_TYPE:STRING=RELEASE \
          -D Trilinos_EXTRA_REPOSITORIES:STRING=SCOREC \
          -D Trilinos_ENABLE_SCOREC:BOOL=ON \
          -D SCOREC_DISABLE_STRONG_WARNINGS:BOOL=ON \
      	-D Trilinos_ENABLE_SCORECpumi_geom_parasolid:BOOL=ON \
      \
      	-D TPL_ENABLE_Parasolid:BOOL=ON \
      	-D Parasolid_INCLUDE_DIRS:PATH="/usr/local/parasolid/25.1.181" \
      	-D Parasolid_LIBRARY_DIRS:PATH="/usr/local/parasolid/25.1.181/shared_object" \
      \
          -D TPL_ENABLE_MPI:BOOL=ON \
          -D MPI_BASE_DIR:PATH=$MPI_BASE_DIR \
      \
          -D Trilinos_ENABLE_Kokkos:BOOL=ON \
          -D Trilinos_ENABLE_KokkosCore:BOOL=ON \
          -D Trilinos_ENABLE_KokkosContainers:BOOL=ON \
          -D Trilinos_ENABLE_KokkosLinAlg:BOOL=ON \
          -D Trilinos_ENABLE_KokkosCompat:BOOL=ON \
      \
          -D Boost_INCLUDE_DIRS:PATH=$LIB_DIR/include \
          -D BoostLib_INCLUDE_DIRS:PATH=$LIB_DIR/include \
          -D BoostAlbLib_INCLUDE_DIRS:PATH=$LIB_DIR/include \
          -D Boost_LIBRARY_DIRS:PATH=$LIB_DIR/lib \
          -D BoostLib_LIBRARY_DIRS:PATH=$LIB_DIR/lib \
          -D BoostAlbLib_LIBRARY_DIRS:PATH=$LIB_DIR/lib \
      \
          -D TPL_ENABLE_Netcdf:STRING=ON \
          -D Netcdf_INCLUDE_DIRS:PATH="${LIB_DIR}/include" \
          -D Netcdf_LIBRARY_DIRS:PATH="${LIB_DIR}/lib" \
          -D TPL_ENABLE_HDF5:STRING=ON \
          -D HDF5_INCLUDE_DIRS:PATH="${LIB_DIR}/include" \
          -D TPL_HDF5_LIBRARIES:PATH="${LIB_DIR}/lib/libnetcdf.a;${LIB_DIR}/lib/libhdf5_hl.a;${LIB_DIR}/lib/libhdf5.a;${LIB_DIR}/lib/libz.a" \
          -D TPL_ENABLE_Zlib:STRING=ON \
          -D Zlib_INCLUDE_DIRS:PATH="${LIB_DIR}/include" \
          -D TPL_Zlib_LIBRARIES:PATH="${LIB_DIR}/lib/libz.a" \
      \
          -D TPL_ENABLE_ParMETIS:STRING=ON \
          -D ParMETIS_INCLUDE_DIRS:PATH="${LIB_DIR}/include" \
          -D ParMETIS_LIBRARY_DIRS:PATH="${LIB_DIR}/lib" \
      \
          -D TPL_ENABLE_SuperLU:STRING=ON \
          -D SuperLU_INCLUDE_DIRS:STRING="${SUPERLUDIR}/include" \
          -D SuperLU_LIBRARY_DIRS:STRING="${SUPERLUDIR}/lib" \
      \
          -D Trilinos_ASSERT_MISSING_PACKAGES=OFF \
          -D CMAKE_INSTALL_PREFIX:PATH=${INSTALL_DIR} \
      \
      $EXTRA_ARGS \
      ${TRILINOS_HOME}
    

NOTE: If you use the MKL libraries, please make sure you use the 32 bit Intel library mkl_intel_lp64, NOT the 64 bit version mkl_intel_ilp64. Both Lapack and Blas functions are called within Albany's local (on rank) address space, which is the local ordinal (LO) size, typically int.

To build Trilinos, use an out-of-source build. I usually make a build directory that matches the configuration of Trilinos that I am building in the Trilinos root directory:

    mkdir /lore/ghansen/tri_build
	cd /lore/ghansen/tri_build
	mkdir MPI_REL
  • Place the above do-configure file into the MPI_REL directory, change directories into MPI_REL, and perform the configure, build, and install:

      cd MPI_REL
      ./do-configure
      make -j 4
      make -j 4 install
    

Build Albany

  • CMake configure script:

      #!/bin/sh
      
      export ALB_64BIT_INT=ON
      export ALB_ENABLE_SCOREC=ON
      export ALB_ENABLE_LCM=ON
      export ALB_ENABLE_LAME=ON
      export ALB_ENABLE_HYDRIDE=ON
      export ALB_ENABLE_QCAD=ON
      export ALB_ENABLE_MOR=ON
      export ALB_ENABLE_ASCR=ON
      export ALB_ENABLE_AERAS=ON
      export ALB_ENABLE_FELIX=ON
      export ALB_ENABLE_AlbanyCI=OFF
      export ALB_DEBUG=OFF
      export ALB_ENABLE_CHECK_FPE=OFF
      export ALB_ENABLE_SPECULATIVE=OFF
      export ALB_ENABLE_SG_MP=ON
      export ALB_ENABLE_PERF_TESTS=ON
      export TRILINOS_INSTALL_DIR=/users/ghansen/Trilinos/MPI_REL
      
      export ALB_VERBOSE=OFF
      
      cmake \
            -D ALBANY_TRILINOS_DIR:FILEPATH="$TRILINOS_INSTALL_DIR" \
            -D ENABLE_LCM:BOOL=${ALB_ENABLE_LCM} \
            -D ENABLE_AERAS:BOOL=${ALB_ENABLE_AERAS} \
            -D ENABLE_QCAD:BOOL=${ALB_ENABLE_QCAD} \
            -D ENABLE_HYDRIDE:BOOL=${ALB_ENABLE_HYDRIDE} \
            -D ENABLE_LCM_SPECULATIVE:BOOL=${ALB_ENABLE_SPECULATIVE} \
            -D ENABLE_LAME:BOOL=${ALB_ENABLE_LAME} \
            -D CMAKE_VERBOSE_MAKEFILE:BOOL=${ALB_VERBOSE} \
            -D ENABLE_DEBUGGING:BOOL=${ALB_DEBUG} \
            -D ENABLE_CHECK_FPE:BOOL=${ALB_ENABLE_CHECK_FPE} \
            -D ENABLE_SCOREC:BOOL=${ALB_ENABLE_SCOREC} \
            -D ENABLE_FELIX:BOOL=${ALB_ENABLE_FELIX} \
            -D ENABLE_MOR:BOOL=${ALB_ENABLE_MOR} \
            -D ENABLE_ALBANY_CI:BOOL=${ALB_ENABLE_AlbanyCI} \
            -D ENABLE_ASCR:BOOL=${ALB_ENABLE_ASCR} \
            -D ENABLE_SG_MP:BOOL=${ALB_ENABLE_SG_MP} \
            -D ENABLE_PERFORMANCE_TESTS:BOOL=${ALB_ENABLE_PERF_TESTS} \
            -D ENABLE_64BIT_INT:BOOL=${ALB_64BIT_INT} \
            -D ALBANY_CTEST_TIMEOUT:INTEGER=70 \
      \
      ..
    

Clone the Albany repo in a suitable place:

	cd /users/ghansen
	git clone https://github.com/gahansen/Albany.git
  • Create an out-of-source build directory

      cd Albany
      mkdir build
    
  • Place the above configure script into the build directory, then configure, build, and test

      cd build
      ./buildAlbany
      make -j 4
      ctest
    
Clone this wiki locally
You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.
Press h to open a hovercard with more details.