ALCF Vesta

Glen Hansen edited this page Jul 30, 2016 · 23 revisions

Instructions for 2016

My .soft file looks like this:

@default
+cmake
+bgqtoolchain-gcc472
+mpiwrapper-gcc

Note that the +bgqtoolchain-gcc472 line does not appear to set the toolchain, so I also placed the line:

soft add +bgqtoolchain-gcc472

in my .bash_profile

Building Trilinos

The Trilinos configure script used for the tutorials this year is committed to the Albany repository at:

doc/do-cmake-trilinos-vesta

Note that this build uses LAPACK 3.6.1. This version has deprecated the calls dggsvd and sggsvd. Please edit the Trilinos file packages/epetra/src/Epetra_LAPACK_wrappers.h and change the lines:

#define DGGSVD_F77  F77_BLAS_MANGLE(dggsvd3,DGGSVD3)
#define SGGSVD_F77  F77_BLAS_MANGLE(sggsvd3,SGGSVD3)

Building Albany

The Albany configure script used for the tutorials this year is committed to the Albany repository at:

doc/do-cmake-albany-vesta

Running the Albany regression test suite

  • Note that the above Albany configure script also sets up ctest to run the regression suite on BG/Q, employing the script /projects/FASTMath/ATPESC-2016/installs/albany/bin/run_Albany to dispatch each parallel test to the compute nodes while ctest itself runs on the frontend node. From the build directory on the frontend node, typing

      qsub -A ATPESC16_Instructors -n 4 -t 60 --mode script /projects/FASTMath/ATPESC-2016/installs/albany/bin/reserve.sh
      qstat
    
  • Get the blockid info from qstat, and now run ctest

      export MPIEXEC_BLOCKID=VST-02400-13511-32
      ctest
    
  • will run the regression suite in this manner. Note also that all the usual ctest options apply to this command.

Instructions for 2015

Add this to your .soft file:

+mpiwrapper-bgclang
+cmake-3.0.2

Configure Trilinos as follows. (The block starting with Trilinos_ENABLE_Zoltan is for another user of this Trilinos build and can be omitted in a build meant only for Albany.)

# July 2015.
#   Need cmake:
#     => Use this line in your .soft file:
#          +cmake-3.0.2
#   C++11 requirement in Trilinos, especially in Kokkos:
#     => Use bgclang.
#     => Use this line in your .soft file:
#          +mpiwrapper-bgclang
#   However, (i) bgclang has no Fortran equivalent, and (ii) XL's
# binaries appear not to be compatible with bgclang's:
#     => Use the CMAKE_Fortran_COMPILER flag to point to gcc's Fortran compiler.

# Install next to the build directory.
TRILINSTALLDIR=../install-atpesc

# These TPLs are already built for us.
NETCDFDIR=/projects/FASTMath/install/netcdf/4.3.0/linux-rhel_6-ppc64-gcc-4.4.6
HDF5DIR=/projects/FASTMath/install/hdf5/1.8.11/linux-rhel_6-ppc64-gcc-4.4.6
ZLIBDIR=/projects/FASTMath/install/zlib/1.2.8/linux-rhel_6-ppc64-gcc-4.4.6
BLASDIR=/projects/FASTMath/install/blas/19Apr11/linux-rhel_6-ppc64-gcc-4.4.6
LAPACKDIR=/projects/FASTMath/install/lapack/3.4.2/linux-rhel_6-ppc64-gcc-4.4.6
BOOSTDIR=/projects/FASTMath/install/boost/1.54.0/linux-rhel_6-ppc64-gcc-4.4.6
# These are not.
MYTPLDIR=/projects/FASTMath/ATPESC-2015/install/Trilinos/tpl

# Make Trilinos choose the .a rather than .so files.
BOOST_LIBS="$BOOSTDIR/lib/libboost_program_options.a;$BOOSTDIR/lib/libboost_system.a"

# netcdf and hdf5 ordering are wrong for static linking. Fix with the
# usual hack of appending these as extra link flags. List libz.a
# specifically so libz.so is not used.
EXTRA_LINK_FLAGS="${HDF5DIR}/lib/libhdf5_hl.a;${HDF5DIR}/lib/libhdf5.a;${ZLIBDIR}/lib/libz.a"

# Turn on for a 64-bit local ordinal.
HAVE_LL=OFF

rm -f CMakeCache.txt;

cmake \
    -D Trilinos_DISABLE_ENABLED_FORWARD_DEP_PACKAGES=ON \
    -D CMAKE_INSTALL_PREFIX:PATH=${TRILINSTALLDIR} \
    -D CMAKE_BUILD_TYPE:STRING="RELEASE" \
    -D TPL_ENABLE_MPI:BOOL=ON \
    -D CMAKE_VERBOSE_MAKEFILE:BOOL=OFF \
    -D BUILD_SHARED_LIBS:BOOL=OFF \
    -D Trilinos_ENABLE_SECONDARY_TESTED_CODE:BOOL=OFF \
    \
    -D Trilinos_ENABLE_ALL_PACKAGES:BOOL=OFF \
    -D Trilinos_WARNINGS_AS_ERRORS_FLAGS:STRING="" \
    -D Trilinos_ENABLE_Teuchos:BOOL=ON \
    -D Trilinos_ENABLE_Shards:BOOL=ON \
    -D Trilinos_ENABLE_Sacado:BOOL=ON \
    -D Trilinos_ENABLE_Epetra:BOOL=ON \
    -D Trilinos_ENABLE_EpetraExt:BOOL=ON \
    -D Trilinos_ENABLE_Ifpack:BOOL=ON \
    -D Trilinos_ENABLE_AztecOO:BOOL=ON \
    -D Trilinos_ENABLE_Amesos:BOOL=ON \
    -D Trilinos_ENABLE_Anasazi:BOOL=ON \
    -D Trilinos_ENABLE_Belos:BOOL=ON \
    -D Trilinos_ENABLE_ML:BOOL=ON \
    -D Trilinos_ENABLE_Phalanx:BOOL=ON \
    -D Trilinos_ENABLE_Intrepid:BOOL=ON \
    -D Trilinos_ENABLE_NOX:BOOL=ON \
    -D Trilinos_ENABLE_Stratimikos:BOOL=ON \
    -D Trilinos_ENABLE_Thyra:BOOL=ON \
    -D Trilinos_ENABLE_Rythmos:BOOL=ON \
    -D Trilinos_ENABLE_MOOCHO:BOOL=ON \
    -D Trilinos_ENABLE_Stokhos:BOOL=ON \
    -D Trilinos_ENABLE_Piro:BOOL=ON \
    -D Trilinos_ENABLE_Teko:BOOL=ON \
    -D Trilinos_ENABLE_Moertel:BOOL=ON \
    \
    -D Trilinos_ENABLE_STKIO:BOOL=ON \
    -D Trilinos_ENABLE_STKMesh:BOOL=ON \
    -D TPL_ENABLE_Boost:BOOL=ON \
    -D Boost_INCLUDE_DIRS:FILEPATH="$BOOSTDIR/include" \
    -D TPL_Boost_LIBRARIES=${BOOST_LIBS} \
    -D TPL_ENABLE_BoostLib:BOOL=ON \
    -D BoostLib_INCLUDE_DIRS:FILEPATH="$BOOSTDIR/include" \
    -D TPL_BoostLib_LIBRARIES=${BOOST_LIBS} \
    \
    -D Trilinos_ENABLE_SEACAS:BOOL=ON \
    -D TPL_ENABLE_X11:BOOL=OFF \
    -D TPL_ENABLE_Matio:BOOL=OFF \
    -D Trilinos_ENABLE_SEACASIoss:BOOL=ON \
    -D Trilinos_ENABLE_SEACASExodus:BOOL=ON \
    \
    -D TPL_ENABLE_Netcdf:BOOL=ON \
    -D Netcdf_INCLUDE_DIRS:PATH="$NETCDFDIR/include" \
    -D TPL_Netcdf_LIBRARIES="$NETCDFDIR/lib/libnetcdf.a" \
    -D TPL_ENABLE_HDF5:BOOL=ON \
    -D TPL_HDF5_LIBRARIES="$HDF5DIR/lib/libhdf5_hl.a;$HDF5DIR/lib/libhdf5.a;$ZLIBDIR/lib/libz.a" \
    -D HDF5_INCLUDE_DIRS:PATH="$HDF5DIR/include" \
    \
    -D Trilinos_ENABLE_Tpetra:BOOL=ON \
    -D Trilinos_ENABLE_Kokkos:BOOL=ON \
    -D Trilinos_ENABLE_Ifpack2:BOOL=ON \
    -D Trilinos_ENABLE_Amesos2:BOOL=ON \
    -D Trilinos_ENABLE_Zoltan2:BOOL=ON \
    -D Trilinos_ENABLE_MueLu:BOOL=ON \
    -D Amesos2_ENABLE_KLU2:BOOL=ON \
    -D Amesos2_ENABLE_Basker:BOOL=ON \
    \
    -D Trilinos_ENABLE_SCOREC:BOOL=ON \
    -D SCOREC_DISABLE_STRONG_WARNINGS:BOOL=ON \
    -D Trilinos_ENABLE_EXPORT_MAKEFILES:BOOL=OFF \
    -D Trilinos_ASSERT_MISSING_PACKAGES:BOOL=OFF \
    -D MDS_ID_TYPE:STRING="int" \
    \
    -D TPL_ENABLE_ParMETIS:STRING=ON \
    -D ParMETIS_INCLUDE_DIRS:PATH="${MYTPLDIR}/include" \
    -D ParMETIS_LIBRARY_DIRS:PATH="${MYTPLDIR}/lib" \
    -D TPL_ENABLE_METIS:STRING=ON \
    -D METIS_INCLUDE_DIRS:PATH="${MYTPLDIR}/include" \
    -D METIS_LIBRARY_DIRS:PATH="${MYTPLDIR}/lib" \
    -D Zoltan_ENABLE_ULLONG_IDS:BOOL=ON \
    -D Teuchos_ENABLE_LONG_LONG_INT:BOOL=$HAVE_LL \
    \
    -D Trilinos_ENABLE_EXPLICIT_INSTANTIATION:BOOL=ON \
    -D Tpetra_INST_FLOAT=OFF \
    -D Tpetra_INST_INT_INT=ON \
    -D Tpetra_INST_DOUBLE=ON \
    -D Tpetra_INST_COMPLEX_FLOAT=OFF \
    -D Tpetra_INST_COMPLEX_DOUBLE=OFF \
    -D Tpetra_INST_INT_LONG=OFF \
    -D Tpetra_INST_INT_UNSIGNED=OFF \
    -D Tpetra_INST_INT_LONG_LONG=$HAVE_LL \
    \
    -D Trilinos_ENABLE_Kokkos:BOOL=ON \
    -D Trilinos_ENABLE_KokkosCore:BOOL=ON \
    -D Phalanx_KOKKOS_DEVICE_TYPE:STRING="SERIAL" \
    -D Phalanx_INDEX_SIZE_TYPE:STRING="INT" \
    -D Phalanx_SHOW_DEPRECATED_WARNINGS:BOOL=OFF \
    -D Kokkos_ENABLE_Serial:BOOL=ON \
    -D Kokkos_ENABLE_OpenMP:BOOL=OFF \
    -D Kokkos_ENABLE_Pthread:BOOL=OFF \
    \
    -D Trilinos_ENABLE_Zoltan:BOOL=ON \
    -D Zoltan_ENABLE_TESTS:BOOL=ON \
    -D Zoltan_ENABLE_EXAMPLES:BOOL=ON \
    -D Zoltan_ENABLE_F90INTERFACE:BOOL=ON \
    -D Zoltan_ENABLE_ParMETIS:BOOL=ON \
    -D Zoltan_ENABLE_Scotch:BOOL=ON \
    -D Scotch_LIBRARY_DIRS:FILEPATH="${MYTPLDIR}/lib" \
    -D Scotch_INCLUDE_DIRS:FILEPATH="${MYTPLDIR}/include" \
    \
    -D CMAKE_CXX_FLAGS:STRING="-Wno-deprecated-declarations -Wno-sign-compare" \
    -D TPL_BLAS_LIBRARIES="$BLASDIR/lib/libblas.a" \
    -D TPL_LAPACK_LIBRARIES="$LAPACKDIR/lib/liblapack.a;$LAPACKDIR/lib/libtmglib.a" \
    -D CMAKE_Fortran_COMPILER="/soft/compilers/wrappers/gcc/mpif90" \
    -D Trilinos_EXTRA_LINK_FLAGS:STRING=${EXTRA_LINK_FLAGS} \
    -D F77_FUNC:STRING="(name,NAME) name" \
    -D F77_FUNC_:STRING="(name,NAME) name" \
    \
    ../

Instructions for 2014

What follows are instructions for the 2014 ATPESC build.

Set up your build environment

Please place the following lines in your ~/.soft file:

	@default
	+hpctoolkit
	+cmake
	+mpiwrapper-gcc

Install the required Third Party Libraries (TPLs) for Trilinos

These install directions assume that a base install directory of /home/ghansen/FASTMath/src/ is used

Building ZLib

  • Version 1.2.8

      cd /home/ghansen/FASTMath/src/
      wget http://zlib.net/zlib-1.2.8.tar.gz
      tar -xvf zlib-1.2.8.tar.gz
      cd zlib-1.2.8
      CC=mpicc CXX=mpicxx CFLAGS=-O3 CXXFLAGS=-O3 ./configure --64 --static --prefix=/home/ghansen/FASTMath
      make install
    

Building HDF5

  • Version 1.8.13

      cd /home/ghansen/FASTMath/src/
      wget http://www.hdfgroup.org/ftp/HDF5/current/src/hdf5-1.8.13.tar.gz
      tar -xvf hdf5-1.8.13.tar.gz
      cd hdf5-1.8.13
      ./configure CC=mpicc FC=mpif90 CXX=mpicxx CXXFLAGS="-fPIC -O3" CFLAGS="-fPIC -O3" FCFLAGS="-fPIC -O3" \
      --enable-parallel --enable-shared=no --with-zlib=/home/ghansen/FASTMath --prefix=/home/ghansen/FASTMath
      make -j 8
      make install
    

Building Netcdf

  • Version of the day from the git repo. Note that hdf5-1.8.13 made some changes that are not reflected in the latest release, so we are using the votd for now.

      cd /home/ghansen/FASTMath/src/
      git clone https://github.com/Unidata/netcdf-c.git
      cd netcdf-c
    
  • Edit the file include/netcdf.h. Make the following edits near line 228:

      #define NC_MAX_DIMS    65536    /* max dimensions per file */
      #define NC_MAX_ATTRS    8192    
      #define NC_MAX_VARS   524288    /* max variables per file */
      #define NC_MAX_NAME   256 
      #define NC_MAX_VAR_DIMS      8     /* max per variable dimensions */
    
  • Now, configure using CMake and build:

      mkdir build
      cd build
      cmake \
      -DCMAKE_C_COMPILER=mpicc \
      -DCMAKE_C_FLAGS="-I/home/ghansen/FASTMath/include -O3" \
      -DCMAKE_EXE_LINKER_FLAGS="-L/home/ghansen/FASTMath/lib -O3" \
      -DCMAKE_INSTALL_PREFIX=/home/ghansen/FASTMath \
      -DENABLE_DAP=OFF -DBUILD_SHARED_LIBS=OFF -DENABLE_FSYNC=OFF -DENABLE_CDMREMOTE=OFF \
      -DENABLE_DOXYGEN=OFF -DENABLE_NETCDF_4=ON -DCMAKE_BUILD_TYPE=RELEASE -DENABLE_EXAMPLES=OFF \
      -DENABLE_TESTS=OFF -DCMAKE_SKIP_INSTALL_RPATH=ON \
      -DZLIB_INCLUDE_DIRS=/home/ghansen/FASTMath/include \
      -DZLIB_LIBRARY=/home/ghansen/FASTMath/lib/libz.a \
      -DBUILD_UTILITIES=OFF \
      ..
    
      make -j 8 
      make install
    

Building boost

  • Version 1.55.0

      cd /home/ghansen/FASTMath/src/
      wget http://sourceforge.net/projects/boost/files/boost/1.55.0/boost_1_55_0.tar.gz/download
      tar -xvf boost_1_55_0.tar.gz
      cd boost_1_55_0
    
  • Build boost with the commands

      echo "using gcc : 4.4.7 : /bgsys/drivers/ppcfloor/gnu-linux/bin/powerpc64-bgq-linux-g++ ;" \
      >> ./tools/build/v2/user-config.jam
      echo "using mpi : /bgsys/drivers/ppcfloor/comm/bin/gcc/mpicxx ;" >> ./tools/build/v2/user-config.jam
      ./bootstrap.sh \
      --with-libraries=signals,regex,filesystem,system,mpi,serialization,thread,program_options,exception \
      --prefix=/home/ghansen/FASTMath
      ./b2 -j 8
      ./b2 -j 8 install
    

Building ParMetis

  • Version 4.0.3

      cd /home/ghansen/FASTMath/src/
      wget http://glaros.dtc.umn.edu/gkhome/fetch/sw/parmetis/parmetis-4.0.3.tar.gz
      tar -xvf parmetis-4.0.3.tar.gz
      cd parmetis-4.0.3
    
  • Edit metis.h to use an IDXTYPEWIDTH of 64

      cd metis/include
      vi metis.h
      #define IDXTYPEWIDTH 64
    
  • Run CMake to build parmetis

      cd /home/ghansen/FASTMath/src/parmetis-4.0.3
      cd build
      export metis=/home/ghansen/FASTMath/src/parmetis-4.0.3/metis
    
      cmake \
      -DCMAKE_INSTALL_PREFIX=/home/ghansen/FASTMath \
      -DMETIS_PATH=$metis \
      -DGKLIB_PATH=$metis/GKlib \
      -DCMAKE_C_COMPILER=mpicc \
      -DCMAKE_CXX_COMPILER=mpicxx \
      ..
    
      make -j 8
      make install
    
      cp libmetis/libmetis.a /home/ghansen/FASTMath/lib
      cp $metis/include/metis.h /home/ghansen/FASTMath/include
    

SuperLU

  • Version 4.3

      cd /home/ghansen/FASTMath/src
      wget http://crd-legacy.lbl.gov/~xiaoye/SuperLU/superlu_4.3.tar.gz
      tar -xvf superlu_4.3.tar.gz
      cd superlu_4.3
    
      mkdir /home/ghansen/FASTMath/SuperLU_4.3
      mkdir /home/ghansen/FASTMath/SuperLU_4.3/include
      mkdir /home/ghansen/FASTMath/SuperLU_4.3/lib
    
  • Edit make.inc for your machine and environment

      PLAT = _linux
      SuperLUroot = /home/ghansen/FASTMath/SuperLU_4.3
      SUPERLULIB      = $(SuperLUroot)/lib/libsuperlu_4.3.a
      TMGLIB          = libtmglib.a
      BLASDEF     = -DUSE_VENDOR_BLAS
      BLASLIB     = /gpfs/vesta-fs0/projects/FASTMath/ATPESC-2014/install/blas/19Apr11/linux-rhel_6-ppc64-gcc-4.4.6/lib/blas.a
      LIBS        = $(SUPERLULIB) $(BLASLIB)
      ARCH         = ar
      ARCHFLAGS    = cr
      RANLIB       = ranlib
      CC           = mpicc
      CFLAGS       = -DPRNTlevel=0 -O3
      NOOPTS       =
      FORTRAN      = mpif90
      FFLAGS       = -O3
      LOADER       = $(CC)
      LOADOPTS     =
      CDEFS        = -DNoChange
      MATLAB       =
    
  • Build SuperLU

      make -j 4
      cd SRC
      cp *.h /home/ghansen/FASTMath/SuperLU_4.3/include
      ar d home/ghansen/FASTMath/SuperLU_4.3/lib/libsuperlu_4.3.a dlamch.o slamch.o
    
  • The last command deletes the dlamch and slamch objects, which already exist in the LAPACK library on Vesta

Build Trilinos

  • Download Trilinos

      cd /home/ghansen/FASTMath/src
      git clone https://software.sandia.gov/trilinos/repositories/publicTrilinos
    
  • Download SCOREC

      cd /home/ghansen/FASTMath/src/publicTrilinos
      git clone https://github.com/SCOREC/core.git SCOREC
    
  • Configure the Trilinos build

      mkdir build
      cd build
      ./do-configure
    
  • Example do-configure script executed above

      #!/bin/sh
      #
      export TRILINOS_HOME=/home/ghansen/FASTMath/src/publicTrilinos
      BUILD_DIR=`pwd`
      INSTALL_DIR=/home/ghansen/FASTMath/Trilinos/MPI_REL
      BOOST_DIR=/home/ghansen/FASTMath
      MPI_BASE_DIR=/bgsys/drivers/V1R2M1/ppc64/comm
      NETCDF=/home/ghansen/FASTMath
      HDFDIR=/home/ghansen/FASTMath
      PARMETISDIR=/home/ghansen/FASTMath
      BLAS=/gpfs/vesta-fs0/projects/FASTMath/ATPESC-2014/install/blas/19Apr11/linux-rhel_6-ppc64-gcc-4.4.6
      LAPACK=/gpfs/vesta-fs0/projects/FASTMath/ATPESC-2014/install/lapack/3.5.0/linux-rhel_6-ppc64-gcc-4.4.6
      
      export BOOST_ROOT=$BOOST_DIR
      EXTRA_ARGS=$@
      
      cmake \
      -Wno-dev \
      -D Trilinos_EXTRA_REPOSITORIES:STRING=SCOREC \
      -D Trilinos_ENABLE_SCOREC:BOOL=ON \
      -D SCOREC_DISABLE_STRONG_WARNINGS:BOOL=ON \
      \
      -D Trilinos_CONFIGURE_OPTIONS_FILE:FILEPATH=$TRILINOS_HOME/sampleScripts/AlbanySettings.cmake \
      -D CMAKE_BUILD_TYPE:STRING=NONE \
      -D TPL_FIND_SHARED_LIBS:BOOL=OFF \
      \
      -D CMAKE_C_FLAGS:STRING="-O3 -ffast-math -funroll-loops -DCODE_MP -DDISABLE_MPIIO -DNO_FILE_LOCK -DTEMPLATE_FRIENDS_NOT_SUPPORTED -DMPICH_SKIP_MPICXX -DNDEBUG -DHAVE_CONFIG_H" \
      -D CMAKE_CXX_FLAGS:STRING="-O3 -ffast-math -funroll-loops -DCODE_MP -DDISABLE_MPIIO -DNO_FILE_LOCK -DTEMPLATE_FRIENDS_NOT_SUPPORTED -DMPICH_SKIP_MPICXX -DNDEBUG -DHAVE_CONFIG_H" \
      -D CMAKE_Fortran_FLAGS:STRING="-O3 -ffast-math -funroll-loops" \
      \
      -D TPL_ENABLE_MPI:BOOL=ON \
      -D MPI_BASE_DIR:PATH=$MPI_BASE_DIR \
      -D Trilinos_EXTRA_LINK_FLAGS:STRING="-ldl" \
      \
      -D SEACAS_ENABLE_SEACASSVDI:BOOL=OFF \
      -D Trilinos_ENABLE_SEACASFastq:BOOL=OFF \
      -D Trilinos_ENABLE_SEACASBlot:BOOL=OFF \
      -D Trilinos_ENABLE_SEACASPLT:BOOL=OFF \
      -D TPL_ENABLE_X11:BOOL=OFF \
      \
      -D Boost_INCLUDE_DIRS:PATH=$BOOST_DIR/include \
      -D Boost_LIBRARY_DIRS:PATH=$BOOST_DIR/lib \
      -D BoostLib_INCLUDE_DIRS:PATH=$BOOST_DIR/include \
      -D BoostLib_LIBRARY_DIRS:PATH=$BOOST_DIR/lib \
      -D BoostAlbLib_INCLUDE_DIRS:PATH=$BOOST_DIR/include \
      -D BoostAlbLib_LIBRARY_DIRS:PATH=$BOOST_DIR/lib \
      \
      -D TPL_ENABLE_Netcdf:BOOL=ON \
      -D Netcdf_INCLUDE_DIRS:PATH="${NETCDF}/include" \
      -D Netcdf_LIBRARY_DIRS:PATH="${NETCDF}/lib" \
      -D TPL_Netcdf_LIBRARIES:STRING="/home/ghansen/FASTMath/lib/libnetcdf.a;/home/ghansen/FASTMath/lib/libhdf5_hl.a;/home/ghansen/FASTMath/lib/libhdf5.a;/home/ghansen/FASTMath/lib/libz.a" \
      \
      -D TPL_ENABLE_HDF5:BOOL=ON \
      -D HDF5_INCLUDE_DIRS:PATH="${HDFDIR}/include" \
      -D HDF5_LIBRARY_DIRS:PATH="${HDFDIR}/lib" \
      \
      -D TPL_ENABLE_Zlib:STRING=ON \
      -D Zlib_INCLUDE_DIRS:PATH="${HDFDIR}/include" \
      -D Zlib_LIBRARY_DIRS:PATH="${HDFDIR}/lib" \
      \
      -D TPL_ENABLE_ParMETIS:STRING=ON \
      -D ParMETIS_INCLUDE_DIRS:PATH="${PARMETISDIR}/include" \
      -D ParMETIS_LIBRARY_DIRS:PATH="${PARMETISDIR}/lib" \
      \
      -D TPL_ENABLE_SuperLU:STRING=ON \
      -D SuperLU_INCLUDE_DIRS:PATH="/home/ghansen/FASTMath/SuperLU_4.3/include" \
      -D SuperLU_LIBRARY_DIRS:PATH="/home/ghansen/FASTMath/SuperLU_4.3/lib" \
      \
      -D TPL_ENABLE_BLAS:STRING=ON \
      -D TPL_BLAS_LIBRARIES:FILEPATH="$BLAS/lib/blas.a" \
      -D TPL_ENABLE_LAPACK:STRING=ON \
      -D LAPACK_LIBRARY_DIRS:FILEPATH="$LAPACK/lib" \
      \
      -D CMAKE_INSTALL_PREFIX:PATH=${INSTALL_DIR} \
      \
      $EXTRA_ARGS \
      ${TRILINOS_HOME}
    
  • Build Trilinos

      make -j 8
      make install
    

Build Albany

  • Clone the repo

      cd /home/ghansen/FASTMath/src
      git clone https://github.com/gahansen/Albany.git
      cd Albany
      git checkout -b tpetra origin/tpetra
    
  • Configure the Albany build

      mkdir build
      cd build
      ./do-configure
    
  • CMake configure script ./do-configure:

      #!/bin/sh
      #
      export ALB_ENABLE_SCOREC=ON
      export ALB_ENABLE_LCM=ON
      export ALB_ENABLE_MOR=OFF
      export ALB_ENABLE_SPECULATIVE=OFF
      export ALB_ENABLE_LAME=OFF
      export ALB_ENABLE_SG_MP=OFF
      export ALB_ENABLE_INSTALL=ON
      export TRILINOS_INSTALL_DIR=/home/ghansen/FASTMath/Trilinos/MPI_REL
      export ALBANY_INSTALL_DIR=/home/ghansen/FASTMath/Albany
    
      cmake \
      -D ALBANY_TRILINOS_DIR:FILEPATH="$TRILINOS_INSTALL_DIR" \
      -D ENABLE_LCM:BOOL=${ALB_ENABLE_LCM} \
      -D ENABLE_LCM_SPECULATIVE:BOOL=${ALB_ENABLE_SPECULATIVE} \
      -D ENABLE_SCOREC:BOOL=${ALB_ENABLE_SCOREC} \
      -D ENABLE_MOR:BOOL=${ALB_ENABLE_MOR} \
      -D ENABLE_SG_MP:BOOL=${ALB_ENABLE_SG_MP} \
      -D ENABLE_INSTALL:BOOL=${ALB_ENABLE_INSTALL} \
      -D CMAKE_INSTALL_PREFIX:FILEPATH=${ALBANY_INSTALL_DIR} \
      -D ALBANY_MPI_OPTIONS:BOOL=ON \
      -D ALBANY_MPI_EXEC:STRING="/home/ghansen/FASTMath/scripts/run_Albany" \
      -D ALBANY_MPI_EXEC_NUMPROCS_FLAG:STRING="-n" \
      -D ALBANY_MPI_EXEC_MAX_NUMPROCS:STRING="4" \
      -D ALBANY_MPI_LEADING_OPTIONS:STRING="" \
      \
      /home/ghansen/FASTMath/src/Albany
    
  • Build Albany

      make -j 8
      make install
    

Running the Albany regression test suite

  • Note that the above Albany configure script also sets up ctest to run the regression suite on BG/Q, employing the script /home/ghansen/FASTMath/scripts/run_Albany to dispatch each parallel test to the compute nodes while ctest itself runs on the frontend node. From the build directory on the frontend node, typing

      qsub -A FASTMath -n 4 -t 60 --mode script /home/ghansen/FASTMath/scripts/reserve.sh
      qstat
    
  • Get the blockid info from qstat, and now run ctest

      export MPIEXEC_BLOCKID=VST-20220-31331-32
      ctest
    
  • will run the regression suite in this manner. Note also that all the usual ctest options apply to this command.

Running a typical Albany analysis

Using Albany to perform an analysis is very similar to using any typical finite element analysis code. Two preparation steps are required: a) define the problem spatial domain and nature by meshing the parts and assemblies that participate in the analysis, and b) define the materials, properties, boundary and initial conditions and specify numerical solver parameters within the Albany input (driver) file.

  1. Given the mesh, one decomposes the mesh into a set of subdomains for parallel execution, with one subdomain per processing element (or MPI rank).
  1. One then constructs an xml input file for Albany that specifies how to perform the simulation on the supplied decomposed mesh files.
  • Reserve a run allocation, this example is for 128 processors for 30 minutes

      qsub -A FASTMath -n 128 -t 30 --mode script /home/ghansen/FASTMath/scripts/reserve.sh
      qstat
    
  • Get the blockid info from qstat and set the environment variable

      export MPIEXEC_BLOCKID=VST-20220-31331-32
    
  • Use the script /home/ghansen/FASTMath/scripts/run_Albany to dispatch the parallel job to the compute nodes.

      export PATH=/home/ghansen/FASTMath/scripts:/home/ghansen/FASTMath/bin
      run_Albany -np 128 AlbanyT input.xml
    
Clone this wiki locally
You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.
Press h to open a hovercard with more details.