Building on Amos with bgclang

Glen Hansen edited this page May 24, 2018 · 1 revision

Install bgclang on the RPI BGQ

Get the bgclang rpms

From the website http://www.mcs.anl.gov/~hfinkel/bgclang/ locally download the following rpms:

bgclang-binutils-r284961-stable-1-5.ppc64.rpm
bgclang-compiler-rt-r284961-stable-1-5.ppc64.rpm
bgclang-gdb-r284961-stable-1-5.ppc64.rpm
bgclang-libcxx-r284961-stable-1-5.ppc64.rpm
bgclang-mpich3-r284961-stable-1-5.ppc64.rpm
bgclang-r284961-stable-1-5.ppc64.rpm
bgclang-stage1-3.4-1.ppc64.rpm
bgclang-stage1-libcxx-3.4-2.ppc64.rpm
bgclang-stage2-3.5.1-1.ppc64.rpm
bgclang-stage2-libcxx-3.5.1-1.ppc64.rpm
bgclang-stage3-libcxx-r266865-1.ppc64.rpm
bgclang-stage3-r266865-1.ppc64.rpm
toolchain-fixup-4.7.2-4.ppc64.rpm
vpkg-bin-sh-1-1.ppc64.rpm

scp the rpms to the BGQ

scp *.*.rpm PGESgnzw:lp01.cci.rpi.edu:~scratch/bgq/bgclang/rpm
scp the rpms to the BGQ

Install the bgclang packages

First specify the directories:

export prefix=/gpfs/u/home/PGES/PGESgnzw/scratch/bgq/bgclang
export dbdir=/gpfs/u/home/PGES/PGESgnzw/scratch/bgq/bgclang/rpmdb

Install the vpkg package:

rpm -Uhv --dbpath $dbdir --prefix $prefix \
vpkg-bin-sh-1-1.ppc64.rpm

Install the stage1 packages:

rpm -Uhv --dbpath $dbdir --prefix $prefix \
bgclang-stage1-libcxx-3.4-2.ppc64.rpm \
bgclang-stage1-3.4-1.ppc64.rpm

Install the stage2 packages:

rpm -Uhv --dbpath $dbdir --prefix $prefix \
bgclang-stage2-3.5.1-1.ppc64.rpm \
bgclang-stage2-libcxx-3.5.1-1.ppc64.rpm

Install the stage3 packages:

rpm -Uhv --dbpath $dbdir --prefix $prefix \
bgclang-stage3-r266865-1.ppc64.rpm \
bgclang-stage3-libcxx-r266865-1.ppc64.rpm

Install the toolchain fixup package:

rpm -Uhv --dbpath $dbdir --prefix $prefix \
toolchain-fixup-4.7.2-4.ppc64.rpm

Install the bin utils

rpm -Uhv --dbpath $dbdir --prefix $prefix \
bgclang-binutils-r284961-stable-1-5.ppc64.rpm

Install gdb:

rpm -Uhv --dbpath $dbdir --prefix $prefix \
bgclang-gdb-r284961-stable-1-5.ppc64.rpm

Install bgclang:

rpm -Uhv --dbpath $dbdir --prefix $prefix \
bgclang-r284961-stable-1-5.ppc64.rpm

Install compiler rt:

rpm -Uhv --dbpath $dbdir --prefix $prefix \
bgclang-compiler-rt-r284961-stable-1-5.ppc64.rpm

Install lib cxx:

rpm -Uhv --dbpath $dbdir --prefix $prefix \
bgclang-libcxx-r284961-stable-1-5.ppc64.rpm

Install mpich:

rpm -Uhv --dbpath $dbdir --prefix $prefix \
bgclang-mpich3-r284961-stable-1-5.ppc64.rpm

Third party library installation for Trilinos

Setup the environment

export PATH=/gpfs/u/home/PGES/PGESgnzw/scratch/bgq/bgclang/bin:$PATH
export PATH=/gpfs/u/home/PGES/PGESgnzw/scratch/bgq/bgclang/mpi/bgclang/bin:$PATH
export tpls=/gpfs/u/home/PGES/PGESgnzw/scratch/bgq/tpls/install

Install zlib

Download zlib from:
https://zlib.net/
scp it to the BGQ.

scp zlib-1.2.8.tar.gz PGESgnzw@lp01.cci.rpi.edu:~/scratch/bgq/tpls

On the BGQ, install

cd ~/scratch/bgq/tpls
tar -xzvf zlib-1.2.8.tar.gz
cd zlib-1.2.8
CC=mpiclang CFLAGS="-O3 -fPIC" ./configure --64 --prefix=${tpls} --static

Install boost

Download boost (v 1.65.1) from:
http://www.boost.org/users/download/
scp it to the BGQ

scp boost_1_65_1.tar.gz PGESgnzw@lp01.cci.rpi.edu:~/scratch/bgq/tpls

On the BGQ, install

cd ~/scratch/bgq/tpls
tar -xzvf boost_1_65_1.tar.gz
cd boost_1_65_1
echo "using clang : /gpfs/u/home/PGES/PGESgnzw/scratch/bgq/bgclang/bin/bgclang ;" >> ~/user-config.jam
echo "using mpi : /gpfs/u/home/PGES/PGESgnzw/scratch/bgq/bgclang/mpi/bgclang/bin/mpiclang++11 ;" >> ~/user-config.jam
./bootstrap.sh --with-toolset=clang --with-libraries=signals,regex,filesystem,system,mpi,serialization,thread,program_options,exception --prefix=${tpls}
./b2 link=static
./b2 install
cd $tpls/lib
rm *.so *.so.*

Copy fortran math libraries

Download the files found here (for-bgq.zip)

scp the zip file to the BGQ

scp for-bgq.zip PGESgnzw@lp01.cci.rpi.edu:~/scratch/bgq/tpls

On the BGQ, unzip the files and move the libraries into the install directory

cd ~/scratch/bgq/tpls
unzip for-bgq.zip
cd for-bgq
mv liblapack-gnu472-O3-unroll-nofastmath.a ../install/lib
mv librefblas-gnu472-O3-unroll-nofastmath.a ../install/lib

Install Trilinos

Clone Trilinos

cd ~/scratch/bgq
mkdir trilinos
cd trilinos
git clone https://github.com/trilinos/Trilinos.git src
mkdir build

Build Trilinos

do-config-trilinos-lite.sh

Here is the Trilinos CMake configuration I used. For different installs, the first few lines of this file will need to be modified to point to the correct location.

#!/bin/bash
# Modify these paths for your system.
TOOLS=/gpfs/u/home/PGES/PGESgnzw/scratch/bgq/tpls/install
MPIDIR=/gpf/u/home/PGES/PGESgnzw/scratch/bgq/bgclang/mpi/bgclang/bin
TRIL_INSTALL_DIR=/gpfs/u/home/PGES/PGESgnzw/scratch/bgq/trilinos/install
LAPACKDIR=$TOOLS
BOOSTDIR=$TOOLS
ZLIBDIR=$TOOLS
PARMETISDIR=$TOOLS

cmake \
\
 -D CMAKE_INSTALL_PREFIX=$TRIL_INSTALL_DIR \
 -D CMAKE_BUILD_TYPE=NONE \
 -D CMAKE_C_COMPILER=mpiclang \
 -D CMAKE_CXX_COMPILER=mpiclang++11 \
 -D CMAKE_C_FLAGS="-O2 -g" \
 -D CMAKE_CXX_FLAGS="-O2 -g -Wno-deprecated-declarations -Wno-sign-compare" \
 -D CMAKE_VERBOSE_MAKEFILE=OFF \
 -D BUILD_SHARED_LIBS=OFF \
 -D CMAKE_INSTALL_RPATH_USE_LINK_PATH=True \
\
 -D Trilinos_WARNINGS_AS_ERRORS_FLAGS="" \
 -D Trilinos_ENABLE_Fortran=OFF \
 -D Trilinos_EXTRA_LINK_FLAGS="-ldl -lgfortran" \
 -D Trilinos_ENABLE_ALL_PACKAGES=OFF \
 -D Trilinos_ENABLE_EXPLICIT_INSTANTIATION=ON \
 -D Trilinos_ENABLE_SECONDARY_TESTED_CODE=ON \
 -D Trilinos_DISABLE_ENABLED_FORWARD_DEP_PACKAGES=ON \
\
 -D Trilinos_ENABLE_Shards=ON \
 -D Trilinos_ENABLE_Sacado=ON \
 -D Trilinos_ENABLE_Epetra=ON \
 -D Trilinos_ENABLE_EpetraExt=ON \
 -D Trilinos_ENABLE_Ifpack=ON \
 -D Trilinos_ENABLE_AztecOO=ON \
 -D Trilinos_ENABLE_Amesos=ON \
 -D Trilinos_ENABLE_Anasazi=ON \
 -D Trilinos_ENABLE_Belos=ON \
 -D Trilinos_ENABLE_ML=ON \
 -D Trilinos_ENABLE_MiniTensor=ON \
 -D Trilinos_ENABLE_NOX=ON \
 -D Trilinos_ENABLE_Stratimikos=ON \
 -D Trilinos_ENABLE_Thyra=ON \
 -D Trilinos_ENABLE_Rythmos=ON \
 -D Trilinos_ENABLE_Piro=ON \
 -D Trilinos_ENABLE_Teko=ON \
 -D Trilinos_ENABLE_Ifpack2=ON \
 -D Trilinos_ENABLE_MueLu=ON \
 -D Trilinos_ENABLE_Intrepid2=ON \
 -D Trilinos_ENABLE_Intrepid=ON \
\
 -D Trilinos_ENABLE_Tpetra=ON \
 -D Tpetra_INST_INT_LONG_LONG=ON \
 -D Tpetra_INST_INT_LONG=OFF \
 -D Tpetra_INST_INT_INT=OFF \
 -D Tpetra_INST_DOUBLE=ON \
 -D Tpetra_INST_FLOAT=OFF \
 -D Tpetra_INST_COMPLEX_FLOAT=OFF \
 -D Tpetra_INST_COMPLEX_DOUBLE=OFF \
 -D Tpetra_INST_INT_UNSIGNED=OFF \
 -D Tpetra_INST_INT_UNSIGNED_LONG=OFF \
\
 -D Trilinos_ENABLE_Zoltan2=ON \
 -D Zoltan_ENABLE_ULONG_IDS=ON \
 -D ZOLTAN_BUILD_ZFDRIVE=OFF \
\
 -D Trilinos_ENABLE_Teuchos=ON \
 -D Teuchos_ENABLE_COMPLEX=OFF \
 -D Teuchos_ENABLE_LONG_LONG_INT=ON \
\
 -D Trilinos_ENABLE_Phalanx=ON \
 -D Phalanx_INDEX_SIZE_TYPE="KOKKOS" \
\
 -D Trilinos_ENABLE_Kokkos=ON \
 -D Kokkos_ENABLE_Serial=ON \
 -D Kokkos_ENABLE_OpenMP=OFF \
 -D Kokkos_ENABLE_Pthread=OFF \
\
 -D Trilinos_ENABLE_Amesos2=ON \
 -D Amesos2_ENABLE_KLU2=ON \
 -D Amesos2_ENABLE_SuperLUDist=OFF \
 -D Amesos2_ENABLE_Epetra=OFF \
\
 -D Trilinos_ENABLE_STKIO=OFF \
 -D Trilinos_ENABLE_STKMesh=OFF \
 -D Trilinos_ENABLE_Stokhos=OFF \
 -D Trilinos_ENABLE_SEACAS=OFF \
 -D Trilinos_ENABLE_SEACASIoss=OFF \
 -D Trilinos_ENABLE_SEACASExodus=OFF \
 -D Trilinos_ENABLE_SEACASFastq=OFF \
 -D Trilinos_ENABLE_SEACASBlot=OFF \
 -D Trilinos_ENABLE_SEACASPLT=OFF \
\
 -D TPL_ENABLE_BLAS=ON \
 -D TPL_BLAS_LIBRARIES="$LAPACKDIR/lib/librefblas-gnu472-O3-unroll-nofastmath.a" \
\
 -D TPL_ENABLE_LAPACK=ON \
 -D TPL_LAPACK_LIBRARIES="$LAPACKDIR/lib/liblapack-gnu472-O3-unroll-nofastmath.a" \
\
 -D TPL_ENABLE_Zlib=ON \
 -D Zlib_INCLUDE_DIRS="$ZLIBDIR/include" \
 -D Zlib_LIBRARY_DIRS="$ZLIBDIR/lib" \
\
 -D TPL_ENABLE_ParMETIS=ON \
 -D ParMETIS_INCLUDE_DIRS="$PARMETISDIR/include" \
 -D ParMETIS_LIBRARY_DIRS="$PARMETISDIR/lib" \
\
 -D TPL_ENABLE_METIS=ON \
 -D METIS_INCLUDE_DIRS="$PARMETISDIR/include" \
 -D METIS_LIBRARY_DIRS="$PARMETISDIR/lib" \
\
 -D TPL_ENABLE_Boost=ON \
 -D Boost_INCLUDE_DIRS="$BOOSTDIR/include" \
 -D Boost_LIBRARY_DIRS="$BOOSTDIR/lib" \
 -D TPL_ENABLE_BoostLib=ON \
 -D BoostLib_INCLUDE_DIRS="$BOOSTDIR/include" \
 -D BoostLib_LIBRARY_DIRS="$BOOSTDIR/lib" \
\
 -D TPL_ENABLE_MPI=ON \
 -D MPI_BASE_DIR=$MPIDIR \
\
 -D TPL_ENABLE_Netcdf=OFF \
 -D TPL_ENABLE_HDF5=OFF \
 -D TPL_ENABLE_Matio=OFF \
 -D TPL_ENABLE_X11=OFF \
\
../src \
2>&1 | tee config_log

Install Trilinos

On the BGQ, install

cd ~/scratch/bgq/trilinos/build
chmod +x do-config-trilinos.sh
./do-config-trilinos.sh
make -j 4
make install

Install SCOREC/core

clone core

cd ~/scratch/bgq
git clone https://github.com/SCOREC/core.git
cd core
mkdir build

do-config-scorec.sh

All appropriate file paths will need to be modified

cmake .. \
-DCMAKE_C_COMPILER:FILEPATH=mpiclang \
-DCMAKE_CXX_COMPILER:FILEPATH=mpiclang++11 \
-DCMAKE_INSTALL_PREFIX:PATH=/gpfs/u/home/PGES/PGESgnzw/scratch/bgq/core/install \
-DBUILD_SHARED_LIBS:BOOL=OFF \
-DENABLE_ZOLTAN:BOOL=ON \
-DENABLE_STK:BOOL=ON \
-DENABLE_STK_MESH:BOOL=OFF \
-DTrilinos_PREFIX:PATH=/gpfs/u/home/PGES/PGESgnzw/scratch/bgq/trilinos/install \
-DZOLTAN_PREFIX:PATH=/gpfs/u/home/PGES/PGESgnzw/scratch/bgq/trilinos/install \
-DPARMETIS_PREFIX:PATH=/gpfs/u/home/PGES/PGESgnzw/scratch/bgq/tpls/install \
-DCMAKE_SKIP_RPATH:BOOL=ON \
2>&1 | tee config_log

Install SCOREC/core

On the BGQ

cd ~/scratch/bgq/core/build
chmod +x do-config-scorec.sh
./do-config-scorec.sh
make -j 4
make install

Install Albany

Clone Albany

cd ~/scratch/bgq
git clone https://github.com/gahansen/Albany albany
cd albany
mkdir build

do-config-albany.sh

Here is my Albany CMake configuration script. The first two CMake option will need to be modified to point to the correct location.

#!/bin/sh

cmake \
 -D ALBANY_TRILINOS_DIR:FILEPATH=/gpfs/u/home/PGES/PGESgnzw/scratch/bgq/trilinos/install \
 -D ALBANY_SCOREC_DIR:FILEPATH=/gpfs/u/home/PGES/PGESgnzw/scratch/bgq/core/install \
 -D ENABLE_LCM:BOOL=ON \
 -D ENABLE_CTM:BOOL=OFF \
 -D ENABLE_AMP:BOOL=OFF \
 -D ENABLE_ANISO:BOOL=OFF \
 -D ENABLE_OMEGA_H:BOOL=OFF \
 -D ENABLE_QCAD:BOOL=OFF \
 -D ENABLE_LCM_SPECULATIVE:BOOL=OFF \
 -D ENABLE_LAME:BOOL=OFF \
 -D CMAKE_VERBOSE_MAKEFILE:BOOL=OFF \
 -D ENABLE_DEBUGGING:BOOL=OFF \
 -D ENABLE_CHECK_FPE:BOOL=OFF \
 -D ENABLE_HYDRIDE:BOOL=OFF \
 -D ENABLE_SCOREC:BOOL=ON \
 -D ENABLE_FELIX:BOOL=OFF \
 -D ENABLE_MOR:BOOL=OFF \
 -D ENABLE_ALBANY_CI:BOOL=OFF \
 -D ENABLE_ASCR:BOOL=OFF \
 -D ENABLE_64BIT_INT:BOOL=ON \
 -D ENABLE_ALBANY_EPETRA_EXE:BOOL=OFF \
 .. \
 2>&1 | tee config_log

Install Albany

cd ~/scratch/bgq/albany/build
chmod +x do-config-albany.sh
./do-config-albany.sh
make -j 4
make install
Clone this wiki locally
You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.
Press h to open a hovercard with more details.