Skip to content

build_on_perlmutter

Jump to bottom
Marc Paterno edited this page Apr 6, 2022 · 82 revisions

Building y3_cluster_cpp on Perlmutter

These instructions set you up to be able to use the GPU-enabled version of the y3_cluster_cpp analysis pipeline code.

Just set up to run the y3_cluster_cpp, assuming everything else is built

If you want to be able to change pipeline configuration but do not need to modify module code, then this is the simplest option.

# working directory, use high performance shared area on perlmutter

# Note that we have both TOPDIR and TOP_DIR; be careful!
export TOP_DIR=/global/common/software/des/$(id -un)
export COSMOSIS_REPO_DIR=${TOP_DIR}/cosmosis
export CSL_DIR=${TOP_DIR}/cosmosis-standard-library

export TOPDIR=${TOP_DIR}/y3_pipe_under
export Y3PIPE_DIR=${TOP_DIR}/y3_cluster_cpp
export Y3_CLUSTER_WORK_DIR=${Y3PIPE_DIR}/release-build
export Y3_CLUSTER_CPP_DIR=${Y3PIPE_DIR}
export COSMOSIS_STANDRD_LIBRARY=${CSL_DIR}

cd ${COSMOSIS_REPO_DIR}
# Note that the following uses a script only found in
# the `annis/cosmosis` fork of the repository, and there
# only in the `perlmutter` branch.
source setup-cosmosis-nersc /global/common/software/des/common/Conda_Envs/cosmosis-global

salloc --nodes 1 --qos interactive --time 02:00:00 --constraint gpu --gpus 1 --account=des_g
cd ${Y3PIPE_DIR}/y1_rerun

srun -n1 cosmosis --mpi sigma_mort_mcubes.ini

But I need to compile y3_cluster_cpp

If you wish to modify the code in y3_cluster_cpp modules and models, then you need to build your own versions of them.

Do the exports and source setup-cosmosis-nersc, then:

# get cmake
wget https://github.com/Kitware/CMake/releases/download/v3.23.0/cmake-3.23.0-linux-x86_64.tar.gz
# Untar this, which will create a directory cmake-3.23.0-linux-x86_64,
# which will contain a bin directory. Put that bin directory onto your PATH.

Build and install cuba 

cd ${TOPDIR}
git clone https://github.com/marcpaterno/cuba.git
cd cuba
./configure
./makesharedlib.sh
mkdir include; mkdir lib
mv cuba.h include/ ; mv libcuba.so lib/
export CUBA_DIR=${TOPDIR}/cuba

Clone cubacpp

Note that you do not have to build cubapp. If you are doing development work on cubacpp you will want to run tests; then also follow the optional build instructions.

cd ${TOPDIR}
git clone http://bitbucket.org/mpaterno/cubacpp.git
export CUBA_CPP_DIR=$TOPDIR/cubacpp

Optional build instructions for cubacpp

Coming soon!

Clone gpuintegration

Note that you do not have to build gpuintegration. If you are doing development work on gpuintegration you will want to run tests; then also follow the optional build instructions.

cd ${TOPDIR}
git clone http://github.com/marcpaterno/gpuintegration.git
export GPU_INT_DIR=${TOPDIR}/gpuintegration
mkdir build
cd build
cmake -DPAGANI_DIR=${GPU_INT_DIR} -DCMAKE_BUILD_TYPE=Release -DPAGANI_TARGET_ARCH=80-real -G Ninja ..
ninja
ctest

Optional build instructions for gpuintegration

Coming soon!

Clone and build y3_cluster_cpp

I bet you thought we'd never get here, huh?

cd ${Y3PIPE_DIR}
git clone http://bitbucket.org/mpaterno/y3_cluster_cpp.git

cd ${Y3PIPE_DIR}/y3_cluster_cpp
mkdir release-build
cd release-build/
cmake -DUSE_CUDA=On -DY3GCC_TARGET_ARCH=80-real -DPAGANI_DIR=${PAGANI_DIR} -DGSL_ROOT_DIR=$CONDA_PREFIX -DCMAKE_MODULE_PATH="$CUBA_CPP_DIR/cmake/modules" -DCUBACPP_DIR=$CUBA_CPP_DIR -DCUBA_DIR=$CUBA_DIR  -DCMAKE_BUILD_TYPE=Release  -G Ninja ../
ninja

But I have to make CosmoSIS and CosmoSIS standard library!

Deep breath. Take one. And another. Then do the exports listed above.

Note that these instructions still have you sharing the common Conda environment.

cd $TOP_DIR
git clone -b perlmutter cosmosis # placeholder for right command.
git clone -b perlmutter cosmosis-standard-library # placeholder for right command.
cd $COSMOSIS_REPO_DIR
#export MY_PATH_FOR_A_CONDA_ENV=jack
# Note: If you have created your own Conda environment, use the path to that environment here
source setup-cosmosis-nersc /global/common/software/des/common/Conda_Envs/cosmosis-global


# now the build
cd ${COSMOSIS_SRC_DIR}
make
cd $CSL_DIR
make

salloc --nodes 1 --qos interactive --time 02:00:00 --constraint gpu --gpus 1 --account=des_g
srun -n 4 python -m mpi4py.bench helloworld # test mpi using benchmark
srun -n 5 cosmosis --mpi demos/demo5.ini    # test python mpi using EMCEE
# change emcee to multinest in demo5.ini
srun -n 5 cosmosis --mpi demos/demo5.ini.   # test compiled fortran mpi using Multinest

I want to make my own Conda environment

You should only bother with this if you need to install some new python module (or other utility available through conda), and waiting on a request to install that module into the shared Conda environment is not feasible.

If you do build your own Conda environment you will also have to build your own CosmoSIS. Make sure to use the name of the environment you create when you source the setup-cosmosis-nersc script.

First define all the environment variables, go to COSMOSIS_REPO_DIR and source the setup-cosmosis-nersc script, as shown above. At this point you will have an active Conda environment; we have to deactivate it before creating a new one. Make sure to set MY_PATH_FOR_A_CONDA_ENV appropriately.

Note: you will be warned that a newer version of conda is available. Do not attempt to update conda. We are using a module provided by NERSC to obtain conda, rather than our own installation, and we can not update it ourselves.

conda deactivate
conda create --yes --prefix $MY_PATH_FOR_A_CONDA_ENV \
  cfitsio \
  conda-tree \
  Cython \
  PyYAML \
  astropy \
  fftw \
  fitsio \
  gsl \
  htop \
  ipython \
  matplotlib \
  mpi4py \
  mpich="3.3.*=external_*" \
  ninja \
  numpy \
  python=3.9 \
  ripgrep \
  scikit-learn \
  scipy

This installation should take a few minutes. The environment is not yet complete -- we also require cluster_toolkit, which is not available through Conda. We also need several other packages which, if taken from conda, will introduce bad dependencies on versions of other libraries (related to the Fortran and C runtimes. These conda packages seem to present a problem because of they are associated with divergent versions of the GCC suite. These include:

  • gcc_impl_linux-64 9.4.0 h03d3576_14 conda-forge
  • gcc_linux-64 9.4.0 h391b98a_9 conda-forge
  • gfortran_impl_linux-64 9.4.0 h0003116_14 conda-forge
  • gfortran_linux-64 9.4.0 hf0ab688_9 conda-forge
  • libgcc-devel_linux-64 9.4.0 hd854feb_14 conda-forge
  • libsanitizer 9.4.0 h79bfe98_14 conda-forge

To install these in the environment we use pip. To do this you must first activate the environment.

conda activate $MY_PATH_FOR_A_CONDA_ENV

python -m pip install \
  camb \
  emcee \
  future \
  kombine \
  mpmath \
  sympy

mkdir -p ${TOPDIR}/tmp
cd ${TOPDIR}/tmp
wget https://github.com/marcpaterno/cluster_toolkit/archive/master.tar.gz
tar xf master.tar.gz
cd cluster_toolkit-master/
python setup.py install     # This will install into the environment
cd ${TOPDIR}
rm -r tmp/

Clone this wiki locally