build_on_perlmutter

Building y3_cluster_cpp on Perlmutter

These instructions set you up to be able to use the GPU-enabled version of the y3_cluster_cpp analysis pipeline code.

Just set up to run the y3_cluster_cpp, assuming everything else is built

If you want to be able to change pipeline configuration but do not need to modify module code, then this is the simplest option.

# working directory, use high performance shared area on perlmutter

# Note that we have both TOPDIR and TOP_DIR; be careful!
export TOP_DIR=/global/common/software/des/$(id -un)
export COSMOSIS_REPO_DIR=${TOP_DIR}/cosmosis
export CSL_DIR=${TOP_DIR}/cosmosis-standard-library

export TOPDIR=${TOP_DIR}/y3_pipe_under
export Y3PIPE_DIR=${TOP_DIR}/y3_cluster_cpp
export Y3_CLUSTER_WORK_DIR=${Y3PIPE_DIR}/release-build
export Y3_CLUSTER_CPP_DIR=${Y3PIPE_DIR}
export COSMOSIS_STANDRD_LIBRARY=${CSL_DIR}

cd ${COSMOSIS_REPO_DIR}
# Note that the following uses a script only found in
# the `annis/cosmosis` fork of the repository, and there
# only in the `perlmutter` branch.
source setup-cosmosis-nersc /global/common/software/des/common/Conda_Envs/cosmosis-global

salloc --nodes 1 --qos interactive --time 02:00:00 --constraint gpu --gpus 1 --account=des_g
cd ${Y3PIPE_DIR}/y1_rerun

srun -n1 cosmosis --mpi sigma_mort_mcubes.ini

But I need to compile y3_cluster_cpp

If you wish to modify the code in y3_cluster_cpp modules and models, then you need to build your own versions of them.

Do the exports and source setup-cosmosis-nersc, then:

# get cmake
wget https://github.com/Kitware/CMake/releases/download/v3.23.0/cmake-3.23.0-linux-x86_64.tar.gz
# Untar this, which will create a directory cmake-3.23.0-linux-x86_64,
# which will contain a bin directory. Put that bin directory onto your PATH.

Build cluster_toolkit and install it into the conda environment

mkdir -p ${TOPDIR}/tmp
cd ${TOPDIR}/tmp
wget https://github.com/marcpaterno/cluster_toolkit/archive/master.tar.gz
tar xf master.tar.gz
cd cluster_toolkit-master/
python setup.py install     # This will install into the environment
cd ${TOP_DIR}

Build and install cuba

cd ${TOPDIR}
git clone https://github.com/marcpaterno/cuba.git
cd cuba
./configure
./makesharedlib.sh
mkdir include; mkdir lib
mv cuba.h include/ ; mv libcuba.so lib/
export CUBA_DIR=${TOPDIR}/cuba

Clone cubacpp

Note that you do not have to build cubapp. If you are doing development work on cubacpp you will want to run tests; then also follow the optional build instructions.

cd ${TOPDIR}
git clone http://bitbucket.org/mpaterno/cubacpp.git
export CUBA_CPP_DIR=$TOPDIR/cubacpp

Optional build instructions for cubacpp

Coming soon!

Clone gpuintegration

Note that you do not have to build gpuintegration. If you are doing development work on gpuintegration you will want to run tests; then also follow the optional build instructions.

cd ${TOPDIR}
git clone http://github.com/marcpaterno/gpuintegration.git
export GPU_INT_DIR=${TOPDIR}/gpuintegration
mkdir build
cd build
cmake -DPAGANI_DIR=${GPU_INT_DIR} -DCMAKE_BUILD_TYPE=Release -DPAGANI_TARGET_ARCH=80-real -G Ninja ..
ninja
ctest

Optional build instructions for gpuintegration

Coming soon!

Clone and build y3_cluster_cpp

I bet you thought we'd never get here, huh?

cd ${Y3PIPE_DIR}
git clone http://bitbucket.org/mpaterno/y3_cluster_cpp.git

cd ${Y3PIPE_DIR}/y3_cluster_cpp
mkdir release-build
cd release-build/
cmake -DUSE_CUDA=On -DY3GCC_TARGET_ARCH=80-real -DPAGANI_DIR=${PAGANI_DIR} -DGSL_ROOT_DIR=$CONDA_PREFIX -DCMAKE_MODULE_PATH="$CUBA_CPP_DIR/cmake/modules" -DCUBACPP_DIR=$CUBA_CPP_DIR -DCUBA_DIR=$CUBA_DIR  -DCMAKE_BUILD_TYPE=Release  -G Ninja ../
ninja

But I have to make CosmoSIS and CosmoSIS standard library!

Deep breath. Take one. And another. Then do the exports listed above.

Note that these instructions still have you sharing the common Conda environment.

cd $TOP_DIR
git clone -b perlmutter cosmosis # placeholder for right command.
git clone -b perlmutter cosmosis-standard-library # placeholder for right command.
cd $COSMOSIS_REPO_DIR
#export MY_NAME_FOR_A_CONDA_ENV=jack
source setup-cosmosis-nersc /global/common/software/des/common/Conda_Envs/cosmosis-global
#conda create --yes --name $MY_NAME_FOR_A_CONDA_ENV astropy Cython gsl fftw matplotlib mpich="3.3.*=external_*" mpi    4py ninja numpy PyYAML ripgrep scikit-learn scipy  python=3.9
#conda activate $MY_NAME_FOR_A_CONDA_ENV; fix_prompt
# CONDA_PREFIX and COSMOSIS_SRC_DIR are set by the conda activate and the source setup-cosmosis-nersc
# export CFITSIO_DIR=$CONDA_PREFIX
# export CFITSIO_INC=$CFITSIO_DIR/include
# export CFITSIO_LIB=$CFITSIO_DIR/lib
# pip install camb emcee kombine future # fitsio but fitsio wont compile on Perlmutter. fake it with cfitsio
# conda install cfitsio==3.470

## QUESTION from Marc
# Where is the python module 'fitsio'?

# now the build
cd ${COSMOSIS_SRC_DIR}
make
cd $CSL_DIR
make

salloc --nodes 1 --qos interactive --time 02:00:00 --constraint gpu --gpus 1 --account=des_g
srun -n 4 python -m mpi4py.bench helloworld # test mpi using benchmark
srun -n 5 cosmosis --mpi demos/demo5.ini    # test python mpi using EMCEE
# change emcee to multinest in demo5.ini
srun -n 5 cosmosis --mpi demos/demo5.ini.   # test compiled fortran mpi using Multinest