Skip to content

build_on_perlmutter

Jump to bottom
Marc Paterno edited this page Apr 18, 2022 · 82 revisions

Building y3_cluster_cpp on Perlmutter

TOC

These instructions set you up to be able to use the GPU-enabled version of the y3_cluster_cpp analysis pipeline code.

Just set up to run the y3_cluster_cpp, assuming everything else is built

If you want to be able to change pipeline configuration but do not need to modify module code, then this is the simplest option.

# working directory, use high performance shared area on perlmutter

export TOP_DIR=/global/common/software/des/annis
export COSMOSIS_REPO_DIR=${TOP_DIR}/cosmosis
export CSL_DIR=${TOP_DIR}/cosmosis-standard-library

export INTEGRATION_TOOLS_DIR=${TOP_DIR}/y3_pipe_under
export Y3PIPE_DIR=${TOP_DIR}/y3_cluster_cpp
export Y3_CLUSTER_WORK_DIR=${Y3PIPE_DIR}/release-build
export Y3_CLUSTER_CPP_DIR=${Y3PIPE_DIR}
export COSMOSIS_STANDARD_LIBRARY=${CSL_DIR}

export CUBA_DIR=${INTEGRATION_TOOLS_DIR}/cuba
export CUBA_CPP_DIR=$INTEGRATION_TOOLS_DIR/cubacpp
export GPU_INT_DIR=${INTEGRATION_TOOLS_DIR}/gpuintegration 

# We set OMP_NUM_THREADS to avoid oversubscribing the CPU cores.
# This value has not been carefully tuned.
export OMP_NUM_THREADS=4

cd ${COSMOSIS_REPO_DIR}
# Note that the following uses a script only found in
# the `annis/cosmosis` fork of the repository, and there
# only in the `perlmutter` branch.
source setup-cosmosis-nersc /global/common/software/des/common/Conda_Envs/cosmosis-global

salloc --nodes 1 --qos interactive --time 02:00:00 --constraint gpu --gpus 1 --account=des_g
cd ${Y3PIPE_DIR}/y1_rerun

srun -n1 cosmosis --mpi sigma_mort_mcubes.ini

But I need to compile y3_cluster_cpp

If you wish to modify the code in y3_cluster_cpp modules and models, then you need to build your own versions of them.

Do the exports and source setup-cosmosis-nersc, for either the shared central installation or your own personal installation.

Clone and build y3_cluster_cpp

This is the only part of the installation that most people will need to do.

# You can replace the definition of  MY_TOP_DIR with any
# other directory under /global/common/software to which you can write.
export MY_TOP_DIR=/global/common/software/des/$(id -un)
mkdir -p ${MY_TOP_DIR}
cd ${MY_TOP_DIR}
export Y3PIPE_DIR=${MY_TOP_DIR}/y3_cluster_cpp
export Y3_CLUSTER_WORK_DIR=${Y3PIPE_DIR}/release-build
export Y3_CLUSTER_CPP_DIR=${Y3PIPE_DIR}

# Using HTTP access failed for me. It might work for you.
# git clone http://bitbucket.org/mpaterno/y3_cluster_cpp.git
# Using ssh (git) access worked for me, but I was asked for the password for my SSH key.
git clone git@bitbucket.org:mpaterno/y3_cluster_cpp.git

cd ${Y3PIPE_DIR}   # this directory was created by the clone command above
mkdir release-build
cd release-build
cmake -DUSE_CUDA=On -DY3GCC_TARGET_ARCH=80-real -DPAGANI_DIR=${GPU_INT_DIR} -DGSL_ROOT_DIR=${CONDA_PREFIX} -DCMAKE_MODULE_PATH=${CUBA_CPP_DIR}/cmake/modules -DCUBACPP_DIR=${CUBA_CPP_DIR} -DCUBA_DIR=${CUBA_DIR}  -DCMAKE_BUILD_TYPE=Release  -G Ninja ${Y3_CLUSTER_CPP_DIR}
ninja

The remainder of this subsection is only relevant if you want to do development of cuda, 'cudacpp, or gpuintegration`.

Build and install cuba

You only need to do this is you want to build your own version of cuba. That should only be necessary if you are trying to use a newer version than is used in the common installation above.

cd ${INTEGRATION_TOOLS_DIR}
git clone https://github.com/marcpaterno/cuba.git
cd cuba
./configure
./makesharedlib.sh
mkdir include; mkdir lib
mv cuba.h include/ ; mv libcuba.so lib/

Clone cubacpp

Note that you do not have to build cubapp; it is a header-only library. If you are doing development work on cubacpp you will want to run tests; then also follow the optional build instructions.

cd ${INTEGRATION_TOOLS_DIR}
git clone http://bitbucket.org/mpaterno/cubacpp.git

Optional build instructions for cubacpp

Coming soon!

Clone gpuintegration

Note that you do not have to build gpuintegration. If you are doing development work on gpuintegration you will want to run tests; then also follow the optional build instructions.

cd ${INTEGRATION_TOOLS_DIR}
git clone http://github.com/marcpaterno/gpuintegration.git
mkdir ${INTEGRATION_TOOLS_DIR}/release-build-gpuintegration
cd ${INTEGRATION_TOOLS_DIR}/release-build-gpuintegration
# Note: the build will produce many nvcc warnings, but should produce no errors
cmake -DPAGANI_DIR=${GPU_INT_DIR} -DCMAKE_BUILD_TYPE=Release -DPAGANI_TARGET_ARCH=80-real -G Ninja ${INTEGRATION_TOOLS_DIR}/gpuintegration
ninja
ctest

Optional build instructions for gpuintegration

Coming soon!

But I have to make CosmoSIS and CosmoSIS standard library!

Deep breath. Take one. And another. Then do the exports listed above.

Note that these instructions still have you sharing the common Conda environment.

cd $TOP_DIR
git clone -b perlmutter cosmosis # placeholder for right command.
git clone -b perlmutter cosmosis-standard-library # placeholder for right command.
cd $COSMOSIS_REPO_DIR
#export MY_PATH_FOR_A_CONDA_ENV=jack
# Note: If you have created your own Conda environment, use the path to that environment here
source setup-cosmosis-nersc /global/common/software/des/common/Conda_Envs/cosmosis-global


# now the build
cd ${COSMOSIS_SRC_DIR}
make
cd $CSL_DIR
make

salloc --nodes 1 --qos interactive --time 02:00:00 --constraint gpu --gpus 1 --account=des_g
srun -n 4 python -m mpi4py.bench helloworld # test mpi using benchmark
cd ${COSMOSIS_STANDARD_LIBRARY}
srun -n 32 cosmosis --mpi demos/demo5.ini    # test python mpi using EMCEE; should take ~ 1 minute
srun -n 32 cosmosis --mpi demos/demo9.ini.   # test compiled fortran mpi using Multinest; should take ~30 seconds

I want to make my own Conda environment

You should only bother with this if you need to install some new python module (or other utility available through conda), and waiting on a request to install that module into the shared Conda environment is not feasible.

If you do build your own Conda environment you will also have to build your own CosmoSIS. Make sure to use the name of the environment you create when you source the setup-cosmosis-nersc script.

First define all the environment variables, go to COSMOSIS_REPO_DIR and source the setup-cosmosis-nersc script, as shown above. At this point you will have an active Conda environment; we have to deactivate it before creating a new one. Make sure to set MY_PATH_FOR_A_CONDA_ENV appropriately.

Note: you will be warned that a newer version of conda is available. Do not attempt to update conda. We are using a module provided by NERSC to obtain conda, rather than our own installation, and we can not update it ourselves.

conda deactivate
conda create --yes --prefix $MY_PATH_FOR_A_CONDA_ENV \
  cfitsio \
  cmake \
  conda-tree \
  Cython \
  PyYAML \
  astropy \
  fftw \
  fitsio \
  gsl \
  htop \
  ipython \
  matplotlib \
  mpi4py \
  mpich="3.3.*=external_*" \
  ninja \
  numpy \
  python=3.9 \
  ripgrep \
  scikit-learn \
  scipy

This installation should take a few minutes. The environment is not yet complete -- we also require cluster_toolkit, which is not available through Conda. We also need several other packages which, if taken from conda, will introduce bad dependencies on versions of other libraries (related to the Fortran and C runtimes. These conda packages seem to present a problem because of they are associated with divergent versions of the GCC suite. These include:

  • gcc_impl_linux-64 9.4.0 h03d3576_14 conda-forge
  • gcc_linux-64 9.4.0 h391b98a_9 conda-forge
  • gfortran_impl_linux-64 9.4.0 h0003116_14 conda-forge
  • gfortran_linux-64 9.4.0 hf0ab688_9 conda-forge
  • libgcc-devel_linux-64 9.4.0 hd854feb_14 conda-forge
  • libsanitizer 9.4.0 h79bfe98_14 conda-forge

To install these in the environment we use pip. To do this you must first activate the environment.

conda activate $MY_PATH_FOR_A_CONDA_ENV

python -m pip install \
  camb \
  emcee \
  future \
  kombine \
  mpmath \
  sympy

mkdir -p ${INTEGRATION_TOOLS_DIR}/tmp
cd ${INTEGRATION_TOOLS_DIR}/tmp
wget https://github.com/marcpaterno/cluster_toolkit/archive/master.tar.gz
tar xf master.tar.gz
cd cluster_toolkit-master/
python setup.py install     # This will install into the environment
cd ${INTEGRATION_TOOLS_DIR}
rm -r tmp/

Clone this wiki locally