Skip to content

build_on_perlmutter

Jump to bottom
jamesannis edited this page May 15, 2023 · 82 revisions

Building y3_cluster_cpp on Perlmutter

TOC

These instructions set you up to be able to use the GPU-enabled version of the y3_cluster_cpp analysis pipeline code.

Prior to Before you begin

THESE INSTRUCTIONS GIVE A BUILD, BUT ARE FRAGILE! Deviate at your own risk to your own sanity!

These instructions work best if you follow a programmers workflow when you make code changes- see Marc's workflow For us, use the Cosmosis and Cosmosis-standard-libary branches y3_cluster_cpp_main as the "main" branch to work in. (This helps us fold our fork back into Joe's main branches. DON'T PUSH TO cosmosis/cosmosis-standard-library's MAIN!)

Before you begin

Before beginning the installation of the software please make sure your environment does not have any settings that are likely to lead to problems. If one of these things is defined, it will conflict with our installation. It is best to remove from your login files (.bashrc, .profile, etc.) whatever commands set up the potential collision. Trying to un-setup after login is likely to lead to troubles, especially when running MPI programs or when running batch jobs.

  1. Make sure you do not have a conda environment activated. Look at the value of the environment variable CONDA_PREFIX. If you have a conda environment active this environment variable will be defined to show the directory in which that environment is found.
  2. Make sure you do not have a Python virtual environment activated. Look at the value of the environment variable VIRTUAL_ENV. If you have a Python virtual environment active this environment variable will be defined to show the directory in which that environment is found.
  3. Make sure you do not have the environment variable PYTHONPATH set. Having this set will subvert the Python environment management (using virtualenv) we are relying upon.
  4. Make sure you do not have some other cosmosis in your PATH. The command type cosmosis should tell you type: cosmosis: not found
  5. If you have the environment variable LD_LIBRARY_PATH set, beware of runtime failures. Setting LD_LIBRARY_PATH can cause the runtime linker to find the wrong libraries. In some circumstances, it can also lead to link-time failures.

Just set up to run your own copy of y3_cluster_cpp, assuming everything else is built

If you want to be able to change pipeline configuration but do not need to modify module code, then this is the simplest option.

# working directory, use high performance shared area on perlmutter

export TOP_DIR=/global/common/software/des/annis
export COSMOSIS_REPO_DIR=${TOP_DIR}/cosmosis
export CSL_DIR=${TOP_DIR}/cosmosis-standard-library

export INTEGRATION_TOOLS_DIR=${TOP_DIR}/y3_pipe_under
export Y3PIPE_DIR=${TOP_DIR}/y3_cluster_cpp
export Y3_CLUSTER_WORK_DIR=${Y3PIPE_DIR}/release-build
export Y3_CLUSTER_CPP_DIR=${Y3PIPE_DIR}
export COSMOSIS_STANDARD_LIBRARY=${CSL_DIR}

export CUBA_DIR=${INTEGRATION_TOOLS_DIR}/cuba
export CUBA_CPP_DIR=$INTEGRATION_TOOLS_DIR/cubacpp
export GPU_INT_DIR=${INTEGRATION_TOOLS_DIR}/gpuintegration 

# We set OMP_NUM_THREADS to avoid oversubscribing the CPU cores.
# This value has not been carefully tuned.
export OMP_NUM_THREADS=4

# Now set up CosmoSIS
source ${COSMOSIS_REPO_DIR}/setup-cosmosis-nersc /global/common/software/des/common/Conda_Envs/cosmosis-global

# And let's work in our own individual y3_cluster_cpp directory
export MY_TOP_DIR=/global/common/software/des/$(id -un)
export Y3PIPE_DIR=${MY_TOP_DIR}/y3_cluster_cpp
export Y3_CLUSTER_CPP_DIR=${Y3PIPE_DIR}
export Y3_CLUSTER_WORK_DIR=${Y3PIPE_DIR}/release-build
cd ${Y3PIPE_DIR}; 
#short-prompt

To run a CosmoSIS job, the instructions are as follows First, if you haven't already done so, source the CosmoSIS setup script, providing the path to the Conda environment that you'll use:

cd ${COSMOSIS_REPO_DIR}
# Note that the following uses a script only found in
# the `annis/cosmosis` fork of the repository, and there
# only in the `perlmutter` branch.
source setup-cosmosis-nersc /global/common/software/des/common/Conda_Envs/cosmosis-global

Then allocate one node and 4 GPU to run interactive jobs (In general, don't expect running from the login node to work well):

salloc --nodes 1 --qos interactive --time 02:00:00 --constraint gpu --gpus 4 --account=des_g
cd ${Y3PIPE_DIR}/y1_rerun

When that command returns, you have sole access to one compute node and one GPU. The srun MPI launcher will send you jobs to this node. For example, you can run:

srun -n 1 cosmosis --mpi demo_sigma_mort_cuda.ini

If things don't run right in some brittle way

If things don't run right in some brittle way, spend some time looking at cosmosis/setup-cosmosis-nersc, which has many Perlmutter supercomputer environment hacks embedded, which might need to be tweaked.

But I need to checkout and compile y3_cluster_cpp

You will want your own y3_cluster_cpp modules and models, and you will need to build your own versions of them.

Do the exports and source setup-cosmosis-nersc, for either the shared central installation or your own personal installation.

Clone and build y3_cluster_cpp

This is the only part of the installation that most people will need to do.

# You can replace the definition of  MY_TOP_DIR with any
# other directory under /global/common/software to which you can write.
export MY_TOP_DIR=/global/common/software/des/$(id -un)
mkdir -p ${MY_TOP_DIR}   # if dir exists, this command does nothing
cd ${MY_TOP_DIR}

If you haven't ever gotten y3_cluster_cpp, then the next step is to clone the y3_cluster_cpp repository. Of course, you only need to do this once.

# Using HTTP access works for most. It might work for you.
git clone http://bitbucket.org/mpaterno/y3_cluster_cpp.git
# Using ssh (git) access worked for me, but I was asked for the password for my SSH key.
# git clone git@bitbucket.org:mpaterno/y3_cluster_cpp.git

If you already have a y3_cluster_cpp, then usually one does:

cd $Y3PIPE_DIR
git pull

Next, you need to configure the build of the code.

cd ${Y3PIPE_DIR}   # this directory was created by the clone command above
mkdir -p release-build
cd release-build
cmake -DUSE_CUDA=On -DY3GCC_TARGET_ARCH=80-real -DPAGANI_DIR=${GPU_INT_DIR} -DGSL_ROOT_DIR=${CONDA_PREFIX} -DCMAKE_MODULE_PATH=${CUBA_CPP_DIR}/cmake/modules -DCUBACPP_DIR=${CUBA_CPP_DIR} -DCUBA_DIR=${CUBA_DIR}  -DCMAKE_BUILD_TYPE=Release  -G Ninja ${Y3_CLUSTER_CPP_DIR}

Look at the result of the configuration- if it has failed, try deleting the release-build directory and doing it again. Often this fixes the problem. (A successful result should have a line with "-- Build files have been written to:".)

Finally, you need to actually build the code:

ninja
ctest -j 10

The remainder of this subsection is only relevant if you want to do development of cuda, 'cudacpp, or gpuintegration`.

Build and install cuba

You only need to do this is you want to build your own version of cuba. That should only be necessary if you are trying to use a newer version than is used in the common installation above.

cd ${INTEGRATION_TOOLS_DIR}
git clone https://github.com/marcpaterno/cuba.git
cd cuba
./configure
./makesharedlib.sh
mkdir include; mkdir lib
mv cuba.h include/ ; mv libcuba.so lib/

Clone cubacpp

Note that you do not have to build cubapp; it is a header-only library. If you are doing development work on cubacpp you will want to run tests; then also follow the optional build instructions.

cd ${INTEGRATION_TOOLS_DIR}
git clone http://bitbucket.org/mpaterno/cubacpp.git

Optional build instructions for cubacpp

Coming soon!

Clone gpuintegration

Note that you do not have to build gpuintegration. If you are doing development work on gpuintegration you will want to run tests; then also follow the optional build instructions.

cd ${INTEGRATION_TOOLS_DIR}
git clone http://github.com/marcpaterno/gpuintegration.git
mkdir ${INTEGRATION_TOOLS_DIR}/release-build-gpuintegration
cd ${INTEGRATION_TOOLS_DIR}/release-build-gpuintegration
# Note: the build will produce many nvcc warnings, but should produce no errors
cmake -DPAGANI_DIR=${GPU_INT_DIR} -DCMAKE_BUILD_TYPE=Release -DPAGANI_TARGET_ARCH=80-real -G Ninja ${INTEGRATION_TOOLS_DIR}/gpuintegration
ninja
ctest

Optional build instructions for gpuintegration

Coming soon!

But I have to make CosmoSIS and CosmoSIS standard library!

Deep breath. Take one. And another. Then do the exports listed above.

Note that these instructions still have you sharing the common Conda environment.

cd $TOP_DIR
git clone -b y3_cluster_cpp_main https://github.com/annis/cosmosis.git
git clone -b y3_cluster_cpp_main https://github.com/annis/cosmosis-standard-library 
cd $COSMOSIS_REPO_DIR
#export MY_PATH_FOR_A_CONDA_ENV=jack
# Note: If you have created your own Conda environment, use the path to that environment here
source setup-cosmosis-nersc /global/common/software/des/common/Conda_Envs/cosmosis-global


# now the build
cd ${COSMOSIS_SRC_DIR}
make
cd $CSL_DIR
make

salloc --nodes 1 --qos interactive --time 02:00:00 --constraint gpu --gpus 1 --account=des_g
srun -n 4 python -m mpi4py.bench helloworld # test mpi using benchmark
cd ${COSMOSIS_STANDARD_LIBRARY}
srun -n 32 cosmosis --mpi demos/demo5.ini    # test python mpi using EMCEE; should take ~ 1 minute
srun -n 32 cosmosis --mpi demos/demo9.ini.   # test compiled fortran mpi using Multinest; should take ~30 seconds

I want to make my own Conda environment

You should only bother with this if you need to install some new python module (or other utility available through conda), and waiting on a request to install that module into the shared Conda environment is not feasible.

If you do build your own Conda environment you will also have to build your own CosmoSIS. Make sure to use the name of the environment you create when you source the setup-cosmosis-nersc script.

First define all the environment variables, go to COSMOSIS_REPO_DIR and source the setup-cosmosis-nersc script, as shown above. At this point you will have an active Conda environment; we have to deactivate it before creating a new one. Make sure to set MY_PATH_FOR_A_CONDA_ENV appropriately.

Note: you will be warned that a newer version of conda is available. Do not attempt to update conda. We are using a module provided by NERSC to obtain conda, rather than our own installation, and we can not update it ourselves.

# module load python/3.9-anaconda-2021.11 # to get conda.sh
conda deactivate
cd /global/common/software/des/common/Conda_Envs/
conda create --yes --prefix $NAME_OF_NEW_CONDA_ENV \
  astropy \
  cfitsio \
  cmake \
  conda-tree \
  Cython \
  fftw \
  fitsio \
  graphviz \
  gsl \
  h5py \
  hankel \
  htop \
  ipython \
  matplotlib-base \
  mpi4py \
  mpich="3.3.*=external_*" \
  ninja \
  numpy \
  pylint \
  python=3.9 \
  PyYAML \
  ripgrep \
  quarto \
  r \
  r-feather \
  r-tidyverse \
  r-tinytex \
  scikit-learn \
  scipy \
  zeus-mcmc

This installation should take a few minutes. The environment is not yet complete -- we also require cluster_toolkit, which is not available through Conda. We also need several other packages which, if taken from conda, will introduce bad dependencies on versions of other libraries (related to the Fortran and C runtimes. These conda packages seem to present a problem because of they are associated with divergent versions of the GCC suite. These include:

  • gcc_impl_linux-64 9.4.0 h03d3576_14 conda-forge
  • gcc_linux-64 9.4.0 h391b98a_9 conda-forge
  • gfortran_impl_linux-64 9.4.0 h0003116_14 conda-forge
  • gfortran_linux-64 9.4.0 hf0ab688_9 conda-forge
  • libgcc-devel_linux-64 9.4.0 hd854feb_14 conda-forge
  • libsanitizer 9.4.0 h79bfe98_14 conda-forge

To install these in the environment we use pip. To do this you must first activate the environment. (When codes are in fortran or use MPI, they are more likely to be included in the pip install list)

conda activate $MY_PATH_FOR_A_CONDA_ENV

python -m pip install \
  camb \
  emcee \
  future \
  kombine \
  mpmath \
  sympy

mkdir -p ${INTEGRATION_TOOLS_DIR}/tmp
cd ${INTEGRATION_TOOLS_DIR}/tmp
wget https://github.com/marcpaterno/cluster_toolkit/archive/master.tar.gz
tar xf master.tar.gz
cd cluster_toolkit-master/
python setup.py install     # This will install into the environment
cd ${INTEGRATION_TOOLS_DIR}
rm -r tmp/

Building the whole set up from scratch

How to do the whole thing from scratch:

  1. do the conda environment build
  2. do the pip install
  3. get and build cosmosis
  4. get and build cosmosis-standard-libary
  5. get the cuba, cubacpp, gpuintegration and y3_cluster_cpp codes and build

Clone this wiki locally