build_on_perlmutter

Just set up to run the y3_cluster_cpp, assuming everything else is built

# working directory, use high performance shared area on perlmutter

# Note that we have both TOPDIR and TOP_DIR; be careful!
export TOP_DIR=/global/common/software/des/$(id --user --name)
export COSMOSIS_SRC_DIR=${TOP_DIR}/cosmosis
export CSL_DIR=${TOP_DIR}/cosmosis-standard-library

export TOPDIR=${TOP_DIR}/y3_pipe_under
export Y3PIPE_DIR=${TOP_DIR}/y3_cluster_cpp
export Y3_CLUSTER_WORK_DIR=${Y3PIPE_DIR}/release-build
export Y3_CLUSTER_CPP_DIR=${Y3PIPE_DIR}
export COSMOSIS_STANDRD_LIBRARY=${CSL_DIR}
    
cd ${COSMOSIS_SRC_DIR}
# Note that the following uses a script only found in
# the `annis/cosmosis` fork of the repository, and there
# only in the `perlmutter` branch.
source setup-cosmosis-nersc /global/common/software/des/common/Conda_Envs/cosmosis-global

salloc --nodes 1 --qos interactive --time 02:00:00 --constraint gpu --gpus 1 --account=des_g
cd ${Y3PIPE_DIR}/y1_rerun
    
srun -n1 cosmosis --mpi sigma_mort_mcubes.ini

But I need to compile y3_cluster_cpp

Do the exports and source setup-cosmosis-nersc, then:

# get cmake
wget https://github.com/Kitware/CMake/releases/download/v3.23.0/cmake-3.23.0-linux-x86_64.tar.gz
# Untar this, which will create a directory cmake-3.23.0-linux-x86_64, 
# which will contain a bin directory. Put that bin directory onto your PATH.

Build cluster_toolkit and install it into the conda environment

mkdir -p ${TOPDIR}/tmp
cd ${TOPDIR}/tmp
wget https://github.com/marcpaterno/cluster_toolkit/archive/master.tar.gz
tar xf master.tar.gz
cd cluster_toolkit-master/
python setup.py install     # This will install into the environment
cd ${TOP_DIR}

Build and install cuba

cd ${TOPDIR}
git clone https://github.com/marcpaterno/cuba.git
cd cuba
./configure
./makesharedlib.sh
mkdir include; mkdir lib 
mv cuba.h include/ ; mv libcuba.so lib/
export CUBA_DIR=${TOPDIR}/cuba

Clone cubacpp

Note that you do not have to build cubapp. If you are doing development work on cubacpp you will want to run tests; then also follow the optional build instructions.

cd ${TOPDIR} 
git clone http://bitbucket.org/mpaterno/cubacpp.git
export CUBA_CPP_DIR=$TOPDIR/cubacpp

Optional build instructions for cubacpp

Coming soon!

Clone gpuintegration

Note that you do not have to build gpuintegration. If you are doing development work on gpuintegration you will want to run tests; then also follow the optional build instructions.

cd ${TOPDIR} 
git clone http://github.com/marcpaterno/gpuintegration.git
export GPU_INT_DIR=${TOPDIR}/gpuintegration
mkdir build
cd build
cmake -DPAGANI_DIR=${GPU_INT_DIR} -DCMAKE_BUILD_TYPE=Release -DPAGANI_TARGET_ARCH=80-real -G Ninja ..
ninja
ctest

Optional build instructions for gpuintegration

Coming soon!

Clone and build y3_cluster_cpp

I bet you thought we'd never get here, huh?

cd ${Y3PIPE_DIR}
git clone http://bitbucket.org/mpaterno/y3_cluster_cpp.git

cd ${Y3PIPE_DIR}/y3_cluster_cpp
mkdir release-build
cd release-build/
cmake -DUSE_CUDA=On -DY3GCC_TARGET_ARCH=80-real -DPAGANI_DIR=${PAGANI_DIR} -DGSL_ROOT_DIR=$CONDA_PREFIX -DCMAKE_MODULE_PATH="$CUBA_CPP_DIR/cmake/modules" -DCUBACPP_DIR=$CUBA_CPP_DIR -DCUBA_DIR=$CUBA_DIR  -DCMAKE_BUILD_TYPE=Release  -G Ninja ../
ninja

But I have to make CosmoSIS and CosmoSIS standard library!

Deep breath. Take one. And another.

cd $TOP_DIR
git clone -b perlmutter cosmosis # placeholder for right command.
git clone -b perlmutter cosmosis-standard-library # placeholder for right command.
cd $COSMOSIS_SRC_DIR
export MY_NAME_FOR_A_CONDA_ENV=jack
source setup-cosmosis-nersc /global/common/software/des/common/Conda_Envs/cosmosis-global/cosmosis-global
conda create --yes --name $MY_NAME_FOR_A_CONDA_ENV astropy Cython gsl fftw matplotlib mpich="3.3.*=external_*" mpi    4py ninja numpy PyYAML ripgrep scikit-learn scipy  python=3.9
conda activate $MY_NAME_FOR_A_CONDA_ENV; fix_prompt
# CONDA_PREFIX and COSMOSIS_SRC_DIR are set by the conda activate and the source setup-cosmosis-nersc
export CFITSIO_DIR=$CONDA_PREFIX
export CFITSIO_INC=$CFITSIO_DIR/include
export CFITSIO_LIB=$CFITSIO_DIR/lib
pip install camb emcee kombine future # fitsio but fitsio wont compile on Perlmutter. fake it with cfitsio
conda install cfitsio==3.470

# now the build
cd $COSMOSIS_SRC_DIR
make
cd $CSL_DIR
make

salloc --nodes 1 --qos interactive --time 02:00:00 --constraint gpu --gpus 1 --account=des_g
srun -n 4 python -m mpi4py.bench helloworld # test mpi using benchmark
srun -n 5 cosmosis --mpi demos/demo5.ini    # test python mpi using EMCEE
# change emcee to multinest in demo5.ini
srun -n 5 cosmosis --mpi demos/demo5.ini.   # test compiled fortran mpi using Multinest