build_on_perlmutter

Just set up to run the y3_cluster_cpp, assuming everything else is built

# working directory, use high performance shared area on perlmutter

# Note that we have both TOPDIR and TOP_DIR; be careful!
export TOP_DIR=/global/common/software/des/$(id --user --name)
export COSMOSIS_SRC_DIR=${TOP_DIR}/cosmosis
export CSL_DIR=${TOP_DIR}/cosmosis-standard-library

export TOPDIR=${TOP_DIR}/y3_pipe_under
export Y3PIPE_DIR=${TOP_DIR}/y3_cluster_cpp
export Y3_CLUSTER_WORK_DIR=${Y3PIPE_DIR}/release-build
export Y3_CLUSTER_CPP_DIR=${Y3PIPE_DIR}
export COSMOSIS_STANDRD_LIBRARY=${CSL_DIR}
    
cd ${COSMOSIS_SRC_DIR}
source setup-cosmosis-nersc /global/common/software/des/common/Conda_Envs/cosmosis-global

salloc --nodes 1 --qos interactive --time 02:00:00 --constraint gpu --gpus 1 --account=des_g
cd ${Y3PIPE_DIR}/y1_rerun
    
srun -n1 cosmosis --mpi sigma_mort_mcubes.ini

But I need to compile y3_cluster_cpp

Do the exports and source setup-cosmosis-nersc, then:

# get cmake
wget https://github.com/Kitware/CMake/releases/download/v3.23.0/cmake-3.23.0-linux-x86_64.tar.gz
# Untar this, which will create a directory cmake-3.23.0-linux-x86_64, 
# which will contain a bin directory. Put that bin directory onto your PATH.

Build cluster_toolkit and install it into the conda environment

mkdir -p ${TOPDIR}/tmp
cd ${TOPDIR}/tmp
wget https://github.com/marcpaterno/cluster_toolkit/archive/master.tar.gz
tar xf master.tar.gz
cd cluster_toolkit-master/
python setup.py install     # This will install into the environment
cd ${TOP_DIR}

Build and install cuba

cd ${TOPDIR}
git clone https://github.com/marcpaterno/cuba.git
cd cuba
./configure
./makesharedlib.sh
mkdir include; mkdir lib 
mv cuba.h include/ ; mv libcuba.so lib/
export CUBA_DIR=${TOPDIR}/cuba

Clone cubacpp

Note that you do not have to build cubapp. If you are doing development work on cubacpp you will want to run tests; then also follow the optional build instructions.

cd ${TOPDIR} 
git clone http://bitbucket.org/mpaterno/cubacpp.git
export CUBA_CPP_DIR=$TOPDIR/cubacpp

Optional build instructions for cubacpp

Coming soon!

Clone `gpuintegration'

Note that you do not have to build gpuintegration. If you are doing development work on gpuintegration you will want to run tests; then also follow the optional build instructions.

cd ${TOPDIR} 
git clone http://github.com/marcpaterno/gpuintegration.git
export GPU_INT_DIR=${TOPDIR}/gpuintegration
mkdir build
cd build
cmake -DPAGANI_DIR=${GPU_INT_DIR} -DCMAKE_BUILD_TYPE=Release -DPAGANI_TARGET_ARCH=80-real -G Ninja ..
ninja
ctest

Optional build instructions for gpuintegration

Coming soon!

Clone and build y3_cluster_cpp

I bet you thought we'd never get here, huh?

cd ${Y3PIPE_DIR}
git clone http://bitbucket.org/mpaterno/y3_cluster_cpp.git

cd ${Y3PIPE_DIR}/y3_cluster_cpp
mkdir release-build
cd release-build/
cmake -DUSE_CUDA=On -DY3GCC_TARGET_ARCH=80-real -DPAGANI_DIR=${PAGANI_DIR} -DGSL_ROOT_DIR=$CONDA_PREFIX -DCMAKE_MODULE_PATH="$CUBA_CPP_DIR/cmake/modules" -DCUBACPP_DIR=$CUBA_CPP_DIR -DCUBA_DIR=$CUBA_DIR  -DCMAKE_BUILD_TYPE=Release  -G Ninja ../
ninja

But I have to make CosmoSIS and CosmoSIS standard library!

Deep breath. Take one. And another.

cd $TOP_DIR
git clone cosmosis # placeholder for right command.
git clone cosmosis-standard-library # placeholder for right command.
cd $COSMOSIS_SRC_DIR
export MY_NAME_FOR_A_CONDA_ENV=jack
source setup-cosmosis-nersc /global/common/software/des/common/Conda_Envs/cosmosis-global/cosmosis-global
conda create --yes --name $MY_NAME_FOR_A_CONDA_ENV astropy Cython gsl fftw matplotlib mpich="3.3.*=external_*" mpi    4py ninja numpy PyYAML ripgrep scikit-learn scipy  python=3.9
conda activate $MY_NAME_FOR_A_CONDA_ENV; fix_prompt
# CONDA_PREFIX and COSMOSIS_SRC_DIR are set by the conda activate and the source setup-cosmosis-nersc
export CFITSIO_DIR=$CONDA_PREFIX
export CFITSIO_INC=$CFITSIO_DIR/include
export CFITSIO_LIB=$CFITSIO_DIR/lib
pip install camb emcee kombine future # fitsio but fitsio wont compile on Perlmutter. fake it with cfitsio
conda install cfitsio==3.470

# now the build
cd $COSMOSIS_SRC_DIR
make
cd $CSL_DIR
make

salloc --nodes 1 --qos interactive --time 02:00:00 --constraint gpu --gpus 1 --account=des_g
srun -n 4 python -m mpi4py.bench helloworld # test mpi using benchmark
srun -n 5 cosmosis --mpi demos/demo5.ini    # test python mpi using EMCEE
# change emcee to multinest in demo5.ini
srun -n 5 cosmosis --mpi demos/demo5.ini.   # test compiled fortran mpi using Multinest