build_on_perlmutter

Just run the y3_cluster_cpp, assuming everything else is built

# working directory, use high performance shared area on perimeter
export TOP_DIR =  /global/common/software/des/$(id --user --name)
export COSMOSIS_SRC_DIR=$TOP_DIR/cosmosis
export CSL_DIR=$TOP_DIR/cosmosis-standard-library

export TOPDIR=$TOP_DIR/y3_pipe_under
export Y3PIPE_DIR=$TOP_DIR/y3_cluster_cpp
export Y3_CLUSTER_WORK_DIR=$Y3PIPE_DIR/release-build

export Y3_CLUSTER_CPP_DIR=$Y3PIPE_DIR
export COSMOSIS_STANDRD_LIBRARY=$CSL_DIR
cd $COSMOSIS_SRC_DIR
source setup-cosmosis-nersc /global/common/software/des/common/Conda_Envs/cosmosis-global/cosmosis-global

salloc --nodes 1 --qos interactive --time 02:00:00 --constraint gpu --gpus 1 --account=des_g
cd $Y3PIPE_DIR/y1_rerun
srun -n1 cosmosis --mpi sigma_mort_mcubes.ini

But I need to compile y3_cluster_cpp

do the exports and source setup-cosmosis-nersc, then

# get and make
https://github.com/Kitware/CMake/releases/download/v3.23.0/cmake-3.23.0-linux-x86_64.tar.gz

cluster toolkit

mkdir -p ${TOPDIR}/tmp
cd ${TOPDIR}/tmp
wget https://github.com/marcpaterno/cluster_toolkit/archive/master.tar.gz
tar xf master.tar.gz
cd cluster_toolkit-master/
python setup.py install     # This will install into the environment

The other 3

cd ${TOPDIR}
git clone https://github.com/marcpaterno/cuba.git
cd cuba
./configure
./makesharedlib.sh
mkdir include; mkdir lib 
mv cuba.h include/ ; mv libcuba.so lib/
export CUBA_DIR=$TOPDIR/cuba

cd ${TOPDIR} 
git clone http://bitbucket.org/mpaterno/cubacpp.git
export CUBA_CPP_DIR=$TOPDIR/cubacpp

cd ${TOPDIR} 
git clone http://github.com/marcpaterno/gpuintegration.git
export GPU_INT_DIR=$TOPDIR/gpuintegration
mkdir build; cd build
cmake -DPAGANI_DIR=$GPU_INT_DIR -DCMAKE_BUILD_TYPE=Release -DPAGANI_TARGET_ARCH=80-real -G Ninja ..
ninja
ctest

y3_cluster_cpp

cd ${Y3PIPE_DIR}
git clone http://bitbucket.org/mpaterno/y3_cluster_cpp.git
cd ${Y3PIPE_DIR}/y3_cluster_cpp
mkdir release-build
cd release-build/
cmake -DUSE_CUDA=On -DY3GCC_TARGET_ARCH=80-real -DPAGANI_DIR=${PAGANI_DIR} -DGSL_ROOT_DIR=$CONDA_PREFIX -DCMAKE_MODULE_PATH="$CUBA_CPP_DIR/cmake/modules" -DCUBACPP_DIR=$CUBA_CPP_DIR -DCUBA_DIR=$CUBA_DIR  -DCMAKE_BUILD_TYPE=Release  -G Ninja ../
ninja

But I have to make cosmosis and cosmosis standard library!

Deep breath. Take one. And another.

cd $COSMOSIS_SRC_DIR
export MY_NAME_FOR_A_CONDA_ENV=jack
source setup-cosmosis-nersc /global/common/software/des/common/Conda_Envs/cosmosis-global/cosmosis-global
conda create --yes --name $MY_NAME_FOR_A_CONDA_ENV astropy Cython gsl fftw matplotlib mpich="3.3.*=external_*" mpi    4py ninja numpy PyYAML ripgrep scikit-learn scipy  python=3.9
conda activate $MY_NAME_FOR_A_CONDA_ENV; fix_prompt
# CONDA_PREFIX and COSMOSIS_SRC_DIR are set by the conda activate and the source setup-cosmosis-nersc
export CFITSIO_DIR=$CONDA_PREFIX
export CFITSIO_INC=$CFITSIO_DIR/include
export CFITSIO_LIB=$CFITSIO_DIR/lib
pip install camb emcee kombine future # fitsio but fitsio wont compile on Perlmutter. fake it with cfitsio
conda install cfitsio==3.470

# now the build
cd $COSMOSIS_SRC_DIR
make
cd $CSL_DIR
make

salloc --nodes 1 --qos interactive --time 02:00:00 --constraint gpu --gpus 1 --account=des_g
srun -n 4 python -m mpi4py.bench helloworld # test mpi using benchmark
srun -n 5 cosmosis --mpi demos/demo5.ini    # test python mpi using EMCEE
# change emcee to multinest in demo5.ini
srun -n 5 cosmosis --mpi demos/demo5.ini.   # test compiled fortran mpi using Multinest

Clone the cosmosis and cosmosis_standard_library repositories

We put everything we are going to build on; create the directory if it does not exit.

/global/common/software/des/$(id --user --name)