Skip to content

Setting up to use y3_cluster_cpp through spack and CVMFS

Jump to bottom
Marc Paterno edited this page Oct 30, 2020 · 4 revisions

Purpose of this document

This page describes how to set up to use y3_cluster_cpp, obtaining access to CosmoSIS through Spack, on any system that has access to CVMFS. This includes Fermilab DES interactive nodes and grid nodes, among others.

Unlike the other methods of setting up y3_cluster_cpp, this method does not involve building CosmoSIS. You will need to build only the y3_cluster_cpp-specific software (including CUDA, which is not currently available through Spack and CVMFS).

Setup spack and cosmosis (for development)

source /cvmfs/des.opensciencegrid.org/users/cosmosis/neutrinoless_mass_function_2-20200630/setup-env.sh
spack load -r gcc@8.2.0
spack load -r cosmosis@neutrinoless_mass_function_2
spack load -r cmake

Create your working directory

Create a new (empty) directory somewhere, that will be used for your development and as the place from which you may launch grid jobs. cd to that directory, and define the environmenet variable Y3GCC_DIR to point to that directory. The choice below is an example, not a requirement:

mkdir /data/des61.a/data/${USER}/y3_cluster_work
cd /data/des61.a/data/${USER}/y3_cluster_work

export Y3GCC_DIR=$PWD
export Y3_CLUSTER_CPP_DIR=${Y3GCC_DIR}/y3_cluster_cpp
export Y3_CLUSTER_WORK_DIR=${Y3GCC_DIR}/y3_cluster_cpp

Clone repositories

git clone git@github.com:marcpaterno/cuba.git
git clone git@bitbucket.org:mpaterno/y3_cluster_cpp.git
git clone git@bitbucket.org:mpaterno/cubacpp.git

Create the Python virtual environment and install Python modules

Note that we are not using pip to install these python modules. This is because the grid submission tarball, which we will create later, requires the source of these python modules, to build them on the grid execution node. By downloading these and building from source, we assure that your development environment has the identical version of the modules that will be used in grid jobs.

python -m venv --system-site-packages local-venv
source local-venv/bin/activate

wget https://files.pythonhosted.org/packages/0f/86/e19659527668d70be91d0369aeaa055b4eb396b0f387a4f92293a20035bd/pycparser-2.20.tar.gz
wget https://files.pythonhosted.org/packages/f7/09/88bbe20b76ca76be052c366fe77aa5e3cd6e5f932766e5597fecdd95b2a8/cffi-1.14.2.tar.gz
wget https://github.com/marcpaterno/cluster_toolkit/archive/master.tar.gz
tar xf pycparser-2.20.tar.gz
tar xf cffi-1.14.2.tar.gz
tar xf master.tar.gz
cd pycparser-2.20
python setup.py install
cd $Y3GCC_DIR
cd cffi-1.14.2
python setup.py install
cd $Y3GCC_DIR
cd cluster_toolkit-master/
python setup.py install
cd $Y3GCC_DIR
rm -r cffi-1.14.2/ cluster_toolkit-master/ pycparser-2.20/
deactivate

Build CUBA

cd cuba
CC=$(which gcc) ./configure
./makesharedlib.sh
mkdir include
mkdir lib
mv cuba.h include/
mv libcuba.so lib/
cd $Y3GCC_DIR
export LD_LIBRARY_PATH=${Y3GCC_DIR}/cuba/lib:$LD_LIBRARY_PATH

Build y3_cluster_cpp

We are doing an in-source build, because that makes it easier to tar up the result for delivery to a grid node.

cmake -DCMAKE_MODULE_PATH="${Y3_CLUSTER_CPP_DIR}/cmake;${Y3GCC_DIR}/cubacpp/cmake/modules" -DCUBACPP_DIR=${Y3GCC_DIR}/cubacpp -DCUBA_DIR=${Y3GCC_DIR}/cuba  -DCMAKE_BUILD_TYPE=Release  ${Y3GCC_DIR}
make -j 8 # Adjust the degree of parallelism to match your machine

Setting up a new session

Many of these steps only need to be done once in a given directory. Some of the steps contain bits that establish the correct (shell) evironment for work; these bits need to be done for every shell session. The required commands for this are collected here:

export Y3GCC_DIR=<whatever directory you used in the installation>
cd $Y3GCC_DIR
source /cvmfs/des.opensciencegrid.org/users/cosmosis/neutrinoless_mass_function_2-20200630/setup-env.sh
spack load -r gcc@8.2.0
spack load -r cosmosis@neutrinoless_mass_function_2
spack load -r cmake
export Y3_CLUSTER_CPP_DIR=${Y3GCC_DIR}/y3_cluster_cpp
export Y3_CLUSTER_WORK_DIR=${Y3GCC_DIR}/y3_cluster_cpp
source local-venv/bin/activate
export LD_LIBRARY_PATH=${Y3GCC_DIR}/cuba/lib:$LD_LIBRARY_PATH

Creating a grid job tarball

To submit a grid job, you will need a grid job submission tarball containing your build of the y3_cluster_cpp code. Make sure your tarball is up-to-date with your code before you submit a grid job!

The following command will create the required tarball.

tar --exclude="CMakeCache.txt" --exclude="CTestCostData.txt" --exclude="CMakeFiles" --exclude="CMakeLists.txt" -j -c -f job.tar.bz2 $(find . -name '*.so' -o -name '*.py' -o -name '*.txt' -o -name '*.ini') pycparser-2.20.tar.gz cffi-1.14.2.tar.gz master.tar.gz

Clone this wiki locally