Skip to content

installation_using_conda

Jump to bottom
mpaterno edited this page Jun 19, 2022 · 6 revisions

Installation based on conda

Table of Contents

  1. Cautions
  2. Overview
  3. Make sure your Xcode installation is up-to-date
  4. Make sure your Homebrew installation is up-to-date
  5. Obtain the Homebrew requirements
  6. Prepare to build our code and dependencies
  7. Build and use our Y3 galaxy cluster software
  8. For day-to-day work

These are instructions for getting going with an environment based entirely on conda.

Please report any errors in the instructions to Marc.

Cautions and recommendations

We strongly recommend, and can only help support, a conda installation into a conda environment established for use of y3_cluster_cpp.

We recommend using Miniforge conda: https://github.com/conda-forge/miniforge .

If you are installing on macOS, make sure the conda you are using matches your hardware:

  • Intel or x86_64 (these are two names for the same thing)
  • Apple Silicon or M1 or arm64 (these are three names for the same thing)

Trying to use the x86_64 on Apple Silicon laptops will succeed for the first few steps, but eventually fail near the end when the incompatibilities become too hard for Apple's Rosetta 2 to overcome. Trying to use the Apple Silicon build on an Intel-based Mac will probably fail instantly.

Make sure you have neither an active Conda environment nor a Python virtual environment when you begin.

Make sure you do not have PYTHONPATH defined.

To verify that you have no active Conda environment, look at the environment variable CONDA_PREFIX. If you have an active Conda environment, CONDA_PREFIX will be set to the name of the environment. If that variable is not defined, you do not have an active Conda environment. If you have an active environment conda deactivate will deactivate it.

To verify that you have an active Python virtual environment, look at the environment variable VIRTUAL_ENV. If you have an active Python virtual environment, VIRTUAL_ENV will contain the location of the environment. If that variable is not defined, you do not have an active Python virtual environment.

If you have PYTHONPATH set, please unset it:

unset PYTHONPATH

Make sure that you do not already have some other cosmosis on your PATH. If you do, then remove the directory in which it is found from your PATH.

Overview

We build all code using compilers delivered by conda. We install all necessary additional software using conda. We strongly recommend against trying to mix other system-installed software with conda-installed software. Doing so is very tricky.

We assume you already have conda itself available.

Create the conda environment

Creating a conda environment piecemeal can be difficult. There are known issues with the environment solver that can make it fail to install a new package when installing the desired set of packages all at once works.

We will be installing CosmoSIS using conda. However, one can not install the Cosmosis Standard Library (CSL) using conda. We install the tools we need and build our own version of the CSL.

conda create --name for_y3_clusters -c conda-forge cffi cmake configparser cosmosis cosmosis-build-standard-library gperftools kombine ninja openblas pycparser python==3.9

Prepare to build our code and dependencies

Create the directory structure

Because we want to be able to run Y3 galaxy cluster analysis programs on grid nodes, we have taken care to make sure we can use a central installation of CosmoSIS. While this is not necessary for running on a laptop, setting up to do so helps make sure that the code and configurations we generate will work in that environment.

To establish such an environment, we suggest the following organization. The sections below will go through the details of the installation.

  1. Pick a newly-created (empty) directory under which cosmosis-standard-library, y3_cluster_cpp, and supporting software will be installed. We will call this our top directory TOP_DIR.
  2. Build the CosmoSIS standard library under TOP_DIR.
  3. Create a subdirectory of TOP_DIR into which code part of neither CosmoSIS nor y3_cluster_cpp is to go.
  4. Install y3_cluster_cpp into TOP_DIR.

We will assume that the environment variable TOP_DIR is defined to be the full pathname of your top directory. To create the necessary directories, execute the following:

mkdir -p ${TOP_DIR}
mkdir -p ${TOP_DIR}/central # we will install 3rd-party software here

Download and build the CSL

Our next step is to download and build the CosmoSIS Standard Library:

# Make sure you have the conda environment you created above activated
# conda activate for_y3_clusters
cd ${TOP_DIR}
source ${CONDA_PREFIX}/bin/cosmosis-configure
export COSMOSIS_STANDARD_LIBRARY=${TOP_DIR}/cosmosis-standard-library
cosmosis-build-standard-library

Install the CUBA integration library

Next, you need to build libcuba.dylib. There is a Homebrew installation of cuba available; it does not build a dynamic library, and it does not use the optimization level we want. Do not use it. Do the following instead:

cd ${TOP_DIR}/central
mkdir include
mkdir lib
git clone https://github.com/marcpaterno/cuba.git
cd cuba
CC=$CC CFLAGS="-O3 -funsafe-math-optimizations -march=native -fPIC" ./configure
./makesharedlib.sh
cp cuba.h ${TOP_DIR}/central/include
mv libcuba.dylib ${TOP_DIR}/central/lib
cd ${TOP_DIR}
rm -rf ${TOP_DIR}/central/cuba/
cd ${TOP_DIR}/central/lib
install_name_tool -id ${PWD}/libcuba.dylib libcuba.dylib
cd ${TOP_DIR}

Install cluster toolkit

Some of our pipelines use the Python module cluster_toolkit. Installing this is not done with pip; you must do it more manually. The installed Python module will be put into the same location as your pip-installed modules.

pushd ${TOP_DIR}
# make sure your conda environment is active
# and you have source'd the CosmoSIS setup script, as above
mkdir -p cluster_toolkit_tmp
cd cluster_toolkit_tmp
# we use a particular version that is available from the link below
wget https://github.com/marcpaterno/cluster_toolkit/archive/master.tar.gz
tar xf master.tar.gz
cd cluster_toolkit-master/
python setup.py install
popd
rm -rf ${TOP_DIR}/cluster_toolkit_tmp

Install cubacpp

Next, we need cubacpp. This is a header-only library, so all you need to do is clone the repository. There is nothing to build.

cd ${TOP_DIR}/central
git clone git@bitbucket.org:mpaterno/cubacpp.git

Clone y3_cluster_cpp

This is the repository containing our analysis code, and the code you are likely to be modifying as you write your analysis.

cd ${TOP_DIR}
git clone git@bitbucket.org:mpaterno/y3_cluster_cpp.git

Build and use our Y3 galaxy cluster software

We use CMake to generate the build files (either Ninja or make) for y3_cluster_cpp. The resulting build system fully supports parallel builds.

We rely on several environment variables both during the build configuration and while running pipelines. One other needs to be defined "by hand" to a value that is determined by exactly where you have installed the software. This is Y3_CLUSTER_CPP_DIR, as shown below. During the configuration of the build, we also rely upon TOP_DIR, as defined above.

cd ${TOP_DIR}
# Activate the virtual environment and source the cosmosis-configure
# script if you have not already done so.
# source $[TOP_DIR}/local-venv/bin/activate
# source ${TOP_DIR}/local-venv/bin/cosmosis-configure --brew

export Y3_CLUSTER_CPP_DIR=${TOP_DIR}/y3_cluster_cpp
export Y3_CLUSTER_WORK_DIR=${Y3_CLUSTER_CPP_DIR}/release-build
mkdir -p ${Y3_CLUSTER_WORK_DIR}/data
cd  ${Y3_CLUSTER_WORK_DIR}

cmake -DUSE_CUDA=Off -DCMAKE_MODULE_PATH=${TOP_DIR}/central/cubacpp/cmake/modules -DCUBACPP_DIR=${TOP_DIR}/central/cubacpp -DCUBA_DIR=${TOP_DIR}/central -DCMAKE_BUILD_TYPE=Release -G Ninja ${Y3_CLUSTER_CPP_DIR}
ninja # This will compile and link the code.
ctest -j2 # or -j4, if you have a 4-core laptop. Don’t use -j (without a number).

If you encounter a compilation failure that says:

fatal error: 'optional' file not found

this is an indication that your Xcode installation is out-of-date. You will need to upgrade Xcode before you can continue. After upgrading Xcode, you will need to update your Homebrew installation, and then rebuild your Python virtual environment, to make sure all packages are binary compatible.

For day-to-day work

As noted several times above, every time you want to work with CosmoSIS in a new shell session you need to reestablish the correct environment. Assuming you have followed the instructions above, the following commands should do the job. Note that this begins with cd-ing into the directory you established as TOP_DIR during the installation.

export TOP_DIR=${PWD}
conda activate for_y3_clusters
source ${CONDA_PREFIX}/bin/cosmosis-configure
export COSMOSIS_STANDARD_LIBRARY=${TOP_DIR}/cosmosis-standard-library
export Y3_CLUSTER_CPP_DIR=${TOP_DIR}/y3_cluster_cpp
export Y3_CLUSTER_WORK_DIR=${Y3_CLUSTER_CPP_DIR}/release-build

You should be doing most of your work (editing code, building, running cosmosis) from the y3_cluster_cpp directory, or some directory below that. Pay careful attention to what directory you are in when using git commands: different parts of the TOP_DIR directory tree are under version control in different repositories!

Clone this wiki locally