Native macOS build

Cautions

Make sure you have neither an active Conda environment nor a Python virtual environment when you begin. Make sure you do not have PYTHONPATH defined.

To verify that you have no active Conda environment, look at the environment variable CONDA_PREFIX. If you have an active Conda environment, CONDA_PREFIX will be set to the name of the environment. If that variable is not defined, you do not have an active Conda environment.

To verify that you have an active Python virtual environment, look at the environment variable VIRTUAL_ENV. If you have an active Python virtual environment, VIRTUAL_ENV will contain the location of the environment. If that variable is not defined, you do not have an active Python virtual environment.

If you have PYTHONPATH set, please unset it:

unset PYTHONPATH

Make sure that you do not already have some other cosmosis on your PATH. If you do, then remove the directory in which it is found from your PATH.

Overview

We build C and C++ code for macOS using Apple's compilers (clang and clang++) from their Xcode software development suite. You can use either the full Xcode installation or just the command-line tools installation. Because Apple does not provide a Fortran compiler, we use Fortran from the GNU Compiler Collection (GCC). The name of this compiler is gfortran. Installing the GCC also provides a C compiler (gcc) and a C++ compiler (g++). We do not use these compilers.

CosmoSIS relies upon a number of software products that are not included as part of macOS. We rely upon Homebrew to obtain most of these products.

Some of the products we rely upon are installed with Python's pip. We do this installation in a virtual environment. We recommend never doing a pip installation of any Python package outside of a virtual environment.

Make sure your Xcode installation is up-to-date

We rely on several features of C++ that are only available in recent Xcode installations. Homebrew also requires an up-to-date Xcode for correct behavior.

Make sure your Homebrew installation is up-to-date

If you have a Homebrew installation that hasn't been updated in weeks or months, this may take a while.

Failure to keep your Homebrew installation up-to-date is a common cause of build failures, especially of hard-to-diagnose link-time failures. Before submitting a bug report about the macOS build, please make sure you have updated your Homebrew and Xcode installations. The commands to update Homebrew are:

brew update  # obtains the newest version of Homebrew itself
brew upgrade # obtains the newest versions of packages you have installed with Homebrew
brew cleanup # optional; this helps reduce the disk space taken up by Homebrew
brew doctor  # optional; helps to diagnose potential installation problems

Obtain the Homebrew requirements

Note that we are using Python3 from Homebrew, and not conda. Experience shows that a mixed environment of conda and Homebrew Python together is a common source of confusion and errors. If you choose to use conda rather than Homebrew Python, we recommend:

do not install Homebrew Python at all
install all Python modules using conda, not pip.

Unfortunately, we can provide little help in the use of conda.

Caution

Do not install Homebrew, or new packages into an existing Homebrew installation, when you have either a Python virtual environment or a Conda environment active. Doing so may lead to hard-to-diagnose failures in the future.

After you have followed the Homebrew instructions for installing brew, you can cut-and-past the following to install the required Homebrew packages.

# If the following install commands tell you that you already have some of these things
# installed and up-to-date, that is not a problem.
# Note that we install the GNU Compiler Collection (GCC) for the GNU Fortran compiler (gfortran),
# not for the GNU C compiler (gcc) nor for the GNU C++ compiler (g++).
# The following line installs things needed for CosmoSIS and the CosmoSIS standard library
brew install cfitsio fftw gcc git gsl libomp libyaml minuit2 openblas openmpi python
# The following line install things needed by y3_cluster_cpp
brew install cmake gperftools wget ninja wget

Prepare to build our code and dependencies

Create the directory structure

Because we want to be able to run Y3 galaxy cluster analysis programs on grid nodes, we have taken care to make sure we can use a central installation of CosmoSIS. While this is not necessary for running on a laptop, setting up to do so helps make sure that the code and configurations we generate will work in that environment.

To establish such an environment, we suggest the following organization. The sections below will go through the details of the installation.

Pick a newly-created (empty) directory under which y3_cluster_cpp, and supporting software will be installed. We will call this our top directory TOP_DIR.
Create a Python virtual environment in TOP_DIR.
Install CosmoSIS and many other productions using pip into the virtual environment.
Build the CosmoSIS standard library under TOP_DIR.
Create a subdirectory of TOP_DIR into which code part of neither CosmoSIS nor y3_cluster_cpp is to go.
Install y3_cluster_cpp into TOP_DIR.

We will assume that the environment variable TOP_DIR is defined to be the full pathname of your top directory. To create the necessary directories, execute the following:

mkdir -p ${TOP_DIR}
mkdir -p ${TOP_DIR}/central # we will install 3rd-party software here

Install CosmoSIS the Python modules we require

CosmoSIS needs a few other python modules that can not be installed using brew. We will install them in a Python virtual environment.

We recommend setting up a dedicated Python environment before installing software specific to cluster analyses using this package. Note that we install CosmoSIS itself using pip. This is new as of the release of CosmoSIS v2.0.

The following commands will install CosmoSIS and additional Python packages required by both the CosmoSIS Standard Library and our y3_cluster_cpp pipelines. If you need to install any additional Python packages for work with y3_cluster_cpp make sure to install them in this same environment.

cd ${TOP_DIR}
python3 -m venv local-venv # Make sure you use python3 not python
source local-venv/bin/activate                    # After this python *is* python3
python -m pip install --upgrade pip setuptools wheel
python -m pip install astropy camb cffi configparser cosmosis cython fast-pt fitsio kombine mpi4py pycparser scikit-learn six
source local-venv/bin/cosmosis-configure --brew
# As of this writing, the current (v2.0.1) version of CosmosSIS does not have
# the correct install names in the libraries it builds. We have to patch them.
# If you ever rebuild the CosmoSIS libraries you will have to do this patching
# again.
cd ${COSMOSIS_SRC_DIR}
for lib in $(find . -name '*.so'); do install_name_tool -id $(realpath $lib) $lib; done

To deactivate the virtual environment use:

deactivate  # Now python is the system python again

This has created a virtual environment named local-venv. If you need to exit this environment use the deactivate command. In the future, you will have to source the activate script to set up CosmoSIS.

Our next step is to download and build the CosmoSIS Standard Library:

cd ${TOP_DIR}
# Skip the sourcing of the activate script if you did *not* deactivate the environment
source local-venv/bin/activate
source local-venv/bin/cosmosis-configure --brew
export COSMOSIS_STANDARD_LIBRARY=${TOP_DIR}/cosmosis-standard-library
git clone https://github.com/joezuntz/cosmosis-standard-library
cd ${COSMOSIS_STANDARD_LIBRARY}
# The following line does the actual build. Do not try to use a parallel make;
# the CSL Makefiles do not support a parallel build.
make

The build may generate some warnings but there should be no errors.

At this point, you should have a fully functional CosmoSIS installation. You may wish to run some of the demos to verify this.

Install the CUBA integration library

Next, you need to build libcuba.dylib. There is a Homebrew installation of cuba available; it does not build a dynamic library, and it does not use the optimization level we want. Do not use it. Do the following instead:

cd ${TOP_DIR}/central
mkdir include
mkdir lib
git clone https://github.com/marcpaterno/cuba.git
cd cuba
CC=clang CFLAGS="-O3 -funsafe-math-optimizations -march=native -fPIC" ./configure
./makesharedlib.sh
cp cuba.h ${TOP_DIR}/central/include
mv libcuba.dylib ${TOP_DIR}/central/lib
cd ${TOP_DIR}
rm -rf ${TOP_DIR}/central/cuba/
cd ${TOP_DIR}/central/lib
install_name_tool -id ${PWD}/libcuba.dylib libcuba.dylib
cd ${TOP_DIR}

Install cluster toolkit

Some of our pipelines use the Python module cluster_toolkit. Installing this is not done with pip; you must do it more manually. The installed Python module will be put into the same location as your pip-installed modules.

pushd ${TOP_DIR}
# source local-venv/bin/activate,  if you haven't done so,
mkdir -p cluster_toolkit_tmp
cd cluster_toolkit_tmp
# we use a particular version that is available from the link below
wget https://github.com/marcpaterno/cluster_toolkit/archive/master.tar.gz
tar xf master.tar.gz
cd cluster_toolkit-master/
python setup.py install
popd
rm -rf ${TOP_DIR}/cluster_toolkit_tmp

Install cubacpp

Next, we need cubacpp. This is a header-only library, so all you need to do is clone the repository. There is nothing to build.

cd ${TOP_DIR}/central
git clone git@bitbucket.org:mpaterno/cubacpp.git

Clone y3_cluster_cpp

This is the repository containing our analysis code, and the code you are likely to be modifying as you write your analysis.

cd ${TOP_DIR}
git clone git@bitbucket.org:mpaterno/y3_cluster_cpp.git

Build and use our Y3 galaxy cluster software

We use CMake to generate the build files (either Ninja or make) for y3_cluster_cpp. The resulting build system fully supports parallel builds.

We rely on several environment variables both during the build configuration and while running pipelines. One other needs to be defined "by hand" to a value that is determined by exactly where you have installed the software. This is Y3_CLUSTER_CPP_DIR, as shown below. During the configuration of the build, we also rely upon TOP_DIR, as defined above.

cd ${TOP_DIR}
# Activate the virtual environment and source the cosmosis-configure
# script if you have not already done so.
# source $[TOP_DIR}/local-venv/bin/activate
# source ${TOP_DIR}/local-venv/bin/cosmosis-configure --brew

export Y3_CLUSTER_CPP_DIR=${TOP_DIR}/y3_cluster_cpp
export Y3_CLUSTER_WORK_DIR=${Y3_CLUSTER_CPP_DIR}/release-build
mkdir -p ${Y3_CLUSTER_WORK_DIR}/data
cd  ${Y3_CLUSTER_WORK_DIR}

cmake -DUSE_CUDA=Off -DCMAKE_MODULE_PATH=${TOP_DIR}/central/cubacpp/cmake/modules -DCUBACPP_DIR=${TOP_DIR}/central/cubacpp -DCUBA_DIR=${TOP_DIR}/central -DCMAKE_BUILD_TYPE=Release -G Ninja ${Y3_CLUSTER_CPP_DIR}
ninja # This will compile and link the code.
ctest -j2 # or -j4, if you have a 4-core laptop. Don’t use -j (without a number).

If you encounter a compilation failure that says:

fatal error: 'optional' file not found

this is an indication that your Xcode installation is out-of-date. You will need to upgrade Xcode before you can continue. After upgrading Xcode, you will need to update your Homebrew installation, and then rebuild your Python virtual environment, to make sure all packages are binary compatible.

For day-to-day work

As noted several times above, every time you want to work with CosmoSIS in a new shell session you need to reestablish the correct environment. Assuming you have followed the instructions above, the following commands should do the job. Note that this begins with cd-ing into the directory you established as TOP_DIR during the installation.

export TOP_DIR=${PWD}
source local-venv/bin/activate
export COSMOSIS_SRC_DIR=$(python -c "import os; import cosmosis; print(os.path.dirname(cosmosis.__file__))")
export COSMOSIS_STANDARD_LIBRARY=${TOP_DIR}/cosmosis-standard-library
export COSMOSIS_ALT_COMPILERS=1
export GSL_INC=/usr/local/include
export GSL_LIB=/usr/local/lib
export CFITSIO_INC=/usr/local/include
export CFITSIO_LIB=/usr/local/lib
export FFTW_LIBRARY=/usr/local/include
export FFTW_INCLUDE_DIR=/usr/local/lib
export LAPACK_LINK="-L/usr/local/opt/openblas/lib -llapack"
export Y3_CLUSTER_CPP_DIR=${TOP_DIR}/y3_cluster_cpp
export Y3_CLUSTER_WORK_DIR=${Y3_CLUSTER_CPP_DIR}/release-build

You should be doing most of your work (editing code, building, running cosmosis) from the y3_cluster_cpp directory, or some directory below that. Pay careful attention to what directory you are in when using git commands: different parts of the TOP_DIR directory tree are under version control in different repositories!

Crash dumps on macOS

Whenever a user-mode program (such as cosmosis) terminates unexpectedly (as by a call to abort), macOS stores information about that crash in a log file. You can find the log files in the directory ~/Library/Logs/DiagnosticReports. The file names end in .crash. The beginning of the file name is the name of the running program; for cosmosis, that will be Python (or maybe python, depending on your installation). The filenames also carry a timestamp and an identifier of your machine.

The file is a plain ASCII text file and may contain useful information. In particular, if the program being run contained debug symbols, a stack trace of the program is provided. This is sufficient for some debugging purposes.

Debugging C++ code on macOS

Now that you have a working installation, you might want to run a debugger on your C++ code. After you've done the setup you would use for any other CosmoSIS work, you can use the lldb debugger. Note that this debugger has very different commands from the gdb debugger. A good resource for learning to use lldb is https://lldb.llvm.org/use/map.html. To run a pipeline and have access to code in the debugger, use:

lldb python -- $(which cosmosis) <configuration file>

When the program starts up, you are set up to debug the Python interpreter itself. That is probably not what you want to do. To get to the code you are interested in, it is best to set a breakpoint and then run to that code. Setting breakpoints on C++ functions can be tricky. Here is an example that works. Note that most of the lines are printout from lldb. Input you are to type to the lldb prompt is indicated by "==> "; do not type those symbols at the lldb prompt!

$ lldb python -- $(which cosmosis) y1_rerun/sigma_y1.ini
(lldb) target create "python"
Current executable set to 'python' (x86_64).
(lldb) settings set -- target.run-args  "/Users/paterno/MyProjects/new-cosmosis-playground/cosmosis/bin/cosmosis" "y1_rerun/sigma_mort.ini"
==> (lldb) breakpoint set --name 'y3_cluster::CosmoSISScalarIntegrationModule<SigmaCentY1ScalarIntegrand>::execute(cosmosis::DataBlock&)'
Breakpoint 1: no locations (pending).
WARNING:  Unable to resolve breakpoint to any actual locations.
==> (lldb) run
Process 74122 launched: '/usr/local/bin/python' (x86_64)
Process 74122 stopped
* thread #2, stop reason = exec
    frame #0: 0x0000000100005000 dyld`_dyld_start
dyld`_dyld_start:
->  0x100005000 <+0>: popq   %rdi
    0x100005001 <+1>: pushq  $0x0
    0x100005003 <+3>: movq   %rsp, %rbp
    0x100005006 <+6>: andq   $-0x10, %rsp
Target 0: (Python) stopped.
==> (lldb) continue
Process 74122 resuming
1 location added to breakpoint 1

<many lines of CosmoSIS chatter elided here...>
Process 74122 stopped
* thread #2, queue = 'com.apple.main-thread', stop reason = breakpoint 1.1
    frame #0: 0x000000011c7040b4 SigmaCentY1ScalarIntegrand.so`y3_cluster::CosmoSISScalarIntegrationModule<SigmaCentY1ScalarIntegrand>::execute(this=0x00000001003d2570, sample=0x000000011c635450) at CosmoSISScalarIntegrationModule.hh:128:25
   125 	y3_cluster::CosmoSISScalarIntegrationModule<I>::execute(
   126 	  cosmosis::DataBlock& sample)
   127 	{
-> 128 	  integrand_.set_sample(sample);
   129 	  auto results = use_cartesian_product_of_volumes_and_gridpoints_ ?
   130 	                   integrate_cartesian_product_of_volumes_and_gridpoints() :
   131 	                   integrate_zipped_sequence_of_volumes_and_gridpoints();
Target 0: (Python) stopped.
(lldb)

At this point, you are at the lldb debugger prompt at the start of the function execute in the module of interest.

Navigation Menu

Native macOS build

Native macOS build

Table of Contents

Cautions

Overview

Make sure your Xcode installation is up-to-date

Make sure your Homebrew installation is up-to-date

Obtain the Homebrew requirements

Prepare to build our code and dependencies

Create the directory structure

Install CosmoSIS the Python modules we require

Install the CUBA integration library

Install cluster toolkit

Install cubacpp

Clone y3_cluster_cpp

Build and use our Y3 galaxy cluster software

For day-to-day work

Crash dumps on macOS

Debugging C++ code on macOS

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Clone this wiki locally