Skip to content

Native macOS build

Jump to bottom
Marc Paterno edited this page Mar 27, 2020 · 59 revisions

These are instructions for getting going with the macOS native build, and our Y3 galaxy cluster analysis software.

This macOS Native installation works with macOS version of 10.14+ (Mojave), and especially Xcode Version 10+ (comes with the macOS Mojave updates). Do not try to build on systems with earlier versions. It won't work.

Read the instructions carefully. Do not skip any steps.

Please report any errors in the instructions to Marc (paterno@fnal.gov).

Overview

For macOS, we build C and C++ code using Apple's compilers (clang and clang++), from their Xcode software development suite. Because Apple does not provide a Fortran compiler, we use Fortran from the GNU Compiler Collection (GCC). The name of this compiler is gfortran. Installing the GCC also provides a C compiler (gcc) and a C++ compiler (g++). We do not use these compilers.

CosmoSIS relies upon a number of software products that are not included as part of macOS. To obtain most of these products we rely upon (Homebrew)[https://brew.sh].

Some of the products we rely upon are installed with Pythn's pip. Because we use Python3, we rely upon pip3.

Make sure your Xcode installation is up-to-date

We rely on several features of C++ that are only available in recent Xcode installations. Homebrew also requires an up-to-date Xcode for correct behavior.

Make sure your Homebrew installation is up-to-date

If you have a Homebrew installation that hasn't been updated in weeks or months, this may take a while. Failure to keep your Homebrew installation up-to-date is a common cause of build failures, especially of hard-to-diagnose link-time failures.

#!bash
brew update
brew upgrade
brew cleanup # optional; this helps reduce the disk space taken up by Homebrew

Obtain the Homebrew requirements

Note that we are using Python3 from Homebrew, and not Anaconda Python. Experience shows that a mixed environment of Anaconda Python and Homebrew Python is a common source of confusion and errors. If you choose to use Anaconda Python rather than Homebrew Python, we recommend:

  • do not install Homebrew Python at all
  • install all Python modules using conda, not pip3

Unfortunately, we can provide little help in the use of Anaconda Python.

#!bash
# If the following install commands tell you that you already have some of these things
# installed and up-to-date, that is not a problem.
# Note that we install the GNU Compiler Collection (GCC) for the GNU Fortran compiler (gfortran),
# not for the GNU C compiler (gcc) nor for the GNU C++ compiler (g++).
brew install git gcc gsl cfitsio fftw minuit2 openblas # these are required for CosmoSIS itself
brew install eigen cmake gperftools                    # these are required for Y3 galaxy cluster code
brew install python3 wget

Prepare to build CosmoSIS and y3_cluster_cpp

Because we want to be able to run Y3 galaxy cluster analysis programs on grid nodes, we have taken care to make sure we can use a central installation of CosmoSIS. While this is not necessary for running on a laptop, setting up to do so helps make sure that the code and configurations we generate will work in that environment.

To establish such an environment, we suggest the following organization.

  1. Pick a newly-created (empty) directory under which CosmoSIS, y3_cluster_cpp, and supporting products will be installed. We will call this our top directory TOPDIR.
  2. Install CosmoSIS into TOPDIR.
  3. Create a subdirectory of TOPDIR into which code part of neither CosmoSIS nor y3_cluster_cpp is to go.
  4. Install y3_cluster_cpp into TOPDIR.

We will assume that the environment variable TOPDIR is defined to be the full pathname of your top directory.

#!bash
mkdir -p ${TOPDIR}
mkdir -p ${TOPDIR}/central # we will install 3rd-party software here

Install CosmoSIS and the CSL

We use the develop branch of CosmoSIS and a special branch of the CosmoSIS Standard Library (CSL).

#!bash
cd ${TOPDIR}
git clone http://bitbucket.org/joezuntz/cosmosis
cd cosmosis
git checkout develop
git clone http://bitbucket.org/joezuntz/cosmosis-standard-library
pushd cosmosis-standard-library
git checkout neutrinoless_mass_function # This branch has an update CSL module we require
popd
source config/setup-cosmosis-mac
pip3 install --user -r config/requirements.txt
make # This builds cosmosis and the CSL, but not our Y3 galaxy cluster software

At this point, you should have a fully functional CosmoSIS installation. You may wish to run some of the demos to verify this.

Install the CUBA integration library

Next, you need to build libcuba.dylib. There is a Homebrew installation of cuba available; it does not build a dynamic library, and it does not use the optimization level we want. Do not use it. Do the following instead:

#!bash
pushd ${TOPDIR}/central
mkdir include
mkdir lib
wget http://www.feynarts.de/cuba/Cuba-4.2.tar.gz
tar xf Cuba-4.2.tar.gz
rm Cuba-4.2.tar.gz
cd Cuba-4.2
CC=clang CFLAGS="-O3 -funsafe-math-optimizations -march=native -fPIC" ./configure
cp cuba.h ${TOPDIR}/central/include
make lib
ar x libcuba.a
clang -shared *.o -o libcuba.dylib
mv libcuba.dylib ${TOPDIR}/central/lib
popd
rm -r ${TOPDIR}/central/Cuba-4.2

To install cluster_toolkit

Some Y3 galaxy cluster pipelines use the Python module cluster_toolkit. Installing this is not done with pip3; you must do it more manually. The installed Python module will be put into the same location as your pip3-installed modules. Note that there is a quirk in the installation script that requires an unusual syntax. Make sure you do not put a space after the final = when running setup.py below.

#!bash
pushd ${TOPDIR}
mkdir -p cluster_toolkit_tmp
cd cluster_toolkit_tmp
git clone https://github.com/tmcclintock/cluster_toolkit.git
cd cluster_toolkit
# Note that there must *not* be a space after the = on the next command
python3 setup.py install --user --prefix=
popd
rm -rf ${TOPDIR}/cluster_toolkit_tmp

Clone y3_cluster_cpp

This is the repository containing our analysis code, and the only code you are likely to be modifying as you write your analysis.

#!bash
cd ${TOPDIR}
git clone git@bitbucket.org:mpaterno/y3_cluster_cpp.git

Install cubacpp

Next, we need cubacpp. This is a header-only library, so all you need to do is clone the repository. We do this in externals directory of y3_cluster_cpp.

#!bash
cd ${TOPDIR}/y3_cluster_cpp/externals
git clone git@bitbucket.org:mpaterno/cubacpp.git

To build and use our Y3 galaxy cluster software

Unlike CosmoSIS and the CSL, y3_cluster_cpp is built using makefiles generated by CMake. The resulting makefiles fully support parallel builds.

We rely on several environment variables both during the build configuration and while running pipelines. Some of these environment variables are defined when you source the setup_cosmosis_mac script. One other needs to be defined "by hand", to a value that is determined by exactly where you have installed the software. This is Y3_CLUSTER_CPP_DIR, as shown below. During the configuration of the build, we also rely upon TOPDIR, as define above.

#!bash
cd ${TOPDIR}/cosmosis
source config/setup-cosmosis-mac # only if you haven’t already done it in this shell session
cd ${TOPDIR}/y3_cluster_cpp
export Y3_CLUSTER_CPP_DIR=${PWD}
cmake -DCMAKE_MODULE_PATH="./CMake;${Y3_CLUSTER_CPP_DIR}/externals/cubacpp/cmake/modules" -DCUBACPP_DIR=${Y3_CLUSTER_CPP_DIR}/externals/cubacpp -DCUBA_DIR=${TOPDIR}/central -DCMAKE_BUILD_TYPE=Release .

(If you are using the previously installed version without TOPDIR set up, do so instead

cmake -DCMAKE_MODULE_PATH="${COSMOSIS_SRC_DIR}/cosmosis-standard-library/y3_cluster_cpp/CMake;${COSMOSIS_SRC_DIR}/cubacpp/cmake/modules" -DCUBACPP_DIR=${COSMOSIS_SRC_DIR}/cubacpp -DCUBA_DIR=/usr/local -DCMAKE_BUILD_TYPE=Release .
make -j2  # or -j4, if you have a 4-core laptop. Don’t use -j (without a number).
ctest -j2 # or -j4, if you have a 4-core laptop. Don’t use -j (without a number).

If you encounter a compiation failure that says:

fatal error: 'optional' file not found

this is an indication that your Xcode installation is out-of-date. You will need to upgrade Xcode before you can continue.

For day-to-day work

As noted several times above, every time you want to work with CosmoSIS in a new shell session, you need to source the CosmoSIS setup script. You will also need to define Y3_CLUSTER_CPP_DIR to point to the same directory as was done above. You may find it useful to have TOPDIR defined.

#!bash
cd ${TOPDIR}/CosmoSIS # so that the cosmosis setup script will work.
source config/setup-cosmosis-mac # only if you haven’t already done it in this shell session.
cd ${Y3_CLUSTER_CPP_DIR}
cosmosis-py3 #  to run any cosmosis job, one wants to access the python3 installation, use cosmosis-py3

You should be doing most of your work (editing code, building, running cosmosis) from the y3_cluster_cpp directory, or some directory below that. Pay careful attention to what directory you are in when using git commands: different parts of the cosmosis directory tree are under version control in different repositories!

Debugging C++ code on macOS

Now that you have a working installation, you might want to run a debugger on your C++ code. After you've done the setup you would use for any other CosmoSIS work, you can use the lldb debugger. Note that this debugger has very different commands from the gdb debugger. A good resource for learning to use lldb is https://lldb.llvm.org/use/map.html. To run a pipeline and have access to code in the debugger, use:

#!bash
lldb python -- $(which cosmosis) <configuration file>

When the program starts up, you are actually set up to debug the Python interpreter itself. That is probably not what you really want to do. To get to the code in which you are interested, it is best to set a breakpoint and then to run to that code. Setting breakpoints on C++ functions can be tricky. Here is an example that works. Note that most of the lines are printout from lldb. Input you are to type to the lldb prompt is indicated by "==> "; do not actually type those symbols at the lldb prompt!

#!bash
$ lldb python -- $(which cosmosis) y1_rerun/sigma_y1.ini
(lldb) target create "python"
Current executable set to 'python' (x86_64).
(lldb) settings set -- target.run-args  "/Users/paterno/MyProjects/new-cosmosis-playground/cosmosis/bin/cosmosis" "y1_rerun/sigma_y1.ini"
==> (lldb) breakpoint set --name 'y3_cluster::CosmoSISScalarIntegrationModule<SigmaCentY1ScalarIntegrand>::execute(cosmosis::DataBlock&)'
Breakpoint 1: no locations (pending).
WARNING:  Unable to resolve breakpoint to any actual locations.
==> (lldb) run
Process 74122 launched: '/usr/local/bin/python' (x86_64)
Process 74122 stopped
* thread #2, stop reason = exec
    frame #0: 0x0000000100005000 dyld`_dyld_start
dyld`_dyld_start:
->  0x100005000 <+0>: popq   %rdi
    0x100005001 <+1>: pushq  $0x0
    0x100005003 <+3>: movq   %rsp, %rbp
    0x100005006 <+6>: andq   $-0x10, %rsp
Target 0: (Python) stopped.
==> (lldb) continue
Process 74122 resuming
1 location added to breakpoint 1

<many lines of CosmoSIS chatter elided here...>
Process 74122 stopped
* thread #2, queue = 'com.apple.main-thread', stop reason = breakpoint 1.1
    frame #0: 0x000000011c7040b4 SigmaCentY1ScalarIntegrand.so`y3_cluster::CosmoSISScalarIntegrationModule<SigmaCentY1ScalarIntegrand>::execute(this=0x00000001003d2570, sample=0x000000011c635450) at CosmoSISScalarIntegrationModule.hh:128:25
   125 	y3_cluster::CosmoSISScalarIntegrationModule<I>::execute(
   126 	  cosmosis::DataBlock& sample)
   127 	{
-> 128 	  integrand_.set_sample(sample);
   129 	  auto results = use_cartesian_product_of_volumes_and_gridpoints_ ?
   130 	                   integrate_cartesian_product_of_volumes_and_gridpoints() :
   131 	                   integrate_zipped_sequence_of_volumes_and_gridpoints();
Target 0: (Python) stopped.
(lldb)

At this point, you are at the lldb debugger prompt at the start of the function execute in the module of interest.

Clone this wiki locally