-
Notifications
You must be signed in to change notification settings - Fork 4
Native macOS build
These are instructions for getting going with the macOS native build, and our Y3 galaxy cluster analysis software.
This macOS Native installation works with macOS version of 10.14+ (Mojave), and especially Xcode Version 10+ (comes with the macOS Mojave updates). Do not try to build on systems with earlier versions. It won't work.
Read the instructions carefully. Do not skip any steps.
Please report any errors in the instructions to Marc (paterno@fnal.gov).
For macOS, we build C and C++ code using Apple's compilers (clang and clang++), from their Xcode software development suite.
Because Apple does not provide a Fortran compiler, we use Fortran from the GNU Compiler Collection (GCC).
The name of this compiler is gfortran.
Installing the GCC also provides a C compiler (gcc) and a C++ compiler (g++).
We do not use these compilers.
CosmoSIS relies upon a number of software products that are not included as part of macOS. To obtain most of these products we rely upon (Homebrew)[https://brew.sh].
Some of the products we rely upon are installed with Python's pip.
We do this installation in a virtual environment.
We recommend never doing a pip installation of any Python package outside of a virtual environment.
We rely on several features of C++ that are only available in recent Xcode installations. Homebrew also requires an up-to-date Xcode for correct behavior.
If you have a Homebrew installation that hasn't been updated in weeks or months, this may take a while. Failure to keep your Homebrew installation up-to-date is a common cause of build failures, especially of hard-to-diagnose link-time failures.
#!bash
brew update
brew upgrade
brew cleanup # optional; this helps reduce the disk space taken up by Homebrew
brew doctor # optional; helps to diagnose potential installation problems
Note that we are using Python3 from Homebrew, and not Anaconda Python. Experience shows that a mixed environment of Anaconda Python and Homebrew Python is a common source of confusion and errors. If you choose to use Anaconda Python rather than Homebrew Python, we recommend:
- do not install Homebrew Python at all
- install all Python modules using
conda, not `pip
Unfortunately, we can provide little help in the use of Anaconda Python.
#!bash
# If the following install commands tell you that you already have some of these things
# installed and up-to-date, that is not a problem.
# Note that we install the GNU Compiler Collection (GCC) for the GNU Fortran compiler (gfortran),
# not for the GNU C compiler (gcc) nor for the GNU C++ compiler (g++).
brew install git gcc gsl cfitsio fftw minuit2 openblas # these are required for CosmoSIS itself
brew install boost libyaml openmpi
brew install eigen cmake gperftools # these are required for Y3 galaxy cluster code
brew install python wget numpy mpi4py scipy boost-python3
Because we want to be able to run Y3 galaxy cluster analysis programs on grid nodes, we have taken care to make sure we can use a central installation of CosmoSIS. While this is not necessary for running on a laptop, setting up to do so helps make sure that the code and configurations we generate will work in that environment.
To establish such an environment, we suggest the following organization.
- Pick a newly-created (empty) directory under which CosmoSIS,
y3_cluster_cpp, and supporting products will be installed. We will call this our top directoryTOPDIR. - Install CosmoSIS into TOPDIR.
- Create a subdirectory of
TOPDIRinto which code part of neither CosmoSIS nory3_cluster_cppis to go. - Install
y3_cluster_cppinto TOPDIR.
We will assume that the environment variable TOPDIR is defined to be the full pathname of your top directory.
#!bash
mkdir -p ${TOPDIR}
mkdir -p ${TOPDIR}/central # we will install 3rd-party software here
CosmoSIS would need a few other python modules that can not be installed using brew.
We will install them in a virtual environment.
We recommend setting up a dedicated python environment before installing software specific to cluster analyses using this package.
#!bash
cd ${TOPDIR}
python3 -m venv --system-site-packages local-venv # Make sure you use python3 not python
source local-venv/bin/activate # After this python *is* python3
python -m pip install --upgrade pip
python -m pip install pycparser cffi pyyaml matplotlib future configparser emcee astropy fitsio cython six kombine
python -m pip install sklearn # this one takes long time to install
deactivate # Now python is the system python again
This has initiated a local-venv environment to use. If you need to exit, use the deactivate command.
In the future, you will have to source the activate script before you setup CosmoSIS.
We use the develop branch of both cosmosis and cosmosis-standard-library.
#!bash
cd ${TOPDIR}
git clone -b develop http://bitbucket.org/joezuntz/cosmosis
cd cosmosis
git clone -b develop http://bitbucket.org/joezuntz/cosmosis-standard-library
source ../local-venv/bin/activate # activate the virtual environment before setting up CosmoSIS
source config/setup-cosmosis-mac
make # This builds cosmosis and the CSL, but not our Y3 galaxy cluster software
The build will generate some warnings (until CosmoSIS is updated to use C99 officially), but there should be no errors.
At this point, you should have a fully functional CosmoSIS installation. You may wish to run some of the demos to verify this.
Next, you need to build libcuba.dylib.
There is a Homebrew installation of cuba available; it does not build a dynamic library, and it does not use the optimization level we want.
Do not use it.
Do the following instead:
#!bash
git clone https://github.com/marcpaterno/cuba.gitcd ${TOPDIR}/central
mkdir include
mkdir lib
git clone https://github.com/marcpaterno/cuba.git
cd cuba
CC=clang CFLAGS="-O3 -funsafe-math-optimizations -march=native -fPIC" ./configure
./makesharedlib.sh
cp cuba.h ${TOPDIR}/central/include
mv libcuba.dylib ${TOPDIR}/central/lib
cd ${TOPDIR}
rm -rf ${TOPDIR}/central/cuba/
cd ${TOPDIR}/central/lib
install_name_tool -id ${PWD}/libcuba.dylib libcuba.dylib
cd ${TOPDIR}
Some Y3 galaxy cluster pipelines use the Python module cluster_toolkit.
Installing this is not done with pip3; you must do it more manually.
The installed Python module will be put into the same location as your pip3-installed modules.
Note that there is a quirk in the installation script that requires an unusual syntax.
Make sure you do not put a space after the final = when running setup.py below.
#!bash
# source local-venv/bin/activate, if you haven't done so,
pushd ${TOPDIR}
mkdir -p cluster_toolkit_tmp
cd cluster_toolkit_tmp
# we use a particular version that is available from below
wget https://github.com/marcpaterno/cluster_toolkit/archive/master.tar.gz
tar xf master.tar.gz
cd cluster_toolkit-master/
# Note that there must *not* be a space after the = on the next command
python setup.py install
popd
rm -rf ${TOPDIR}/cluster_toolkit_tmp
This is the repository containing our analysis code, and the only code you are likely to be modifying as you write your analysis.
#!bash
cd ${TOPDIR}
git clone git@bitbucket.org:mpaterno/y3_cluster_cpp.git
Next, we need cubacpp.
This is a header-only library, so all you need to do is clone the repository.
We do this in externals directory of y3_cluster_cpp.
#!bash
cd ${TOPDIR}/y3_cluster_cpp/externals
git clone git@bitbucket.org:mpaterno/cubacpp.git
Unlike CosmoSIS and the CSL, y3_cluster_cpp is built using makefiles generated by CMake.
The resulting makefiles fully support parallel builds.
We rely on several environment variables both during the build configuration and while running pipelines.
Some of these environment variables are defined when you source the setup_cosmosis_mac script.
One other needs to be defined "by hand", to a value that is determined by exactly where you have installed the software.
This is Y3_CLUSTER_CPP_DIR, as shown below.
During the configuration of the build, we also rely upon TOPDIR, as define above.
#!bash
cd ${TOPDIR}/cosmosis
# Pay careful attention to the comments on the next two lines!
source $TOPDIR/local-venv/bin/activate # only if it is not currently active
source config/setup-cosmosis-mac # only if you haven’t already done it in this shell session
cd ${TOPDIR}/y3_cluster_cpp
export Y3_CLUSTER_CPP_DIR=${PWD}
cmake -DCMAKE_MODULE_PATH="./CMake;${Y3_CLUSTER_CPP_DIR}/externals/cubacpp/cmake/modules" -DCUBACPP_DIR=${Y3_CLUSTER_CPP_DIR}/externals/cubacpp -DCUBA_DIR=${TOPDIR}/central -DCMAKE_BUILD_TYPE=Release .
# If you are using the previously installed version without `TOPDIR` set up, do so instead.
# cmake -DCMAKE_MODULE_PATH="${COSMOSIS_SRC_DIR}/cosmosis-standard-library/y3_cluster_cpp/CMake;${COSMOSIS_SRC_DIR}/cubacpp/cmake/modules" -DCUBACPP_DIR=${COSMOSIS_SRC_DIR}/cubacpp -DCUBA_DIR=/usr/local -DCMAKE_BUILD_TYPE=Release .
make -j2 # or -j4, if you have a 4-core laptop. Don’t use -j (without a number).
ctest -j2 # or -j4, if you have a 4-core laptop. Don’t use -j (without a number).
If you encounter a compiation failure that says:
fatal error: 'optional' file not found
this is an indication that your Xcode installation is out-of-date. You will need to upgrade Xcode before you can continue.
As noted several times above, every time you want to work with CosmoSIS in a new shell session, you need to source the CosmoSIS setup script.
You will also need to define Y3_CLUSTER_CPP_DIR to point to the same directory as was done above.
You may find it useful to have TOPDIR defined.
#!bash
export Y3_CLUSTER_CPP_DIR=${TOPDIR}/y3_cluster_cpp
export Y3_CLUSTER_WORK_DIR=${TOPDIR}/y3_cluster_cpp
cd ${TOPDIR}
source local-venv/bin/activate
cd ${TOPDIR}/cosmosis # so that the cosmosis setup script will work.
source config/setup-cosmosis-mac # only if you haven’t already done it in this shell session.
cd ${Y3_CLUSTER_CPP_DIR}
# To run any cosmosis job, you can just use 'cosmosis <ini-file>'; it will use the python
# in the virtual environment, which is python3
You should be doing most of your work (editing code, building, running cosmosis) from the y3_cluster_cpp directory, or some directory below that.
Pay careful attention to what directory you are in when using git commands: different parts of the cosmosis directory tree are under version control in different repositories!
Whenever a user-mode program (such as cosmosis) terminates unexpectedly (as by a call to abort),
macOS stores information about that crash in a log file. You can find the log files in the directory
~/Library/Logs/DiagnosticReports. The file names end in .crash. The beginning of the file name is the
name of the running program; for cosmosis, that will be Python (or maybe python, depending on the
interpreter you are using). The filenames also carry a timestamp, and an identifier of your machine.
The file is a plain ASCII text file, and may contain useful information. In particular, if the program being run contained debug symbols, a stack trace of the program is provided. This is sufficient for some debugging purposes.
Now that you have a working installation, you might want to run a debugger on your C++ code.
After you've done the setup you would use for any other CosmoSIS work, you can use the lldb debugger.
Note that this debugger has very different commands from the gdb debugger.
A good resource for learning to use lldb is https://lldb.llvm.org/use/map.html.
To run a pipeline and have access to code in the debugger, use:
#!bash
lldb python -- $(which cosmosis) <configuration file>
When the program starts up, you are actually set up to debug the Python interpreter itself.
That is probably not what you really want to do.
To get to the code in which you are interested, it is best to set a breakpoint and then to run to that code.
Setting breakpoints on C++ functions can be tricky.
Here is an example that works.
Note that most of the lines are printout from lldb.
Input you are to type to the lldb prompt is indicated by "==> "; do not actually type those symbols at the lldb prompt!
#!bash
$ lldb python -- $(which cosmosis) y1_rerun/sigma_y1.ini
(lldb) target create "python"
Current executable set to 'python' (x86_64).
(lldb) settings set -- target.run-args "/Users/paterno/MyProjects/new-cosmosis-playground/cosmosis/bin/cosmosis" "y1_rerun/sigma_y1.ini"
==> (lldb) breakpoint set --name 'y3_cluster::CosmoSISScalarIntegrationModule<SigmaCentY1ScalarIntegrand>::execute(cosmosis::DataBlock&)'
Breakpoint 1: no locations (pending).
WARNING: Unable to resolve breakpoint to any actual locations.
==> (lldb) run
Process 74122 launched: '/usr/local/bin/python' (x86_64)
Process 74122 stopped
* thread #2, stop reason = exec
frame #0: 0x0000000100005000 dyld`_dyld_start
dyld`_dyld_start:
-> 0x100005000 <+0>: popq %rdi
0x100005001 <+1>: pushq $0x0
0x100005003 <+3>: movq %rsp, %rbp
0x100005006 <+6>: andq $-0x10, %rsp
Target 0: (Python) stopped.
==> (lldb) continue
Process 74122 resuming
1 location added to breakpoint 1
<many lines of CosmoSIS chatter elided here...>
Process 74122 stopped
* thread #2, queue = 'com.apple.main-thread', stop reason = breakpoint 1.1
frame #0: 0x000000011c7040b4 SigmaCentY1ScalarIntegrand.so`y3_cluster::CosmoSISScalarIntegrationModule<SigmaCentY1ScalarIntegrand>::execute(this=0x00000001003d2570, sample=0x000000011c635450) at CosmoSISScalarIntegrationModule.hh:128:25
125 y3_cluster::CosmoSISScalarIntegrationModule<I>::execute(
126 cosmosis::DataBlock& sample)
127 {
-> 128 integrand_.set_sample(sample);
129 auto results = use_cartesian_product_of_volumes_and_gridpoints_ ?
130 integrate_cartesian_product_of_volumes_and_gridpoints() :
131 integrate_zipped_sequence_of_volumes_and_gridpoints();
Target 0: (Python) stopped.
(lldb)
At this point, you are at the lldb debugger prompt at the start of the function execute in the module of interest.