Installation on Factory Resetted Mac OSx Mavericks

dorislee0309 edited this page Sep 3, 2014 · 11 revisions

The code in the project requires a lot of dependencies and various software as an astronomical toolbox. I had a tough time installing these due to the horrible dependency chain that these software have. Here, I have listed the detailed steps of installation that I conducted after I factory-resetted my computer due to bad package management and package version control. I successfully finish all the necessary dependencies with the help of multiple sources which I will link below. ##Brew and preliminary installation setup ruby -e "$(curl -fsSL https://raw.github.com/Homebrew/homebrew/go/install)”

brew doctor

Brew doctor throws the error that

This will cause some formulae to fail to build.```

Even after installing gcc command line tools it is still like that.But according to home-brew 
```Please note that these warnings are just used to help the Homebrew maintainers with debugging if you file an issue. If everything you use Homebrew for is
working fine: please don't worry and just ignore them. Thanks!``` So I just ignored it. 

After doing some research on MacPorts v.s. Homebrew, I convinced myself to use homebrew even though Macports has more packages. Most of the scientific computing pacakges are avaliable in the tap.  [homebrew/homebrew-science](https://github.com/Homebrew/homebrew-science)

```brew install git tmux wget vim ``` 

``brew install gcc`` #gfortran (now gcc on Homebrew) is the fortran compiler. It is required to build atlas, atlas is required to build SExtractor (and possibly other Astromatic tools)


##Notes from Homebrew and Python
(This section is largely based off this guide on [Homebrew and Python](https://github.com/Homebrew/homebrew/wiki/Homebrew-and-Python) )

``brew tap`` is kinda like importing namespace in Python, it just tells brew that these packages are out there and I might want to install them soon.  

```pip install --upgrade setuptools```

```brew tap homebrew/versions```
```brew info homebrew/versions/gcc47```
```brew install homebrew/versions/gcc47 --enable-all-languages```

Installing packages that should be in the global site-packages:

```pip install virtualenvwrapper pygments sphinx flake8 rope ropemode```
```pip install ipython[all]```

Then I tested ipython notebook and it works.

So there was some trouble with installation because the xcode file path was set incoreectly so I had to reset by 
``sudo xcode-select --reset``
Then you need to verify that ``xcode-select -p`` returns ``/Library/Developer/CommandLineTools`` and not something in the Macintosh HD . 

#Setting up virtualenv 

I brewed python but it was already installed as part of mac . Then I installed python3 : 
```brew install python3``
On the brew website, they said that you should never brew when you are inside a virtualenv.
``pip install virtualenv``
``pip install virtualenvwrapper``
If I cd inside usr/bin/ I will see the versions of python I have, I only want to use 2.7 and install 3.3 (or 3.4 later) .

Doriss-MacBook-Pro:bin dorislee$ which python
/usr/bin/python
Doriss-MacBook-Pro:bin dorislee$ which python3
/usr/local/bin/python3

Note that python3 is installed at a different location

If you mess up, simply do ``pip uninstall virtualenv`` then ``sudo pip install virtualenv`` again. 
Following the instructions on [hackercodex](http://hackercodex.com/guide/python-development-environment-on-mac-osx/) 
```mkdir -p ~/Projects ~/Virtualenvs```
Edit the bashrc file
``vim ~/.bashrc`` by adding 

pip should only run if there is a virtualenv currently activated
```export PIP_REQUIRE_VIRTUALENV=true```
cache pip-installed packages to avoid re-downloading
```export PIP_DOWNLOAD_CACHE=$HOME/.pip/cache```

Re-load our bash environment:

. ~/.bash_profile cd ~/Virtualenvs virtualenv -p python py27dev virtualenv -p python3 py34dev


I ignored the part on forcing pip install non global because I want some stuff for global installation. 
Use ``workon py27dev`` to start, ``deactivate`` to get off.

You need to ``pip install virtualenvwrapper" to easier remove and work with virtualenv. Then you need to load virtualenvwrapper, otherwise you can't really use it after it is installed. Execute the following for "lazy loading": 

export WORKON_HOME=$HOME/.virtualenvs export PROJECT_HOME=$HOME/Devel export VIRTUALENVWRAPPER_SCRIPT=/usr/local/bin/virtualenvwrapper.sh source /usr/local/bin/virtualenvwrapper_lazy.sh

So the chunk of code above should be executed at startup everytime, so that we can use virtualenvwrapper. So I will put it in .bash_profile
Inside the virtualenv directory (suggestion from [SO](http://stackoverflow.com/questions/24300952/installing-python-2-and-3-separately?noredirect=1#comment37558893_24300952))
```mkvirtualenv -p ../../../../usr/bin/python  py27dev```
Inside usr/bin/python I just have to do this
```mkvirtualenv -p /usr/local/bin/python3.4  py34dev```
Then I deleted the VirtualEnv inside doris/ because it was not necessary. 
Both the py27dev and py34dev folder (as well as ENV) was already set up in the outermost level of directory tree (just one cd into Macintosh HD)

I moved venv folders to inside doris because it is easier to have user access without having to type the password everytime. It also makes more sense to do it this way.Double checked that this still works. (Remember need to setup virtualenv wrapper everytime(fix this))

Now py34dev and py27dev both work now!!


##Setting up VMWare

- [https://www.virtualbox.org/wiki/Downloads](Install) using the .pkg file 

Consider using tmux??
## Python Scientific Packages
pip freeze

I decided to install all of these separately so that one does not depend on one another (just in case the make file depends on python version, so you might not want to share installation)

After this, I might have to "rebrew" some of the stuff I previously brewed. Just keep follow [Johnathan Sick's Guide](https://github.com/jonathansick/dotfiles/wiki/Notes-for-Mac-OS-X)
So I am doing this inside the py34dev venv: 
---
brew install umfpack swig nose
pip install numpy
pip install cython
pip install scipy

brew install libpng freetype
There is an error in installing matplotlib which claims that freetype can not be bui;t, which is resolve by [2nd solution on this](http://superuser.com/questions/242190/how-to-install-matplotlib-on-os-x)
The problem is that when the C extensions are compiled, required headers files aren't in the search path, and when they're being linked, shared libraries aren't in the search path either.

Possibly require these, not sure 
export LDFLAGS="-L/usr/X11/lib"
export CFLAGS="-I/usr/X11/include -I/usr/X11/include/freetype2 -I/usr/X11/include/libpng12"
pip install matplotlib-1.0.1.tar.gz
export CFLAGS="-I/usr/X11/include -I/usr/X11/include/freetype -I/usr/X11/include/libpng"
pip install matplotlib-1.0.1.tar.gz
The installation worked after  installign pkg-config which tells setup.py where to look for the installed libraries (not pkg-info)

brew install pkg-config pip install matplotlib


There was something wrong with Numba installation, I'm too tired to fix it. I never use Numba for optimization anyways. I hope nothing has it as dependency later on.

So I had to repeat these inside the venv py27dev and the stuff that I had previously "brewed" is already installed. Now all I have to do is redo the pip install stuff after I switch into the py27dev by ``workon py27dev``.

I forgot about AstroPy installation at both venvs. But I recall that I can not pip it, since pip and macport only gives you the stable 0.3 version and you want the latest developer version 0.4 because you might want to use the SkyCoord class which was something new in 0.4.

cd py34dev git clone https://github.com/astropy/astropy.git cd astropy python setup.py build python setup.py install


Then you can test it by going into the python interactive interpreter

import os os.chdir("..") import astropy astropy.test()

It shows 

../.virtualenvs/py34dev/lib/python3.4/site-packages/astropy-0.4.dev9207-py3.4-macosx-10.9-x86_64.egg/astropy/config/configuration.py:855: FileNotFoundError ========================= 1 failed, 4961 passed, 147 skipped, 12 xfailed, 1 xpassed in 141.67 seconds ========================== 1

In the astropy.test() for py27dev,

../.virtualenvs/py27dev/lib/python2.7/site-packages/astropy-0.4.dev9207-py2.7-macosx-10.9-intel.egg/astropy/wcs/tests/test_wcsprm.py ..............Segmentation fault: 11

But astropy seems to work, so I'm just going to ignore those failed test.
This should be repeated on the py27dev. More specifics about the AstroPy installation [here](https://github.com/astropy/astropy/blob/master/docs/install.rst#id7).

## Trying to install Astrometric softwares (and its horrible dependency chain)

sudo wget http://www.netlib.org/lapack/lapack-3.4.2.tgz Need to manually grab atlas tar file from sourceforge: bunzip2 -c atlas3.10.1.tar.bz2 | tar xfm -

I give up on trying to install atlas. It seems like a waste of my hours trying to install atlas and I dont want to risk anymore by messing with bash files again. I always breaks during the configure steps, and esp on Mavericks (and Mac OSx) system because apparently the tarball "unzip" is different form other Debian/Linux OS. So I am just going to install SciSoft and then delete their python so that it does not conflict with my venv. Its a huge installation though. I am building the [10.8.x of SciSoft for MacOsX](http://scisoftosx.dyndns.org/). 

This is a pretty GUI installer and you just have to click on it (with a cursor!)
Then I went into usr/local/scisoft/packages and manually drrag and deleted the python directory because I do not want any possibility of conflict even though scisoft promised not to have that problem..
I also deleted ipython files in bin/: 
iptest
ipython
ipython-wx
ipythonx
I need to configure the bash profile later to execute the startup script. That's the only fuss I have to do.
So I am not sure why sudo bash doesn't work and instead source works, so I guess Mac Terminal is some strange mix of bash, csh and tcsh shell.
```source  /usr/local/scisoft/bin/Setup.bash```
So this is the magical command that works. (To test, just type ``sex`` in terminal and see if SExtractor opens up with E.Bertin and version info.)

So I had to add these lines into my .bash_profile (you can see this is you do ls -a ) , I also convieniently added some aliases

source /usr/local/scisoft/bin/Setup.bash alias ..='cd ..' alias l='ls'

Then I ecountered the same issue as on my ipynb 6/17 
When I ran SExtractor without any paramerters,

> *ERROR*: can't read default.param


Same error ecountered here: http://www.astromatic.net/forum/showthread.php?tid=422
This is resolved by going into /usr/local/scisoft/packages/sextractor/config and copying all the defult files into the directory you are calling sextractor on. 

I have set the param , conv...etc path name inside the default.sex file.
But whenever I run it I need to specify location of default.sex by : 

sex SDSS_i_155.925_12.5605555556.fits -c /usr/local/scisoft/packages/sextractor/config/default.sex
I also forgot that I had to install Montage then the montage python wrapper 
You download the tar.gz file from the Montage website. There is a problem with building montage on mavericks which is detailed in my 5/16/2014 notebook

Montage Installation - kind of a fuss because apparently the C code compiler - there seemed to be lots of warning, but it is negligible - but what is crashing is the program the "error" , basically just wrong return type - after various searches, I found the patch([1](http://trac.macports.org/ticket/41076 - https://www.google.com/search?q=error%3A+non-void+function+%27closefitsfile%27+should+return+a+value+%5B-Wreturn-type%5D+montage&rlz=1C5CHFA_enTW569TW569&oq=error%3A+non-void+function+%27closefitsfile%27+should+return+a+value+%5B-Wreturn-type%5D+montage&aqs=chrome..69i57j69i58.3027j0j7&sourceid=chrome&es_sm=91&ie=UTF-8),[2](http://trac.macports.org/attachment/ticket/41076/patch-lib-src-two_plane_v1.1-initdistdata.c.diff)) on macports 

lib/src/two_plane_v1.1/initdistdata.c old new 28 28 if (I_fits_return_status != 0) 29 29 { 30 30 fprintf(stderr, "Error closing file\n"); 31 return; #Old code, BAD!!! 31 return -1; 32 32 } 33 return; #Old code, BAD!!! 33 return 0; 34 34 }

After changing this I did ``make`` which made everything inside the directory. But you need to put the directory where you specify the path (for me it is the dorislee/ dir) so you have to add this into your .bash_profile  (which configures you startup shell) 
``PATH=$PATH:$HOME/Montage_v3.3/bin``
To verify that installation is done correctly,try ``mAdd``
Then simply do ``pip install montage-wrapper``


##Other Useful Guides
- [HomeBrew Science tap](https://github.com/Homebrew/homebrew-science)
- [XCode Command Line Setup](http://railsapps.github.io/xcode-command-line-tools.html)
- [Using HomeBrew for Python Installation](https://github.com/Homebrew/homebrew/wiki/Homebrew-and-Python)
- [Another software installer bundle for IRAF](http://www.astrobetter.com/ureka-a-new-easy-to-install-iraf-pyraf-distribution-from-stsci-and-gemini/)

__Other guides on scientific packages/software installation on Mac__
- https://github.com/jonathansick/dotfiles/wiki/Notes-for-Mac-OS-X
- http://www.lowindata.com/2013/installing-scientific-python-on-mac-os-x/#quickstart
- http://osxastrotricks.wordpress.com/2010/07/19/new-mac-setup/