Support pre-installed SciPy on Python

[pelegm]

Many Python packages depend on SciPy so it would be helpful to have a recent SciPy installed for each supported Python platform. It can be installed manually using a quite simple process (as I've described in #2638), but it takes about 15 minutes, and it is not easy to guess the requirements without referring to this ticket.

[thedrow]

Actually the entire science stack takes quite a while to install.
NumPy, SciPy, scikit-learn, pandas, geopandas, statsmodels all require a significant time in order to compile.
Preinstalling those packages (or some of them) might be a good idea.

[BanzaiMan]

@thedrow Could you give estimates on how long those packages take to compile?

[thedrow]

Without ccache and f90cache both NumPy and SciPy take about 15-20 minutes to compile each.
With ccache and f90cache (providing that the package has been compiled before) about 5 minutes each, sometimes less.
SciPy also needs a fortran compiler such as gfortran in order to compile.
statsmodels can take about 10 minutes to install and it requires pandas, patsy which take about 5 minutes to install each.
geopandas require libgeos-dev and libgdal1-dev which take about 5 minutes to install.
Installing geopandas itself can take another 5 minutes.

So if you have a projec that deals with machine learning and geographic distances the installation process is very long.

[dstufft]

Since Travis has a consistent environment, you might be able to just create a travis-ci wheel repository that pip will install from automatically. This will people install these projects in a pre-compiled fashion (and make things go faster) without needing to compile them during image build.

[cancan101]

It seems like people are using miniconda (see here and here) as a workaround. This is not ideal as miniconda creates its own virtualenv.

I would 👍 to see "scientific python" as a version of python that can be specified.

[nirum]

Seconded, would love to see better scientific python support

[AndreaCensi]

Hi, I was wondering if the miniconda workaround is still the preferred way of installing Numpy and Scipy.

Basically if you do any kind of scientific computing using Python, you are going to use Numpy and Scipy.
Why not have them in the default image?

Note also that it is very tricky to install them using pip from sources.

[johnyf]

Another workaround is to use apt and allow system-wide site-packages, as described here.

[Samreay]

Has there been any development on this issue? Whether it be a scientific-python distribution we can access, or is the current fix still to use work arounds?

[peterjc]

Nowadays there are pre-compiled binary wheels for NumPy and SciPy available via pip, which work nicely with TravisCI, e.g.

pip install --only-binary=numpy,scipy numpy scipy

If you leave out the --only-binary=numpy,scipy bit there is a chance pip would end up trying to compile things (and fail or take ages).

[thorade]

@peterjc When I copy&paste your pip install command I get no such option: --only-binary from Travis. Did I miss anything?

PS: This is my first day as Travis user & the very first line of code I wrote into the .travis.yml file was for installing scipy, leading me here...

[dstufft]

@thorade You probably need to first do pip install --upgrade pip setuptools wheel.

[BanzaiMan]

Closing this now. Perhaps we can also document #2650 (comment).

[mroberge]

I found that using pip to install binaries worked really well for me- I was able to cut my testing in Python 3.4 down from 9 minutes to less than 1 minute. Python 3.5 and 3.6 already have numpy in their environments, so this didn't speed them up by more than a few seconds. I just added this to the install section:
pip install --upgrade pip setuptools wheel
pip install --only-binary=numpy,scipy numpy scipy

My complete .travis.yml