Installation of python requirements with pip -r

[pgao]

Minor change, but this is the recommended way of installing requirements files.

https://pip.pypa.io/en/latest/user_guide.html#requirements-files

[seanbell]

I'm pretty sure that your recommend change won't work for a fresh environment with nothing installed (but correct me if I'm wrong).

Installing everything from scratch with a single pip install command won't work, since some libraries require cython and numpy to be installed before running the pip install command for that library. This is why the current for-loop expansion is necessary.

[pgao]

On the page I linked to:

Logically, a Requirements file is just a list of pip install arguments placed in a file.

Which makes me believe that the change works the same way as doing the for-loop expansion, by running a bunch of pip install commands in the order specified in the requirements file.

To test, I made a virtualenv and ran the pip install -r command on requirements.txt with these commands from the caffe base directory:

virtualenv venv
source venv/bin/activate
pip install -r python/requirements.txt

and it seemed to install everything fine (on my OSX setup): first cython is installed, then numpy, then scipy, etc.

You might want to try this on your computer, though. I'm not sure if you'll get the same result.

[shelhamer]

I included the for loop expansion because I'm fairly certain running pip install -r requirements.txt did not respect the order / dependencies when I once tried it. If there are further confirmations to the contrary however, not using the expansion is slightly simpler.

[longjon]

See http://stackoverflow.com/questions/5394356/how-to-specify-install-order-for-python-pip, etc. Unless this has been changed, it seems that order is not respected in general, although maybe it shouldn't matter if all dependencies are correctly specified.

In any case it's a bit simpler to use xargs -n1, if you like.

[seanbell]

TLDR: The proposed change does not work on Ubuntu 14.04.

I created a fresh Ubuntu 14.04 server EC2 instance to test this out.

Installing packages:

sudo apt-get update -y
sudo apt-get upgrade -y
sudo apt-get install -y libprotobuf-dev libleveldb-dev libsnappy-dev \
    libopencv-dev libboost-all-dev libhdf5-serial-dev libgflags-dev \
    libgoogle-glog-dev liblmdb-dev protobuf-compiler
sudo apt-get install -y python-dev
sudo apt-get install -y python-pip
sudo apt-get install -y gfortran
sudo apt-get install -y libatlas-base-dev
sudo apt-get install -y git

Installing python requirements:

git clone https://github.com/BVLC/caffe
cd caffe/python
sudo pip install -r requirements.txt

This fails at installing scikit-image:

...
Downloading/unpacking scikit-image>=0.9.3 (from -r requirements.txt (line 4))
  Downloading scikit-image-0.10.1.tar.gz (16.7MB): 16.7MB downloaded
  Running setup.py (path:/tmp/pip_build_root/scikit-image/setup.py) egg_info for package scikit-image
    Traceback (most recent call last):
      File "", line 17, in 
      File "/tmp/pip_build_root/scikit-image/setup.py", line 37, in 
        from numpy.distutils.core import setup
    ImportError: No module named numpy.distutils.core
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
  File "", line 17, in 
  File "/tmp/pip_build_root/scikit-image/setup.py", line 37, in 
    from numpy.distutils.core import setup
ImportError: No module named numpy.distutils.core
----------------------------------------
Cleaning up...
Command python setup.py egg_info failed with error code 1 in /tmp/pip_build_root/scikit-image
Traceback (most recent call last):
  File "/usr/bin/pip", line 9, in 
    load_entry_point('pip==1.5.4', 'console_scripts', 'pip')()
  File "/usr/lib/python2.7/dist-packages/pip/__init__.py", line 185, in main
    return command.main(cmd_args)
  File "/usr/lib/python2.7/dist-packages/pip/basecommand.py", line 161, in main
    text = '\n'.join(complete_log)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 72: ordinal not in range(128)
ubuntu@ip-10-139-88-123:~/caffe/python$

But this works:

for req in $(cat requirements.txt); do sudo pip install $req; done

It seems that scikit-image requires numpy to be already installed, i.e. the installers are broken for some packages (e.g. scikit-image) and don't work with a single install command.

I also tried using a virtualenv: even after globally installing the packages (one at a time), and then creating a virtualenv, it also fails with the same problem (scikit-image can't find numpy even though it's a dependency).

Side note: It seems that h5py (on the requirement list) is another example with an installer that prevents a single requirements.txt file: h5py/h5py#535 (I get the same bug on my Ubuntu 14.04 test machine).

[pgao]

Ah, fair enough. I guess it's working on my machine because of something to do with OSX:

Downloading/unpacking Cython>=0.19.2 (from -r python/requirements.txt (line 1))
Downloading/unpacking numpy>=1.7.1 (from -r python/requirements.txt (line 2))
Downloading/unpacking scipy>=0.13.2 (from -r python/requirements.txt (line 3))
Downloading/unpacking scikit-image>=0.9.3 (from -r python/requirements.txt (line 4))
Downloading/unpacking scikit-learn>=0.14.1 (from -r python/requirements.txt (line 5))
Downloading/unpacking matplotlib>=1.3.1 (from -r python/requirements.txt (line 6))
Downloading/unpacking ipython>=1.1.0 (from -r python/requirements.txt (line 7))
Downloading/unpacking h5py>=2.2.0 (from -r python/requirements.txt (line 8))
Downloading/unpacking leveldb>=0.191 (from -r python/requirements.txt (line 9))
Running setup.py (path:/Users/pgao/caffe/venv/build/leveldb/setup.py) egg_info for package leveldb

 warning: no files found matching 'leveldb/*.cc'
Downloading/unpacking networkx>=1.8.1 (from -r python/requirements.txt (line 10))
Downloading/unpacking nose>=1.3.0 (from -r python/requirements.txt (line 11))
Downloading/unpacking pandas>=0.12.0 (from -r python/requirements.txt (line 12))
Downloading/unpacking python-dateutil>=1.4,<2 (from -r python/requirements.txt (line 13))
Running setup.py (path:/Users/pgao/caffe/venv/build/python-dateutil/setup.py) egg_info for package python-dateutil

Downloading/unpacking protobuf>=2.5.0 (from -r python/requirements.txt (line 14))
Running setup.py (path:/Users/pgao/caffe/venv/build/protobuf/setup.py) egg_info for package protobuf

And at the end...

Successfully installed Cython numpy scipy scikit-image scikit-learn matplotlib ipython h5py leveldb networkx nose pandas python-dateutil protobuf python-gflags pyyaml pytz six mock pyparsing gnureadline decorator

Here's the full setup log for posterity: http://pastebin.com/NAQPXeAw, but I'm satisfied with closing this.

[taion]

@pgao @seanbell

The problem occurs because pip installs things in the opposite of the order specified. If you re-order your requirements.txt so that e.g. Cython shows up after h5py, then things will work.

This installation works for @pgao on OS X because a wheel is available for h5py, so you don't need to build it from scratch.

[stuaxo]

Re-ordering so h5py is before cython would make the most sense here..

[taion]

BTW, with any recent pip version (>= 6.1.0), all dependencies should now install in the correct order regardless.