Nipype cannot be included in a requirements.txt file assuming an empty environment

[mwaskom]

Nipype installation has some dependency on nibabel, which means that you cannot write a requirements.txt file with both packages and expect it to work (see below for exact issue).

This is problematic as I am trying to put together a minimal set of files to initialize the environment used when analyzing data for a paper. Nipype is supposedly aimed at making research more reproducible, so I don't think the installation process should cause extra headaches for people trying to do that.

$ conda create -n paper pip
$ source activate paper
$ conda install --file conda_requirements.txt
<...>
$ pip install -r pip_requirements.txt
Downloading/unpacking nibabel==1.3.0 (from -r pip_requirements.txt (line 2))
  Downloading nibabel-1.3.0.tar.gz (2.2MB): 2.2MB downloaded
  Running setup.py (path:/private/var/folders/RK/RK5q1cfCE6CoFSziaUGCxU++636/-Tmp-/pip_build_mwaskom/nibabel/setup.py) egg_info for package nibabel
    Missing optional package "dicom" provided by package "pydicom"; you may get run-time errors

Downloading/unpacking nipype==0.9.1 (from -r pip_requirements.txt (line 3))
  Downloading nipype-0.9.1.tar.gz (13.4MB): 13.4MB downloaded
  Running setup.py (path:/private/var/folders/RK/RK5q1cfCE6CoFSziaUGCxU++636/-Tmp-/pip_build_mwaskom/nipype/setup.py) egg_info for package nipype
    Traceback (most recent call last):
      File "<string>", line 17, in <module>
      File "/private/var/folders/RK/RK5q1cfCE6CoFSziaUGCxU++636/-Tmp-/pip_build_mwaskom/nipype/setup.py", line 19, in <module>
        raise RuntimeError('Need nisext package from nibabel installation'
    RuntimeError: Need nisext package from nibabel installation - please install nibabel first
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):

  File "<string>", line 17, in <module>

  File "/private/var/folders/RK/RK5q1cfCE6CoFSziaUGCxU++636/-Tmp-/pip_build_mwaskom/nipype/setup.py", line 19, in <module>

    raise RuntimeError('Need nisext package from nibabel installation'

RuntimeError: Need nisext package from nibabel installation - please install nibabel first

----------------------------------------
Cleaning up...
Command python setup.py egg_info failed with error code 1 in /private/var/folders/RK/RK5q1cfCE6CoFSziaUGCxU++636/-Tmp-/pip_build_mwaskom/nipype
Storing debug log for failure in /Users/mwaskom/.pip/pip.log

[chrisgorgo]

Sorry for that. Let me try to understand this more. This would work if you would do it in two steps (install nibabel first)?

[satra]

i've noticed some strange issues with pip and conda. can you try:

/fullpath/to/envs/paper/bin/pip -r requirements.txt

[mwaskom]

Sorry for that. Let me try to understand this more. This would work if you would do it in two steps (install nibabel first)?

Right, that works fine, but it defeats the purpose of shipping a single requirements.txt file to set up the environment with.

[satra]

this worked fine for me: (i had to comment out dateutil in requirements.txt, added nipype and used conda to install the other dependencies because i didn't want pip to install them)

conda create -n testinstall pip scipy networkx traits dateutil nose
source activate testinstall
/software/anaconda/envs/testinstall/bin/pip install -r requirements.txt 

requirements.txt:

numpy>=1.3
scipy>=0.7
networkx>=1.0
traits>=4.0
#dateutil>=1.5
nibabel>=1.0
nose>=1.0
nipype>=0.9.2

[mwaskom]

Can't reproduce that...

conda install numpy scipy
~/anaconda/envs/paper/bin/pip install -r pip_requirements.txt

Downloading/unpacking nibabel==1.3.0 (from -r pip_requirements.txt (line 2))
  Downloading nibabel-1.3.0.tar.gz (2.2MB): 2.2MB downloaded
  Running setup.py (path:/private/var/folders/RK/RK5q1cfCE6CoFSziaUGCxU++636/-Tmp-/pip_build_mwaskom/nibabel/setup.py) egg_info for package nibabel
    Missing optional package "dicom" provided by package "pydicom"; you may get run-time errors

Downloading/unpacking nipype==0.9.1 (from -r pip_requirements.txt (line 3))
  Downloading nipype-0.9.1.tar.gz (13.4MB): 13.4MB downloaded
  Running setup.py (path:/private/var/folders/RK/RK5q1cfCE6CoFSziaUGCxU++636/-Tmp-/pip_build_mwaskom/nipype/setup.py) egg_info for package nipype
    Traceback (most recent call last):
      File "<string>", line 17, in <module>
      File "/private/var/folders/RK/RK5q1cfCE6CoFSziaUGCxU++636/-Tmp-/pip_build_mwaskom/nipype/setup.py", line 19, in <module>
        raise RuntimeError('Need nisext package from nibabel installation'
    RuntimeError: Need nisext package from nibabel installation - please install nibabel first
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):

  File "<string>", line 17, in <module>

  File "/private/var/folders/RK/RK5q1cfCE6CoFSziaUGCxU++636/-Tmp-/pip_build_mwaskom/nipype/setup.py", line 19, in <module>

    raise RuntimeError('Need nisext package from nibabel installation'

RuntimeError: Need nisext package from nibabel installation - please install nibabel first

----------------------------------------
Cleaning up...
Command python setup.py egg_info failed with error code 1 in /private/var/folders/RK/RK5q1cfCE6CoFSziaUGCxU++636/-Tmp-/pip_build_mwaskom/nipype
Storing debug log for failure in /Users/mwaskom/.pip/pip.log

[satra]

ok this worked because i didn't initially have nipype in the requirements file and then added it. for some reason pip doesn't complete nibabel installation before starting nipype installation. if i do it in two steps this works fine.

pip install -r req1.txt # contains nibabel
pip install -r req2.txt # contains nipype

don't know why this is happening though.

[mwaskom]

Whether or not it's a problem with pip is debatable, but it's something we should work around if possible. From what I can tell, nipype is just using a function that's about 20 lines of code to version based off the git commit, and the package_check function that duplicates a functionality that's already in setuptools (install_requires). I don't see any reason we shouldn't just move this stuff into nipype to avoid this headache (it pops up outside of requirements file, too, e.g. pip install nibabel nipype errors for the same reason, which is very annoying).

[mwaskom]

In general it seems like bad form to import a dependency in the setup script as it precludes any sort of automatic dependency management.

[matthew-brett]

Related discussion for scipy here: scipy/scipy#3379

The summary is, pip's dependency checking is not always what you want, but interoperation with pip could certainly be improved...

[ag-bell]

Does pip come with Anaconda? I just installed Anaconda recently. When I am in the Anaconda command prompt it still defaults to the pip from my other/previous python. From what Satra said earlier, this might lead to problems?

Also is it necessary to do "source activate" on your virtualenv? What does this
do differently that just "activate". Thanks for the insights.

[sburns]

In general it seems like bad form to import a dependency in the setup script as it precludes any sort of automatic dependency management.

+1 to that.

What does numpy.distutils.setup provide that setuptools.setup doesn't?

[satra]

@mwaskom - could you submit a PR for this?

[sburns]

Does pip come with Anaconda? I just installed Anaconda recently. When I am in the Anaconda command prompt it still defaults to the pip from my other/previous python. From what Satra said earlier, this might lead to problems?

@ag-bell When building a conda environment, you can install pip. From then forward, when using that environment, pip will "just work".

$ conda create myenv pip numpy scipy ...
$ pip install something-not-available-from-conda # <-- goes into myenv