Recursive dependency upgrades should be subject to global version constraints

[JoshBroadway]

Hi,

The changes to the installation of dependencies in the latest version of pybuilder (0.11.8) seem to be causing problems for a couple of our builds.

Our project has a dependency on a specific version of scipy (0.17.0), we are not ready to upgrade to the latest version of scipy (0.18.0).

We also have a dependency on Theano (any version), which itself also depends on scipy, however it's dependency on scipy is only >= 0.11.1.

Our pybuild script dependencies looks like:
`

project.depends_on('numpy')
project.depends_on('Theano')
project.depends_on('scipy', '==0.17.0')

`
Up until now this has worked for us because we've set the 'install_dependencies_upgrade' flag to False, so that when Theano is installed, it doesn't try to upgrade scipy.

Having looked through the code for 0.11.8 pybuilder, it looks like that flag is no longer being used, with the upgrade flag being passed through as True by default. This causes pip to try to upgrade scipy to 0.18.0 and leads to build failures due to our code not supporting the latest version of scipy.

It looks like this issue has arisen out of the fix for #340. Is there another way to prevent pip upgrading dependencies?

[arcivanov]

I'm not sure I'm reading this right. You have forced scipy==0.17.0, which means that PIP will not be installing scipy 0.18.0, no matter what Theano needs. Are you saying that PIP is ignoring your scipy request?

1. If you're using 0.11.8, please update to the latest with "--pre" and try again.
2. Please post a -X -v log of your build, or at least of the relevant portion and logs/install_dependencies.

[JoshBroadway]

Thanks for your response,

I don't think the problem is that PIP is ignoring the scipy request, I think the problem is that passing the --upgrade flag to pip when it tries to install Theano is causing pip to try to upgrade scipy. The pip documentation indicates that the --upgrade flag is "recursive regardless of whether a dependency is already satisfied".

I've attached two sets of logs for you to have a look at. Our full list of dependencies is:

# Runtime dependencies
project.depends_on('numpy')
project.depends_on('Theano')
project.depends_on('keras', '==1.0.3')
project.depends_on('enum34')
project.depends_on('scipy', '==0.17.0')
# Build dependencies
project.build_depends_on('nosexcover')
project.build_depends_on('mock')
project.build_depends_on('flake8', '==2.6.2')

Our build is a bit more complex than what I originally explained. We cannot simply 'pip install scipy' (We are building on Windows and there are no Windows .whl packages in Pypi. 'pip install scipy' downloads the scipy source package which requires Visual Studio and a bunch of other dependencies).

So to avoid having to install Visual Studio, our build script pre-installs a local .whl copy of Scipy before we try to run Pybuilder install-dependencies.

I've attached logs from two Pybuilder runs:

• install_batch_with_scipy_pre_installed.log shows the problem that I outlined in my original post. While installing Theano pip tries to upgrade scipy to version 0.18.0.
• install_batch_without_scipy_pre_installed.log was generated after I commented out the lines that install scipy before running Pybuilder. It's shows the behaviour that you were expecting - pip tries to install 0.17.0.

So maybe the fact that scipy is already installed is causing pybuilder to not pass it through to pip?

install_batch_with_scipy_pre_installed.txt
install_batch_without_scipy_pre_installed.txt

[arcivanov]

The pip documentation indicates that the --upgrade flag is "recursive regardless of whether a dependency is already satisfied".

Ah, sorry about that. Yep, this needs fixing as this is not the intended behavior.

[arcivanov]

Dev notes:

1. pip install --upgrade Theano scipy==0.17.0 installs versions "Theano (0.8.2)" and "scipy (0.17.0)" correctly.
2. The reason PyB is screwing up is that, per PyBuilder install_dependencies behavior should mirror #340 #346, when a package is installed AND it already satisfies the dependencies, we exclude it from the list of packages to install. This, in turn, causes a constraint (scipy==0.17.0 in this case) to be absent from the requirement list, allowing Theano dependencies to dominate.

The answer for this is to ensure that no matter what you install, the dependency constraints are always observed.

1. One option is to use a constraint file. If this can be accomplished with command line options (instead of a file), this is preferred. - This is what will be implemented.
2. Another option to always include the constraints in the list of requirements no matter what we install. This is inefficient, especially for URL-based requirements since we always force-reinstall those and including the constraints are requirements will also force-reinstall them every time.

[arcivanov]

Constraints work

bash-4.3$ pip install Theano -c constraints
Collecting Theano
Requirement already satisfied (use --upgrade to upgrade): six>=1.9.0 in /home/arcivanov/.pyenv/versions/3.5.2/envs/venv/lib/python3.5/site-packages (from Theano)
Collecting scipy==0.17.0 (from -c constraints (line 1))
  Using cached scipy-0.17.0-cp35-cp35m-manylinux1_x86_64.whl
Collecting numpy>=1.7.1 (from Theano)
  Using cached numpy-1.11.1-cp35-cp35m-manylinux1_x86_64.whl
Installing collected packages: scipy, numpy, Theano
Successfully installed Theano-0.8.2 numpy-1.11.1 scipy-0.17.0

[arcivanov]

@BackroomGibbon please upgrade to latest --pre and you should be all set.

[JoshBroadway]

@arcivanov I've just tested it and confirmed it works.

Thanks for your help!