conflicting numpy dependencies in v5.7 with Python 3.6

[MarkusShepherd]

When installing Turicreate under Python 3.6, there are conflicting (i.e., unresolvable) constraints on the numpy version to be installed. To reproduce:

pipenv install turicreate --python python3.6

It fails with this message:

Could not find a version that matches numpy<1.15.0,==1.16.4,>=1.10,>=1.13.3,>=1.14.5,>=1.8.2

When running

pipenv install turicreate --python python3.6 --skip-lock
pipenv graph

it yields

turicreate==5.7
  - coremltools [required: ==3.0b3, installed: 3.0b3]
    - numpy [required: >=1.14.5, installed: 1.16.4]
    - protobuf [required: >=3.1.0, installed: 3.9.1]
      - setuptools [required: Any, installed: 41.1.0]
      - six [required: >=1.9, installed: 1.12.0]
    - six [required: >=1.10.0, installed: 1.12.0]
  - decorator [required: >=4.0.9, installed: 4.4.0]
  - mxnet [required: >=1.1.0,<1.2.0, installed: 1.1.0.post0]
    - graphviz [required: >=0.8.1,<0.9.0, installed: 0.8.4]
    - numpy [required: >=1.8.2,<1.15.0, installed: 1.16.4]
    - requests [required: >=2.18.4,<2.19.0, installed: 2.22.0]
      - certifi [required: >=2017.4.17, installed: 2019.6.16]
      - chardet [required: >=3.0.2,<3.1.0, installed: 3.0.4]
      - idna [required: >=2.5,<2.9, installed: 2.8]
      - urllib3 [required: >=1.21.1,<1.26,!=1.25.1,!=1.25.0, installed: 1.25.3]
  - numpy [required: ==1.16.4, installed: 1.16.4]
  - pandas [required: >=0.23.2, installed: 0.25.0]
    - numpy [required: >=1.13.3, installed: 1.16.4]
    - python-dateutil [required: >=2.6.1, installed: 2.8.0]
      - six [required: >=1.5, installed: 1.12.0]
    - pytz [required: >=2017.2, installed: 2019.2]
  - pillow [required: >=5.2.0, installed: 6.1.0]
  - prettytable [required: ==0.7.2, installed: 0.7.2]
  - requests [required: >=2.9.1, installed: 2.22.0]
    - certifi [required: >=2017.4.17, installed: 2019.6.16]
    - chardet [required: >=3.0.2,<3.1.0, installed: 3.0.4]
    - idna [required: >=2.5,<2.9, installed: 2.8]
    - urllib3 [required: >=1.21.1,<1.26,!=1.25.1,!=1.25.0, installed: 1.25.3]
  - resampy [required: ==0.2.1, installed: 0.2.1]
    - numba [required: >=0.32, installed: 0.45.1]
      - llvmlite [required: >=0.29.0dev0, installed: 0.29.0]
      - numpy [required: Any, installed: 1.16.4]
    - numpy [required: >=1.10, installed: 1.16.4]
    - scipy [required: >=0.13, installed: 1.3.1]
      - numpy [required: >=1.13.3, installed: 1.16.4]
    - six [required: >=1.3, installed: 1.12.0]
  - scipy [required: >=1.1.0, installed: 1.3.1]
    - numpy [required: >=1.13.3, installed: 1.16.4]
  - six [required: >=1.10.0, installed: 1.12.0]

I.e., turicreate itself fixes numpy at v1.16.4 (bad practice for a library IMHO), while mxnet requires numpy before v1.15.0.

It seems to be working correctly under Python 3.7, though numpy still is fixed at v1.16.4, but mxnet is upgraded (yet fixed) to v1.5.0, which requires numpy >v1.16.0.

It would be great if dependencies resolve properly on all supported Python versions. Also, it'd be much more robust if they are specified with a supported range, not a fixed version, else Turicreate will become quickly unusable alongside other libraries.

[TobyRoseman]

@MarkusShepherd - Thank you for the detailed write up. I appreciate it.

I agree we should do a better job of handling dependency versions. Relying on a range of versions would certainly be better.

Here are all of the distinct requirements for numpy in Python 3.6 that you shared:

- numpy [required: >=1.14.5, installed: 1.16.4]
- numpy [required: >=1.8.2,<1.15.0, installed: 1.16.4]
- numpy [required: Any, installed: 1.16.4]
- numpy [required: >=1.10, installed: 1.16.4]
- numpy [required: >=1.13.3, installed: 1.16.4]

So it sounds like we should change our numpy requirement from numpy==1.16.4 to numpy >=1.14.5, <1.15.0. I think this is the best we can do. What do you think?

[MarkusShepherd]

Yes, I'd think so. Looking at the other requirements there are more dependencies fixed at a specific version (coremltools, prettytable, resampy) which would similarly fail to install along any other library that might fix those same dependencies (or maybe allows only a narrow range).

I'm not familiar with best practices regarding dependencies for libraries, but intuitively it makes sense to bound them according to what functionality your own library actually requires, and let dependencies handle their own bounds. In this concrete case, it should be enough to restrict numpy in your requirements to whatever minimum version you need, and let mxnet handle further restrictions if they are really not compatible with a more recent version. But like I said, there might be good reasons to handle dependencies differently...

[MarkusShepherd]

FYI, I just realised mxnet itself is fixed at v1.5.0 for Python3.7 and should probably be >=1.5.0 instead.

[znation]

Seems to only repro on pipenv. Using virtualenv and pip, the console shows 'ERROR: ' lines pertaining to mxnet versions, but ignoring those errors, the installation seems to complete and work properly.

[MarkusShepherd]

When I pip install turicreate into a fresh virtualenv with Python 3.6, I get error messages:

ERROR: mxnet 1.1.0.post0 has requirement numpy<1.15.0,>=1.8.2, but you'll have numpy 1.16.4 which is incompatible.
ERROR: mxnet 1.1.0.post0 has requirement requests<2.19.0,>=2.18.4, but you'll have requests 2.22.0 which is incompatible.

I'd call that reproducing the issue...

[znation]

@MarkusShepherd As I said in the previous comment,

Using virtualenv and pip, the console shows 'ERROR: ' lines pertaining to mxnet versions, but ignoring those errors, the installation seems to complete and work properly.

[TobyRoseman]

Sadly, we can not fix this in our next release. Although coremltools (one of our dependencies) claims that they supports numpy >= 1.14.5. They actually only support >= 1.16.0.

I opened apple/coremltools#456 with them.

Since MXNet requires numpy >=1.8.2,<1.15.0, there's no way we can make this work, without removing MXNet or using a newer version of MXNet.

[srikris]

Verified a python 3.6 egg using from 6607589.

Installation via virtual-env no longer gives

ERROR: mxnet 1.1.0.post0 has requirement numpy<1.15.0,>=1.8.2, but you'll have numpy 1.16.4 which is incompatible.
ERROR: mxnet 1.1.0.post0 has requirement requests<2.19.0,>=2.18.4, but you'll have requests 2.22.0 which is incompatible.

[MarkusShepherd]

Is it resolved, though? From what I can tell installation might work now as mxnet has been removed as a dependency, but the real underlying cause was the fact that several dependencies are fixed in setup.py. This means the problem is bound to resurface rather sooner than later.

Is there a good reason why numpy et al are fixed to a specific version? This very uncommon for a library, exactly for the reason that it makes resolving dependencies near impossible.

[srikris]

@MarkusShepherd You are right, is is resolved (for now) in our master branch.

@TobyRoseman Here is what our setup.py says. Do you have a recommendation for how we can more permanently fix it.

    install_requires = [
        "coremltools==3.0b3",
        "decorator >= 4.0.9",
        "numpy==1.16.4",
        "pandas >= 0.23.2",
        "pillow >= 5.2.0",
        "prettytable == 0.7.2",
        "resampy == 0.2.1",
        "requests >= 2.9.1",
        "scipy >= 1.1.0",
        "six >= 1.10.0",
        "tensorflow >= 2.0.0"
    ]

[TobyRoseman]

I agree we should not have fixed dependencies. We also shouldn't be depending on a beta version (i.e. coremltools).

In order to avoid potential conflicts in users' environments, numpy is by far the most important dependency to unfix. I believe we fixed the numpy version as a quick way to work around this issue.

[TobyRoseman]

Created to #2664 track unfixing numpy.