Potential dependency conflicts between mlprimitives and numpy

[NeolithEra]

Hi, as shown in the following full dependency graph of mlprimitives, mlprimitives requires numpy >=1.15.2,<1.17, mlprimitives requires statsmodels <1,>=0.9.0 (statsmodels 0.11.1 will be installed, i.e., the newest version satisfying the version constraint), and directed dependency statsmodels 0.11.1 transitively introduces numpy >=1.14.

Obviously, there are multiple version constraints set for numpy in this project. However, according to pip's “first found wins” installation strategy, numpy 1.16.6 (i.e., the newest version satisfying constraint >=1.15.2,<1.17) is the actually installed version.

Although the first found package version numpy 1.16.6 just satisfies the later dependency constraint （numpy >=1.15.2,<1.17), such installed version is very close to the upper bound of the version constraint of numpy specified by statsmodels 0.11.1.

Once statsmodels upgrades，its newest version will be installed, Therefore, it will easily cause a dependency conflict (build failure), if the upgraded statsmodels version introduces a higher version of numpy, violating its another version constraint >=1.15.2,<1.17.

According to the release history of statsmodels, it habitually upgrates Numpy in its recent releases. For instance, statsmodels 0.11.0rc1 upgrated Numpy’s constraint from >=1.11 to >=1.14, and statsmodels next version upgrated Numpy’s constraint from >=1.14 to >=1.15.

As such, it is a warm warning of a potential dependency conflict issue for mlprimitives.

Dependency tree

mlprimitives  - 0.2.4
| +- docutils(install version:0.15.2 version range:>=0.10,<0.16)
| +- featuretools(install version:0.11.0 version range:<0.12,>=0.6.1)
| | +- backports.tempfile(install version:1.0 version range:>=1.0)
| | +- click(install version:7.1.1 version range:>=7.0.0)
| | +- cloudpickle(install version:1.3.0 version range:>=0.4.0)
| | +- dask(install version:2.14.0 version range:>=1.1.0)
| | +- dask(install version:1.2.2 version range:<2.0.0)
| | +- distributed(install version:2.14.0 version range:>=1.24.2)
| | | +- click (install version:7.1.1 version range:>=6.6)
| | | +- cloudpickle (install version:1.3.0 version range:>=0.2.2)
| | | +- dask (install version:2.14.0 version range:>=2.9.0)
| | | +- msgpack (install version:1.0.0 version range:>=0.6.0)
| | | +- psutil (install version:5.7.0 version range:>=5.0)
| | | +- pyyaml(install version:5.3.1 version range:*)
| | | +- setuptools(install version:46.1.3 version range:*)
| | +- distributed(install version:1.28.1 version range:<2.0.0)
| | | +- click(install version:7.1.1 version range:>=6.6)
| | | +- cloudpickle(install version:1.3.0 version range:>=0.2.2)
| | | +- dask(install version:2.14.0 version range:>=0.18.0)
| | | +- futures(install version:3.3.0 version range:*)
| | | +- msgpack(install version:1.0.0 version range:*)
| | | +- psutil(install version:5.7.0 version range:>=5.0)
| | | +- pyyaml(install version:5.3.1 version range:*)
| | | +- singledispatch(install version:3.4.0.3 version range:*)
| | | | +- six(install version:1.14.0 version range:*)
| | | +- six(install version:1.14.0 version range:*)
| | +- funcsigs(install version:1.0.2 version range:>=1.0.2)
| | +- future(install version:0.18.2 version range:>=0.16.0)
| | +- numpy(install version:1.16.6 version range:>=1.13.3)
| | +- pandas(install version:0.24.2 version range:>=0.23.0)
| | +- pathlib(install version:1.0.1 version range:>=1.0.1)
| | +- psutil(install version:5.7.0 version range:>=5.4.8)
| | +- pyyaml(install version:5.3.1 version range:>=3.12)
| | +- s3fs(install version:0.4.2 version range:>=0.2.2)
| | | +- botocore(install version:1.15.39 version range:>=1.12.91)
| | | | +- docutils(install version:0.15.2 version range:>=0.10,<0.16)
| | | | +- jmespath(install version:0.10.0 version range:>=0.7.1,<1.0.0)
| | | | +- python-dateutil(install version:2.8.1 version range:>=2.1,<3.0.0)
| | | | +- urllib3(install version:1.25.9 version range:>=1.20,<1.26)
| | | +- fsspec(install version:0.7.2 version range:>=0.6.0)
| | +- scikit-learn(install version:0.20.4 version range:>=0.20.0)
| | +- scikit-learn(install version:0.20.4 version range:<0.21,>=0.20.0)
| | +- smart-open(install version:1.11.1 version range:>=1.8.4)
| | | +- boto(install version:2.49.0 version range:*)
| | | +- boto3(install version:1.12.39 version range:*)
| | | | +- botocore(install version:1.15.49 version range:>=1.15.39,<1.16.0)
| | | | +- jmespath(install version:0.10.0 version range:>=0.7.1,<1.0.0)
| | | | +- s3transfer(install version:0.3.3 version range:>=0.3.0,<0.4.0)
| | | +- requests(install version:2.23.0 version range:*)
| | | | +- certifi(install version:2020.4.5.1 version range:>=2017.4.17)
| | | | +- chardet(install version:3.0.4 version range:>=3.0.2,<4)
| | | | +- idna(install version:2.9 version range:>=2.5,<3)
| | | | +- urllib3(install version:1.25.9 version range:>=1.21.1,<1.26)
| | +- tqdm(install version:4.45.0 version range:>=4.32.0)
| +- iso639(install version:0.1.4 version range:<0.2,>=0.1.4)
| +- keras(install version:2.3.1 version range:<3,>=2.1.6)
| | +- h5py(install version:2.10.0 version range:*)
| | +- keras_applications(install version: version range:>=1.0.6)
| | +- keras_preprocessing(install version: version range:>=1.0.5)
| | +- numpy(install version:1.16.6 version range:>=1.9.1)
| | +- pyyaml(install version:5.3.1 version range:*)
| | +- scipy(install version:1.4.1 version range:>=0.14)
| | +- six(install version:1.14.0 version range:>=1.9.0)
| +- langdetect(install version:1.0.8 version range:<2,>=1.0.7)
| | +- six(install version:1.14.0 version range:*)
| +- lightfm(install version:1.15 version range:>=1.15,<2)
| | +- 1-15(install version: version range:*)
| | +- numpy(install version:1.16.6 version range:*)
| | +- requests(install version:2.23.0 version range:*)
| | | +- certifi(install version:2020.4.5.1 version range:>=2017.4.17)
| | | +- chardet(install version:3.0.4 version range:>=3.0.2,<4)
| | | +- idna(install version:2.9 version range:>=2.5,<3)
| | | +- urllib3(install version:1.25.9 version range:>=1.21.1,<1.26)
| | +- scipy(install version:1.4.1 version range:>=0.17.0)
| +- mlblocks(install version:0.3.4 version range:>=0.3.4,<0.4)
| +- networkx(install version:2.4 version range:>=2.0,<3)
| | +- decorator(install version:4.4.2 version range:>=4.3.0)
| +- nltk(install version:3.5 version range:>=3.3,<4)
| +- numpy(install version:1.16.6 version range:>=1.15.2,<1.17)
| +- opencv-python(install version:4.2.0.34 version range:<5,>=3.4.0.12)
| +- pandas(install version:0.24.2 version range:>=0.23.4,<0.25)
| +- python-louvain(install version:0.13 version range:>=0.10,<0.14)
| | +- networkx(install version:2.4 version range:*)
| | | +- decorator(install version:4.4.2 version range:>=4.3.0)
| +- scikit-image(install version:0.14.5 version range:<0.15,>=0.13.1)
| +- scikit-learn(install version:0.20.4 version range:>=0.20.0,<0.21)
| +- scipy(install version:1.4.1 version range:<2,>=1.1.0)
| +- setuptools(install version:46.1.3 version range:>=41.0.0)
| +- statsmodels(install version:0.11.1 version range:<1,>=0.9.0)
| | +- numpy(install version:1.16.6 version range:>=1.14)
| | +- pandas(install version:0.24.2 version range:>=0.21)
| | +- patsy(install version:0.5.1 version range:>=0.5)
| | | +- numpy(install version:1.16.6 version range:>=1.4)
| | | +- six(install version:1.14.0 version range:*)
| | +- scipy(install version:1.4.1 version range:>=1.0)
| +- tensorflow(install version:1.15.2 version range:<2,>=1.11.0)
| +- xgboost(install version:0.90 version range:<1,>=0.72.1)

Thanks for your help.
Best,
Neolith

[NeolithEra]

Suggested Solution

1. Loosen the version range of numpy to be >=1.15.2.
2. Remove your direct dependency numpy, and use the numpy transitively introduced by statsmodels.
3. Change your direct dependency statsmodels to be <=0.11.1,>=0.9.0.
   @csala Which solution do you prefer, 1 ,2or 3?
   Please let me know your choice. May I pull a request to solve this issue?

[NeolithEra]

@JDTheRipperPC,@csala，Could you help me review this issue? Thx :p

[csala]

statsmodels is capped on #255 so this is not needed any more