Can't resolve dependencies for Gensim / smart-open / botocore

[whyboris]

Issue description

pipenv install gensim fails to create Pipfile.lock because of incompatible required versions in smart-open of botocore.

Expected result

It should install without conflict ... somehow

Actual result

pipenv graph gives this:

gensim==3.5.0
  - numpy [required: >=1.11.3, installed: 1.15.1]
  - scipy [required: >=0.18.1, installed: 1.1.0]
    - numpy [required: >=1.8.2, installed: 1.15.1]
  - six [required: >=1.5.0, installed: 1.11.0]
  - smart-open [required: >=1.2.1, installed: 1.7.0]
    - boto [required: >=2.32, installed: 2.49.0]
    - boto3 [required: <1.8.0, installed: 1.7.84]
      - botocore [required: >=1.10.84,<1.11.0, installed: 1.10.84]
        - docutils [required: >=0.10, installed: 0.14]
        - jmespath [required: >=0.7.1,<1.0.0, installed: 0.9.3]
        - python-dateutil [required: >=2.1,<3.0.0, installed: 2.7.3]
          - six [required: >=1.5, installed: 1.11.0]
      - jmespath [required: >=0.7.1,<1.0.0, installed: 0.9.3]
      - s3transfer [required: >=0.1.10,<0.2.0, installed: 0.1.13]
        - botocore [required: >=1.3.0,<2.0.0, installed: 1.10.84]
          - docutils [required: >=0.10, installed: 0.14]
          - jmespath [required: >=0.7.1,<1.0.0, installed: 0.9.3]
          - python-dateutil [required: >=2.1,<3.0.0, installed: 2.7.3]
            - six [required: >=1.5, installed: 1.11.0]
    - botocore [required: <1.11.0, installed: 1.10.84]
      - docutils [required: >=0.10, installed: 0.14]
      - jmespath [required: >=0.7.1,<1.0.0, installed: 0.9.3]
      - python-dateutil [required: >=2.1,<3.0.0, installed: 2.7.3]
        - six [required: >=1.5, installed: 1.11.0]
    - bz2file [required: Any, installed: 0.98]
    - requests [required: Any, installed: 2.19.1]
      - certifi [required: >=2017.4.17, installed: 2018.8.24]
      - chardet [required: >=3.0.2,<3.1.0, installed: 3.0.4]
      - idna [required: >=2.5,<2.8, installed: 2.7]
      - urllib3 [required: >=1.21.1,<1.24, installed: 1.23]

Specifically, please note these three lines inside smart-open:

- botocore [required: >=1.10.84,<1.11.0,  installed: 1.10.84]
- botocore [required: >=1.3.0,<2.0.0,     installed: 1.10.84]
- botocore [required: <1.11.0,            installed: 1.10.84]

This may be a problem with smart-open rather than pipenv but I can't tell. I posted the issue on smart-open but I don't know enough to decide who needs to adjust what: RaRe-Technologies/smart_open#227

Steps to replicate

pipenv install gensim
Provide the steps to replicate (which usually at least includes the commands and the Pipfile).

pipenv --support:

$ pipenv --support

Pipenv version: '2018.7.1'

Pipenv location: '/usr/local/Cellar/pipenv/2018.7.1/libexec/lib/python3.7/site-packages/pipenv'

Python location: '/usr/local/Cellar/pipenv/2018.7.1/libexec/bin/python3.7'

Other Python installations in PATH:

• 2.7: /usr/local/bin/python2.7

• 2.7: /usr/local/bin/python2.7

• 2.7: /usr/bin/python2.7

• 3.6: /Library/Frameworks/Python.framework/Versions/3.6/bin/python3.6m

• 3.6: /Library/Frameworks/Python.framework/Versions/3.6/bin/python3.6

• 3.6: /usr/local/bin/python3.6

• 3.7: /Library/Frameworks/Python.framework/Versions/3.7/bin/python3.7m

• 3.7: /Library/Frameworks/Python.framework/Versions/3.7/bin/python3.7

• 3.7: /usr/local/bin/python3.7

• 2.7.15: /usr/local/bin/python

• 2.7.10: /usr/bin/python

• 2.7.15: /usr/local/bin/python2

• 3.7.0: /Library/Frameworks/Python.framework/Versions/3.7/bin/python3

• 3.6.4: /Library/Frameworks/Python.framework/Versions/3.6/bin/python3

• 3.7.0: /usr/local/bin/python3

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.7.0',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '17.7.0',
 'platform_system': 'Darwin',
 'platform_version': 'Darwin Kernel Version 17.7.0: Thu Jun 21 22:53:14 PDT '
                     '2018; root:xnu-4570.71.2~1/RELEASE_X86_64',
 'python_full_version': '3.7.0',
 'python_version': '3.7',
 'sys_platform': 'darwin'}

System environment variables:

• PATH
• TERM_PROGRAM
• NVM_CD_FLAGS
• TERM
• SHELL
• TMPDIR
• Apple_PubSub_Socket_Render
• TERM_PROGRAM_VERSION
• ZSH
• NVM_DIR
• USER
• SSH_AUTH_SOCK
• __CF_USER_TEXT_ENCODING
• PAGER
• LSCOLORS
• _
• PWD
• LANG
• XPC_FLAGS
• XPC_SERVICE_NAME
• SHLVL
• HOME
• LESS
• LOGNAME
• LC_CTYPE
• NVM_BIN
• DISPLAY
• COLORTERM
• PYTHONDONTWRITEBYTECODE
• PIP_PYTHON_PATH

Pipenv–specific environment variables:

Debug–specific environment variables:

• PATH: /usr/local/Cellar/pipenv/2018.7.1/libexec/tools:/usr/local/opt/opencv3/bin:/Users/byakubchik/.nvm/versions/node/v8.11.2/bin:/Library/Frameworks/Python.framework/Versions/3.7/bin:/Library/Frameworks/Python.framework/Versions/3.6/bin:/Library/Frameworks/Python.framework/Versions/2.7/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:~/.composer/vendor/bin:/Library/TeX/texbin:/usr/local/go/bin:/opt/X11/bin
• SHELL: /bin/zsh
• LANG: en_US.UTF-8
• PWD: /Users/byakubchik/Desktop/temp

---

Contents of Pipfile ('/Users/byakubchik/Desktop/temp/Pipfile'):

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
gensim = "*"

[dev-packages]

[requires]
python_version = "3.7"

[whyboris]

It looks like pipenv is doing its best, but the versions are simply incompatible -- two dependencies want two different versions of sub-dependencies:

smart_open wants

        'boto3 < 1.8.0',
        'botocore < 1.11.0'

https://github.com/RaRe-Technologies/smart_open/blob/master/setup.py#L54

While boto3 wants

    'botocore>=1.12.6,<1.13.0',

https://github.com/boto/boto3/blob/develop/setup.py#L17

[whyboris]

Looks like the issue will be resolved on the smart-open end: RaRe-Technologies/smart_open#227 (comment)

I believe that npm and yarn manage to handle different version requirements somehow (e.g. package A wants sub-dependency C at version 1.0 while package B wants sub-dependency C at version 2.0 -- and they make it work in the same project). But this is a tall order, possibly enormously difficult to have pipenv install two different versions and hook up them up to corresponding packages (don't know enough inner workings here). 😅

[petergaultney]

@whyboris - what version of Python are you requiring in your Pipfile?

I seem to be seeing this issue only with a Python 3.6 venv, whereas in a Python 3.7 venv, everything works fine. I'm curious if this is reproducible.

pipenv install gensim --python 3.6 (in an empty directory) fails, whereas pipenv install gensim --python 3.7 (in an empty directory) succeeds.

For my use case, I have to have 3.6 for AWS, but it's very interesting that it seems to work in 3.7.

[whyboris]

The issue was fixed very swiftly at smart_open with a new release that unpins the required dependency.

RaRe-Technologies/smart_open#227 (comment)

I believe the issue I raised is an instance of an inevitable problem when package A wants sub-dependency C at version 1.0 while package B wants sub-dependency C at version 2.0 -- I'm rather surprised you didn't run into a problem.

@petergaultney - I was unable to replicate the error now that the package has been updated, but I too am curious why you experienced different behavior between the two versions of Python.

[petergaultney]

i can still replicate it with pipenv install smart_open==1.7.0 gensim --python 3.6 in an empty directory.

[whyboris]

@petergaultney -- thanks for that command -- I was able to replicate the problem with
pipenv install smart_open==1.7.0 gensim --python 3.6
but the problem occurred with python 3.7 too in an empty directory:
pipenv install smart_open==1.7.0 gensim --python 3.7
I suspect what you experienced before was a fluke 🤷‍♂️ ?

[petergaultney]

Thanks for trying. It doesn't make any sense to me, but weirdly enough I can still get pipenv install smart_open==1.7.0 gensim --python 3.7 to pass cleanly, simply running in an empty directory. A ghost in the machine, I suppose? It doesn't enhance my confidence in the Python packaging ecosystem, unfortunately...

[uranusjr]

Try pipenv lock —clear to purge the resolution cache.

[petergaultney]

that does indeed cause the error as expected. Are there different caches per Python version?

[uranusjr]

Yes, the cache is partitioned by version.