New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Can't resolve dependencies for Gensim / smart-open / botocore #2867

Closed
whyboris opened this Issue Sep 19, 2018 · 10 comments

Comments

Projects
None yet
3 participants
@whyboris

whyboris commented Sep 19, 2018

Issue description

pipenv install gensim fails to create Pipfile.lock because of incompatible required versions in smart-open of botocore.

Expected result

It should install without conflict ... somehow

Actual result

pipenv graph gives this:

gensim==3.5.0
  - numpy [required: >=1.11.3, installed: 1.15.1]
  - scipy [required: >=0.18.1, installed: 1.1.0]
    - numpy [required: >=1.8.2, installed: 1.15.1]
  - six [required: >=1.5.0, installed: 1.11.0]
  - smart-open [required: >=1.2.1, installed: 1.7.0]
    - boto [required: >=2.32, installed: 2.49.0]
    - boto3 [required: <1.8.0, installed: 1.7.84]
      - botocore [required: >=1.10.84,<1.11.0, installed: 1.10.84]
        - docutils [required: >=0.10, installed: 0.14]
        - jmespath [required: >=0.7.1,<1.0.0, installed: 0.9.3]
        - python-dateutil [required: >=2.1,<3.0.0, installed: 2.7.3]
          - six [required: >=1.5, installed: 1.11.0]
      - jmespath [required: >=0.7.1,<1.0.0, installed: 0.9.3]
      - s3transfer [required: >=0.1.10,<0.2.0, installed: 0.1.13]
        - botocore [required: >=1.3.0,<2.0.0, installed: 1.10.84]
          - docutils [required: >=0.10, installed: 0.14]
          - jmespath [required: >=0.7.1,<1.0.0, installed: 0.9.3]
          - python-dateutil [required: >=2.1,<3.0.0, installed: 2.7.3]
            - six [required: >=1.5, installed: 1.11.0]
    - botocore [required: <1.11.0, installed: 1.10.84]
      - docutils [required: >=0.10, installed: 0.14]
      - jmespath [required: >=0.7.1,<1.0.0, installed: 0.9.3]
      - python-dateutil [required: >=2.1,<3.0.0, installed: 2.7.3]
        - six [required: >=1.5, installed: 1.11.0]
    - bz2file [required: Any, installed: 0.98]
    - requests [required: Any, installed: 2.19.1]
      - certifi [required: >=2017.4.17, installed: 2018.8.24]
      - chardet [required: >=3.0.2,<3.1.0, installed: 3.0.4]
      - idna [required: >=2.5,<2.8, installed: 2.7]
      - urllib3 [required: >=1.21.1,<1.24, installed: 1.23]

Specifically, please note these three lines inside smart-open:

- botocore [required: >=1.10.84,<1.11.0,  installed: 1.10.84]
- botocore [required: >=1.3.0,<2.0.0,     installed: 1.10.84]
- botocore [required: <1.11.0,            installed: 1.10.84]

This may be a problem with smart-open rather than pipenv but I can't tell. I posted the issue on smart-open but I don't know enough to decide who needs to adjust what: RaRe-Technologies/smart_open#227

Steps to replicate

pipenv install gensim
Provide the steps to replicate (which usually at least includes the commands and the Pipfile).

pipenv --support:

$ pipenv --support

Pipenv version: '2018.7.1'

Pipenv location: '/usr/local/Cellar/pipenv/2018.7.1/libexec/lib/python3.7/site-packages/pipenv'

Python location: '/usr/local/Cellar/pipenv/2018.7.1/libexec/bin/python3.7'

Other Python installations in PATH:

  • 2.7: /usr/local/bin/python2.7

  • 2.7: /usr/local/bin/python2.7

  • 2.7: /usr/bin/python2.7

  • 3.6: /Library/Frameworks/Python.framework/Versions/3.6/bin/python3.6m

  • 3.6: /Library/Frameworks/Python.framework/Versions/3.6/bin/python3.6

  • 3.6: /usr/local/bin/python3.6

  • 3.7: /Library/Frameworks/Python.framework/Versions/3.7/bin/python3.7m

  • 3.7: /Library/Frameworks/Python.framework/Versions/3.7/bin/python3.7

  • 3.7: /usr/local/bin/python3.7

  • 2.7.15: /usr/local/bin/python

  • 2.7.10: /usr/bin/python

  • 2.7.15: /usr/local/bin/python2

  • 3.7.0: /Library/Frameworks/Python.framework/Versions/3.7/bin/python3

  • 3.6.4: /Library/Frameworks/Python.framework/Versions/3.6/bin/python3

  • 3.7.0: /usr/local/bin/python3

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.7.0',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '17.7.0',
 'platform_system': 'Darwin',
 'platform_version': 'Darwin Kernel Version 17.7.0: Thu Jun 21 22:53:14 PDT '
                     '2018; root:xnu-4570.71.2~1/RELEASE_X86_64',
 'python_full_version': '3.7.0',
 'python_version': '3.7',
 'sys_platform': 'darwin'}

System environment variables:

  • PATH
  • TERM_PROGRAM
  • NVM_CD_FLAGS
  • TERM
  • SHELL
  • TMPDIR
  • Apple_PubSub_Socket_Render
  • TERM_PROGRAM_VERSION
  • ZSH
  • NVM_DIR
  • USER
  • SSH_AUTH_SOCK
  • __CF_USER_TEXT_ENCODING
  • PAGER
  • LSCOLORS
  • _
  • PWD
  • LANG
  • XPC_FLAGS
  • XPC_SERVICE_NAME
  • SHLVL
  • HOME
  • LESS
  • LOGNAME
  • LC_CTYPE
  • NVM_BIN
  • DISPLAY
  • COLORTERM
  • PYTHONDONTWRITEBYTECODE
  • PIP_PYTHON_PATH

Pipenv–specific environment variables:

Debug–specific environment variables:

  • PATH: /usr/local/Cellar/pipenv/2018.7.1/libexec/tools:/usr/local/opt/opencv3/bin:/Users/byakubchik/.nvm/versions/node/v8.11.2/bin:/Library/Frameworks/Python.framework/Versions/3.7/bin:/Library/Frameworks/Python.framework/Versions/3.6/bin:/Library/Frameworks/Python.framework/Versions/2.7/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:~/.composer/vendor/bin:/Library/TeX/texbin:/usr/local/go/bin:/opt/X11/bin
  • SHELL: /bin/zsh
  • LANG: en_US.UTF-8
  • PWD: /Users/byakubchik/Desktop/temp

Contents of Pipfile ('/Users/byakubchik/Desktop/temp/Pipfile'):

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
gensim = "*"

[dev-packages]

[requires]
python_version = "3.7"
@whyboris

This comment has been minimized.

whyboris commented Sep 19, 2018

It looks like pipenv is doing its best, but the versions are simply incompatible -- two dependencies want two different versions of sub-dependencies:

smart_open wants

        'boto3 < 1.8.0',
        'botocore < 1.11.0'

https://github.com/RaRe-Technologies/smart_open/blob/master/setup.py#L54

While boto3 wants

    'botocore>=1.12.6,<1.13.0',

https://github.com/boto/boto3/blob/develop/setup.py#L17

@whyboris

This comment has been minimized.

whyboris commented Sep 19, 2018

Looks like the issue will be resolved on the smart-open end: RaRe-Technologies/smart_open#227 (comment)

I believe that npm and yarn manage to handle different version requirements somehow (e.g. package A wants sub-dependency C at version 1.0 while package B wants sub-dependency C at version 2.0 -- and they make it work in the same project). But this is a tall order, possibly enormously difficult to have pipenv install two different versions and hook up them up to corresponding packages (don't know enough inner workings here). 😅

@petergaultney

This comment has been minimized.

petergaultney commented Sep 19, 2018

@whyboris - what version of Python are you requiring in your Pipfile?

I seem to be seeing this issue only with a Python 3.6 venv, whereas in a Python 3.7 venv, everything works fine. I'm curious if this is reproducible.

pipenv install gensim --python 3.6 (in an empty directory) fails, whereas pipenv install gensim --python 3.7 (in an empty directory) succeeds.

For my use case, I have to have 3.6 for AWS, but it's very interesting that it seems to work in 3.7.

@whyboris

This comment has been minimized.

whyboris commented Sep 19, 2018

The issue was fixed very swiftly at smart_open with a new release that unpins the required dependency.

RaRe-Technologies/smart_open#227 (comment)

I believe the issue I raised is an instance of an inevitable problem when package A wants sub-dependency C at version 1.0 while package B wants sub-dependency C at version 2.0 -- I'm rather surprised you didn't run into a problem.

@petergaultney - I was unable to replicate the error now that the package has been updated, but I too am curious why you experienced different behavior between the two versions of Python.

@petergaultney

This comment has been minimized.

petergaultney commented Sep 19, 2018

i can still replicate it with pipenv install smart_open==1.7.0 gensim --python 3.6 in an empty directory.

@whyboris

This comment has been minimized.

whyboris commented Sep 19, 2018

@petergaultney -- thanks for that command -- I was able to replicate the problem with
pipenv install smart_open==1.7.0 gensim --python 3.6
but the problem occurred with python 3.7 too in an empty directory:
pipenv install smart_open==1.7.0 gensim --python 3.7
I suspect what you experienced before was a fluke 🤷‍♂️ ?

@petergaultney

This comment has been minimized.

petergaultney commented Sep 19, 2018

Thanks for trying. It doesn't make any sense to me, but weirdly enough I can still get pipenv install smart_open==1.7.0 gensim --python 3.7 to pass cleanly, simply running in an empty directory. A ghost in the machine, I suppose? It doesn't enhance my confidence in the Python packaging ecosystem, unfortunately...

screenshot 2018-09-19 12 02 51

@uranusjr

This comment has been minimized.

Member

uranusjr commented Sep 19, 2018

Try pipenv lock —clear to purge the resolution cache.

@petergaultney

This comment has been minimized.

petergaultney commented Sep 19, 2018

that does indeed cause the error as expected. Are there different caches per Python version?

@uranusjr

This comment has been minimized.

Member

uranusjr commented Sep 20, 2018

Yes, the cache is partitioned by version.

techalchemy added a commit that referenced this issue Oct 7, 2018

Add news fragments and final patch for all fixes
- Fixes #2499
- Fixes #2529
- Fixes #2589
- Fixes #2666
- Fixes #2767
- Fixes #2785
- Fixes #2795
- Fixes #2801
- Fixes #2802
- Fixes #2824
- Fixes #2862
- Fixes #2867
- Fixes #2879
- Fixes #2880
- Fixes #2894
- Fixes #2902
- Fixes #2924

Signed-off-by: Dan Ryan <dan@danryan.co>

techalchemy added a commit that referenced this issue Oct 7, 2018

Add news fragments and final patch for all fixes
- Fixes #2499
- Fixes #2529
- Fixes #2589
- Fixes #2666
- Fixes #2767
- Fixes #2785
- Fixes #2795
- Fixes #2801
- Fixes #2802
- Fixes #2824
- Fixes #2862
- Fixes #2867
- Fixes #2879
- Fixes #2880
- Fixes #2894
- Fixes #2902
- Fixes #2924

Signed-off-by: Dan Ryan <dan@danryan.co>

GilbertoCS added a commit to GilbertoCS/pipenv that referenced this issue Oct 15, 2018

Add news fragments and final patch for all fixes
- Fixes pypa#2499
- Fixes pypa#2529
- Fixes pypa#2589
- Fixes pypa#2666
- Fixes pypa#2767
- Fixes pypa#2785
- Fixes pypa#2795
- Fixes pypa#2801
- Fixes pypa#2802
- Fixes pypa#2824
- Fixes pypa#2862
- Fixes pypa#2867
- Fixes pypa#2879
- Fixes pypa#2880
- Fixes pypa#2894
- Fixes pypa#2902
- Fixes pypa#2924

Signed-off-by: Dan Ryan <dan@danryan.co>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment