-
-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Packages using sys.version_info in setup.py may detect wrong version #924
Comments
Just doing some more thinking about this and I suspect there may not be a great solution in Pipenv itself. In my case, I am specifying the python version in requires, but if someone is not, then I'm not sure there's a good way to deal with this. Possibly using markers and specifying the problem dependencies with markers in the Pipfile? I will test and report my results back here. |
That deals with the weasyprint/cairosvg situation if I want to force it to install only the one version of cairosvg (such as for python 2.7 but ignore python 3). For the python-openid / python3-openid thing it still gets weird because python3-openid's dependencies still end up in my lock file without the markers saying they are python 3 only. I'll also bug the package maintainers about this to see if they can solve the problems in a better way, but since I've got two of these on just one project, I have a feeling it may be a more common situation than it should be. |
It's really on the package makers/maintainers to abide by the proper dependecy resolution in their setup.py files. PEP has specifications for this. The behavior you describe is odd however and I will investigate, feel free to check out our vendorered packaging and pip resolution tools. |
Yep, I definitely agree that the root of this is on package maintainers. Without the python version being specified, I don't see how Pipenv even could deal with it reasonably. At least not without adding some more command line options, unless I've missed something in the markers where it can be specified to not install something and/or to ignore dependencies. The immediate issue is more ammo for my "we need to get everything on python 3 push", but there's really nothing stopping this sort of thing from happening on python 3.6 vs 3.7 or even 3.6.3 vs 3.6.4. I appreciate any time you guys are able to put into thinking about a reasonable solution for this sort of situation. I'll keep plugging away at our environments I am trying to get switched over to pipenv to see if I find a good, clean solution with existing functionality. |
As a point of clarification this issue looks different from some of the other dependency resolution issues at face value. However, upon testing it is not. Using a fresh python3 install of pipenv (current master 94640b8) I installed
~ vf tmp --python=python3.6
Running virtualenv with interpreter /home/hawk/.pyenv/shims/python3.6
Using base prefix '/home/hawk/.pyenv/versions/3.6.3'
New python executable in /home/hawk/.virtualenvs/tempenv-364f68930438/bin/python3.6
Also creating executable in /home/hawk/.virtualenvs/tempenv-364f68930438/bin/python
Installing setuptools, pip, wheel...done.
◰³ tempenv-364f68930438 ~ set -gx PIPENV_IGNORE_VIRTUALENVS 1; and set -gx PIPENV_VENV_IN_PROJECT 1; and mkdir /tmp/new
◰³ tempenv-364f68930438 ~ cd /tmp/new
◰³ tempenv-364f68930438 /t/new pip install -e ~/git/pipenv/
.....
Installing collected packages: virtualenv, virtualenv-clone, pew, certifi, idna, urllib3, chardet, requests, mccabe, pycodestyle, pyflakes, flake8, pipenv
Running setup.py develop for pipenv
Successfully installed certifi-2017.11.5 chardet-3.0.4 flake8-3.5.0 idna-2.6 mccabe-0.6.1 pew-1.1.0 pipenv pycodestyle-2.3.1 pyflakes-1.6.0 requests-2.18.4 urllib3-1.22 virtualenv-15.1.0 virtualenv-clone-0.2.6
◰³ tempenv-364f68930438 /t/new pipenv install --python 2.7.14 social-auth-app-django
Creating a virtualenv for this project…
Using /home/hawk/.pyenv/versions/2.7.14/bin/python2.7 to create virtualenv…
⠋Running virtualenv with interpreter /home/hawk/.pyenv/versions/2.7.14/bin/python2.7
New python executable in /tmp/new/.venv/bin/python2.7
Also creating executable in /tmp/new/.venv/bin/python
Installing setuptools, pip, wheel...done.
Virtualenv location: /tmp/new/.venv
Creating a Pipfile for this project…
Installing social-auth-app-django…
Collecting social-auth-app-django
Downloading social_auth_app_django-2.0.0-py2-none-any.whl
Collecting six (from social-auth-app-django)
Using cached six-1.11.0-py2.py3-none-any.whl
Collecting social-auth-core>=1.2.0 (from social-auth-app-django)
Downloading social_auth_core-1.5.0-py2-none-any.whl (278kB)
Collecting requests>=2.9.1 (from social-auth-core>=1.2.0->social-auth-app-django)
Using cached requests-2.18.4-py2.py3-none-any.whl
Collecting python-openid>=2.2.5 (from social-auth-core>=1.2.0->social-auth-app-django)
Collecting PyJWT>=1.4.0 (from social-auth-core>=1.2.0->social-auth-app-django)
Using cached PyJWT-1.5.3-py2.py3-none-any.whl
Collecting oauthlib>=1.0.3 (from social-auth-core>=1.2.0->social-auth-app-django)
Collecting requests-oauthlib>=0.6.1 (from social-auth-core>=1.2.0->social-auth-app-django)
Using cached requests_oauthlib-0.8.0-py2.py3-none-any.whl
Collecting idna<2.7,>=2.5 (from requests>=2.9.1->social-auth-core>=1.2.0->social-auth-app-django)
Using cached idna-2.6-py2.py3-none-any.whl
Collecting urllib3<1.23,>=1.21.1 (from requests>=2.9.1->social-auth-core>=1.2.0->social-auth-app-django)
Using cached urllib3-1.22-py2.py3-none-any.whl
Collecting certifi>=2017.4.17 (from requests>=2.9.1->social-auth-core>=1.2.0->social-auth-app-django)
Using cached certifi-2017.11.5-py2.py3-none-any.whl
Collecting chardet<3.1.0,>=3.0.2 (from requests>=2.9.1->social-auth-core>=1.2.0->social-auth-app-django)
Using cached chardet-3.0.4-py2.py3-none-any.whl
Installing collected packages: six, idna, urllib3, certifi, chardet, requests, python-openid, PyJWT, oauthlib, requests-oauthlib, social-auth-core, social-auth-app-django
Successfully installed PyJWT-1.5.3 certifi-2017.11.5 chardet-3.0.4 idna-2.6 oauthlib-2.0.6 python-openid-2.2.5 requests-2.18.4 requests-oauthlib-0.8.0 six-1.11.0 social-auth-app-django-2.0.0 social-auth-core-1.5.0 urllib3-1.22
Adding social-auth-app-django to Pipfile's [packages]…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
Updated Pipfile.lock (7d56b7)!
◰³ tempenv-364f68930438 /t/new pipenv run pip freeze
certifi==2017.11.5
chardet==3.0.4
idna==2.6
oauthlib==2.0.6
PyJWT==1.5.3
python-openid==2.2.5
requests==2.18.4
requests-oauthlib==0.8.0
six==1.11.0
social-auth-app-django==2.0.0
social-auth-core==1.5.0
urllib3==1.22
{
"_meta": {
"hash": {
"sha256": "26d5e5df1bfda346786d908c52739fcc7a551d885287f14b23d8bd37e17d56b7"
},
"host-environment-markers": {
"implementation_name": "cpython",
"implementation_version": "0",
"os_name": "posix",
"platform_machine": "x86_64",
"platform_python_implementation": "CPython",
"platform_release": "4.10.0-38-generic",
"platform_system": "Linux",
"platform_version": "#42-Ubuntu SMP Tue Oct 10 13:24:27 UTC 2017",
"python_full_version": "2.7.14",
"python_version": "2.7",
"sys_platform": "linux2"
},
"pipfile-spec": 6,
"requires": {
"python_version": "2.7"
},
"sources": [
{
"name": "pypi",
"url": "https://pypi.python.org/simple",
"verify_ssl": true
}
]
},
"default": {
"certifi": {
"hashes": [
"sha256:244be0d93b71e93fc0a0a479862051414d0e00e16435707e5bf5000f92e04694",
"sha256:5ec74291ca1136b40f0379e1128ff80e866597e4e2c1e755739a913bbc3613c0"
],
"version": "==2017.11.5"
},
"chardet": {
"hashes": [
"sha256:fc323ffcaeaed0e0a02bf4d117757b98aed530d9ed4531e3e15460124c106691",
"sha256:84ab92ed1c4d4f16916e05906b6b75a6c0fb5db821cc65e70cbd64a3e2a5eaae"
],
"version": "==3.0.4"
},
"defusedxml": {
"hashes": [
"sha256:702a91ade2968a82beb0db1e0766a6a273f33d4616a6ce8cde475d8e09853b20",
"sha256:24d7f2f94f7f3cb6061acb215685e5125fbcdc40a857eff9de22518820b0a4f4"
],
"version": "==0.5.0"
},
"idna": {
"hashes": [
"sha256:8c7309c718f94b3a625cb648ace320157ad16ff131ae0af362c9f21b80ef6ec4",
"sha256:2c6a5de3089009e3da7c5dde64a141dbc8551d5b7f6cf4ed7c2568d0cc520a8f"
],
"version": "==2.6"
},
"oauthlib": {
"hashes": [
"sha256:ce57b501e906ff4f614e71c36a3ab9eacbb96d35c24d1970d2539bbc3ec70ce1"
],
"version": "==2.0.6"
},
"pyjwt": {
"hashes": [
"sha256:a4e5f1441e3ca7b382fd0c0b416777ced1f97c64ef0c33bfa39daf38505cfd2f",
"sha256:500be75b17a63f70072416843dc80c8821109030be824f4d14758f114978bae7"
],
"version": "==1.5.3"
},
"python3-openid": {
"hashes": [
"sha256:0086da6b6ef3161cfe50fb1ee5cceaf2cda1700019fda03c2c5c440ca6abe4fa",
"sha256:628d365d687e12da12d02c6691170f4451db28d6d68d050007e4a40065868502"
],
"version": "==3.1.0"
},
"requests": {
"hashes": [
"sha256:6a1b267aa90cac58ac3a765d067950e7dbbf75b1da07e895d1f594193a40a38b",
"sha256:9c443e7324ba5b85070c4a818ade28bfabedf16ea10206da1132edaa6dda237e"
],
"version": "==2.18.4"
},
"requests-oauthlib": {
"hashes": [
"sha256:50a8ae2ce8273e384895972b56193c7409601a66d4975774c60c2aed869639ca",
"sha256:883ac416757eada6d3d07054ec7092ac21c7f35cb1d2cf82faf205637081f468"
],
"version": "==0.8.0"
},
"six": {
"hashes": [
"sha256:832dc0e10feb1aa2c68dcc57dbb658f1c7e65b9b61af69048abc87a2db00a0eb",
"sha256:70e8a77beed4562e7f14fe23a786b54f6296e34344c23bc42f07b15018ff98e9"
],
"version": "==1.11.0"
},
"social-auth-app-django": {
"hashes": [
"sha256:22976fae902d5ccb333f9ac0db23a2a0b97ab4d9367655c17a740f89c044b3b1",
"sha256:1c16b994ccaf1574835ec1e2d53916d885952b795d52df094fa434cd2f3227c8",
"sha256:15a33f7cd663a2624fdd5deafa7992044b777225da57ae1f81d790196d8d43d1"
],
"version": "==2.0.0"
},
"social-auth-core": {
"hashes": [
"sha256:ab184b279addd029364bbe175d1a8bb21754287d7048a3b860ead6c6b3a85643",
"sha256:42bb40aef25c462cd02eba9a8e6c30251cc26fc58c1e8c6f1ebd94055c3b8e93",
"sha256:49054c01619f6b1a9bc4c68b010a4b223f86a0aeda376bc1f2d0fa53f56c26dc"
],
"version": "==1.5.0"
},
"urllib3": {
"hashes": [
"sha256:06330f386d6e4b195fbfc736b297f58c5a892e4440e54d294d7004e3a9bbea1b",
"sha256:cc44da8e1145637334317feebd728bd869a35285b93cbb4cca2577da7e62db4f"
],
"version": "==1.22"
}
},
"develop": {}
} Note that this is because of how dependency resolution works when generating the lockfile (it uses |
I think that this might be fixed by the enhancement Kenneth and I just wrote. Can you test in master and see if it works @jmichalicek @techalchemy |
I will try to make some time to give it a test today |
@erinxocon Looks like it might be all set. Thanks to everyone working on this, I really didn't expect a fix after I got to digging into it more since the root cause really seems to be other packages not handling these things correctly. It's in the Pipfile.lock, but what actually gets installed looks good
|
Unfortunately we may have just reverted this fix |
Yep, looks like it. I see the revert merge there and when I install pipenv from master (as I see I was asked to do in the first place but missed it) the python3 stuff is back in |
We reverted the merge because it subsequently broke a bunch of things, and the implications were kind of bad / it seemed likely that more things would break. Had we kept it we also would have had to merge #1082 which modified This is a kind of large task but it’s not impossible. The real issue here is that we need to explicitly fake abi information, and package resolution is handled variously by pip, distutils, packaging, pkg_resources, setuptools, etc, and not only do they all look up version information differently, but even internally each of them does so in potentially half a dozen ways ranging from looking directly at config.h if it’s available to sys.executable invocations to sys.version_info to environment vars. |
Thanks for the explanation. I appreciate all of the effort going into this. It definitely will help with usability and behaving in an expected manner. The more I thought about this after reporting it, the more it sounded like the sort of thing which initially sounds like a good idea but could spiral out of control into a crazy, impossible to maintain mess, though. So I totally understand. I'm going to try to keep an eye on the code for the end fix. I suspect I will learn a lot from watching how you guys do that. |
Be sure to check the existing issues, both open and closed.
Using Python 3 as my main python and pipenv to create and manage a Python 2.7 project, dependencies which use sys.version_info in their setup.py appear to be picking up the wrong version.
I have seen this with two different packages today.
When installing social-auth-app-django and python-social-core the python-social-core app uses sys.version_info to determine whether to install python-openid or python3-openid.
Weasyprint uses the same technique to determine which version of CairSVG to install and consistently takes the python 3 path when installing into a Python 2 environment.
Describe you environment
OS: OS X Sierra 10.12.6
Python and pipenv info:
➜ t python -V
Python 3.6.3
➜ t pipenv --version
pipenv, version 8.2.7
➜ t pipenv run python -V
Python 2.7.12
Expected result
When installing social-auth-app-django and python-social-core the python-social-core app uses sys.version_info to determine whether to install python-openid or python3-openid. With the environment above I should get python-openid in my Pipfile.lock
Actual result
I am actually getting python3-openid in my Pipfile.lock. Here is various pipenv and pip output after installation.
social-auth-app-django results
pipenv graph outout:
Pipfile.lock includes the following:
pip freeze from pipenv shell:
weasyprint results
Steps to replicate
social-auth-app-django Pipfile used to test:
weasyprint Pipfile
Here are links to the actual code having problems:
social-core: https://github.com/python-social-auth/social-core/blob/19382028ef1af08782130f5aeb90ef591955f2ba/setup.py#L46
weasyprint: https://github.com/Kozea/WeasyPrint/blob/master/setup.py#L42
The text was updated successfully, but these errors were encountered: