Skip to content
This repository

Use wheels for setuptools/pip installs #482

merged 2 commits into from 8 months ago

3 participants

Paul Moore Marcus Smith Donald Stufft
Paul Moore

No description provided.

Marcus Smith

just to be clear, the pip/setuptools wheels in here contains only the windows scripts.
this is just a a WIP PR, right?
we would need this first. pypa/pip#1067

Paul Moore

Ha, well spotted. It wasn't WIP in that sense, I'd merely generated the wheels using "pip wheel setuptools" and "pip wheel pip" (on Windows). It never occurred to me that this wouldn't (yet) work.

Yes, pypa/pip#1067 is the missing link here. Sigh, I thought all the pieces needed in pip were in place. And there's no PR for that issue yet, as far as I can see.

Count this PR as "ready to go pending that issue, but a big enough change that it could do with some more reviews" then...

Paul Moore

@dstufft you should be aware of the issue @qwcode pointed out - it will affect ensurepip as well...

Paul Moore

Updated this PR:

  1. Regenerated the pip and setuptools wheels, they now don't include wrappers and the pip wheel is up to date with the script generation code. Tested on Windows, some testing on Unix would be good if anyone has a moment.
  2. Switched the install command to just use "python -m pip" as it's simpler.
  3. Rebased to latest upstream.

Basically good to go now, but see my question on pypa-dev.

Paul Moore

Rats! Python 2.6 does not support "python -m pip" (direct execution of packages). I will fix this, but I wonder whether we should consider dropping support for Python 2.6, given that "python -m pip install -U pip" is the recommended means for upgrading pip to the latest version, at least on Windows where updating using the wrapper exe fails.

Paul Moore

Consensus on pypa-dev suggests that we leave this patch as is, installing the local versions of setuptools and pip that are supplied with virtualenv. Users can add later versions to virtualenv_support if they want to manually include a later version (--extra-search-dir will help here)

A --use-pypi option that adds pypi to the search could be added, but should be a separate PR.

Over the weekend, I'll update the bundled wheels to latest and greatest, and update bin/ Then, unless there are any final objectsions I'll merge.

@dstufft, @qwcode please comment if I've misunderstood anything that was said on pypa-dev.

Marcus Smith

to be clear, this means users can't install pip/setuptools sdists at all (using --extra-search-dir). That might frustrate people who might want a specific older version for some reason. But I guess if that's true, they can build a wheel for it.

Marcus Smith qwcode commented on the diff
((8 lines not shown))
  916 + Scan through SEARCH_DIRS for a wheel for each PROJECT in turn. Return
  917 + a list of the first wheel found for each PROJECT
  918 + """
  919 +
  920 + wheels = []
  921 +
  922 + # Look through SEARCH_DIRS for the first suitable wheel. Don't bother
  923 + # about version checking here, as this is simply to get something we can
  924 + # then use to install the correct version.
  925 + for project in projects:
  926 + for dirname in search_dirs:
  927 + # This relies on only having "universal" wheels available.
  928 + # The pattern could be tightened to require -py2.py3-none-any.whl.
  929 + files = glob.glob(os.path.join(dirname, project + '-*.whl'))
  930 + if files:
  931 + wheels.append(os.path.abspath(files[0]))
Marcus Smith Owner
qwcode added a note

see #497. let's go ahead and fix that here.

Paul Moore Collaborator
pfmoore added a note

This is not the same as #497. The point here is just to get any workable setuptools/pip combination on PYTHONPATH. That version is then used to install the "real version", and does so by a normal "pip install" which will search all specified directories with the normal rules (select the latest version compatible wheel out of all wheels discovered).

The fact that we now use a normal pip install probably means that we've fixed #497 in passing, though.

Marcus Smith Owner
qwcode added a note

#497 is about not assuming that files[0] is a file (and not a symlink). not sure exactly when that comes up, but this code is still vulnerable to that I think. I made a comment in #497. maybe he can explain more when it happens.

Paul Moore Collaborator
pfmoore added a note

I see what you mean now. That seems to me an insane breakage of a virtualenv installation (or an incorrect usage of --extra-search-dirs) but if it can happen in real life, we should indeed fix it. I'll see what I can do.

Paul Moore Collaborator
pfmoore added a note

I have just noticed that this code looks through search_dirs in order - unlike the old _find_files which went in reversed order. This means that virtualenv_support will always be checked before user directories. This is good, because it means that we always use the bundled pip/setuptools to do the install. But it means that we need to do a second search for the wheels to install if we want to use an explicit search of the user search directories rather than pip's search algorithm (see the comments from @qwcode on the block of code below).

For now, I'm just noting this in case the code gets changed as @qwcode suggests. It will be important to clearly comment the code to explain the reason for the double search, as otherwise it could become a maintenance problem.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Paul Moore

You're right (I think) that this won't install sdists. Actually, it might - I can do some testing and report back, it depends on whether pip's setuptools injection works with an setuptools wheel on PYTHONPATH but not in site-packages.

It wouldn't help people wanting to install older versions, though - we're doing pip install setuptools pip in effect, so that will always take the latest version. It might be useful for installing dev versions (although I'd expect to remove the --pre flag on release as we'll be shipping release versions at that point).

Hacking virtualenv to install something other than the bundled version was never fully supported, anyway, so I'm inclined not to stress too much about it here. The new process probably gives more flexibility to allow customisation in the long run, but I'd like to worry about that later.
((56 lines not shown))
930   - '--record', 'record']
931   - logger.start_progress('Installing %s...' % project_name)
932   - logger.indent += 2
933   - try:
934   - call_subprocess(cmd, show_stdout=False, cwd=srcdir,
935   - filter_stdout=filter_install_output)
936   - finally:
937   - logger.indent -= 2
938   - logger.end_progress()
  953 + call_subprocess(cmd, show_stdout=False,
  954 + extra_env = {
  955 + 'PYTHONPATH': pythonpath,
  956 + 'PIP_FIND_LINKS': findlinks,
  957 + 'PIP_USE_WHEEL': '1',
  958 + 'PIP_PRE': '1',
  959 + 'PIP_UPGRADE': '1',
Marcus Smith Owner
qwcode added a note

why upgrade? you're installing in a clean environment

Paul Moore Collaborator
pfmoore added a note

I think there was a reason at some point. I can't remember now, maybe because if we omit it, pip sees the uninstalled wheel on PYTHONPATH as "already there" and won't install. I can do some experiments to establish the details and add a comment to document it. But it's harmless, so I don't think it's a big deal.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Marcus Smith

I'd expect to remove the --pre flag on release as we'll be shipping release versions at that point

I think you should leave it. we want the ability to install pre versions from --extra-search-dirs.

Marcus Smith qwcode commented on the diff
((38 lines not shown))
921 942
922   - tmpdir = tempfile.mkdtemp()
  943 + wheels = find_wheels(['setuptools', 'pip'], search_dirs)
  944 + pythonpath = os.pathsep.join(wheels)
  945 + findlinks = ' '.join(search_dirs)
Marcus Smith Owner
qwcode added a note

OH.....hmm. this is a change. you'll always be installing the latest version found in all of the search dirs.

as it is now, it installs the first version found in the directory search order.

I think the current behavior makes sense.

Paul Moore Collaborator
pfmoore added a note

That's not practical with the new approach (which uses pip to install pip/setuptools and pip doesn't have a "use the first match" approach). If it's not acceptable to change the behaviour, I don't think this whole approach is viable.

What we could do is add new --setuptools-req and ---pip-req options that let you specify a requirement. You could then specify an exact version. But it's still different behaviour. And I suspect it's overcomplicating things for a relatively rare use case.

Is the current behaviour by design, or just a consequence of the old "search for a sdist and install it" algorithm? (I ask because I don't particularly agree that the current behaviour is sensible - it's the only place in the packaging system where we use a "first found" algorithm rather than a "best match/latest version" one, so I see it as a wart).

Marcus Smith Owner
qwcode added a note

--extra-search-dir is specifically about giving the user the power to use their own version. the idea is that it's supposed to find the first one in the search order, because that's the one they are telling us to use.

I don't understand why you can just find the first one like it used to and just install directly from that file path (like it used to). you don't need to use --find-links and --no-index.

Paul Moore Collaborator
pfmoore added a note

Because then I'll need to replicate the compatibility checking that pip does (I know pip and setuptools are universal, but if I find a file named pip-1.5-py33-none-any.whl, and I'm on Python 2.7, I'd need to skip it as pip would fail to install it). I could do something simple and "good enough", but that risks odd corner cases and bugs.

I'm actually not sure this is even worth it any more. Now that setuptools no longer uses 2to3, installation from sdist is fast enough that the performance argument is moot. And I'm not sure there is any other real justification for the switch.

Marcus Smith Owner
qwcode added a note

It seems reasonable to me to just fail (and show the stdout from pip), if someone sticks an unsupported wheel in their extra-search-dir. I.e. virtualenv itself, doesn't need to to the check, and skip.

btw, I actually only just added the checks for wheel support when installing from files directly (pypa/pip#1315).
With that, pip will fail early when trying to install an invalid or unsupported wheel.

Paul Moore Collaborator
pfmoore added a note

So your suggestion would be:

Check each entry in search_dirs for glob.glob("setuptools*.whl") - if you find one, install it by explicit filename. Same for pip*.whl. Skip symlinks and directories, presumably?

OK. That puts back a (small) chunk of complexity that I'd removed by relying on pip doing the install, but I guess it's viable.

And yes, I know you added the checks recently - without those, the above approach would definitely not be viable because we could get all sorts of nasty failures and corner cases if people put invalid/unsupported wheels on search_dirs.

Paul Moore Collaborator
pfmoore added a note

Installing from wheel filenames on Windows fails. Probably because filenames have an initial drive C: and the path separator is backslash. I modified pip.wheel.WheelFile.__init__ to print out the filename and I get:

>pip install C:\Work\Projects\virtualenv-pfm\virtualenv_support\pip-1.5.dev1-py2.py3-none-any.whl
Traceback (most recent call last):
  File "C:\Work\Projects\virtualenv-pfm\foo\lib\site-packages\pip\", line 121, in main
    status =, args)
  File "C:\Work\Projects\virtualenv-pfm\foo\lib\site-packages\pip\commands\", line 233, in run
    InstallRequirement.from_line(name, None))
  File "C:\Work\Projects\virtualenv-pfm\foo\lib\site-packages\pip\", line 138, in from_line
    wheel = Wheel(link.filename) # can raise InvalidWheelFilename
  File "C:\Work\Projects\virtualenv-pfm\foo\lib\site-packages\pip\", line 400, in __init__
    self.version ='ver').replace('_', '-')
AttributeError: 'NoneType' object has no attribute 'replace'

Storing debug log for failure in C:\Users\Gustav\pip\pip.log

Note that the filename is a URL-escaped mess, and there's a dash in the directory, which confuses the wheel filename parsing regexp.

Raised as a pip bug, but I don't have the time or energy to fix it myself. If nothing else, this PR is identifying a lot of things to fix in the wheel support!!!

Marcus Smith Owner
qwcode added a note

I'll get on that bug, but as for your original approach of seeing the search paths as --find-links, maybe I'm the outlier, and others would agree with that. my concern is that it prevents someone from using an older version

Paul Moore Collaborator
pfmoore added a note

Yeah, this is turning into a discussion just between the two of us :-) I'll ask for other opinions on virtualenv-list and pypa-dev, to see if that helps. I see your point regarding the change in behaviour, though. And actually the change was pretty trivial - either approach is possible.

The idea of installing an older version is probably not going to be relevant until pip 1.6 comes out though, as with this change virtualenv won't even work with versions of pip before 1.5. The situation might be different with setuptools, I don't follow setuptools versions closely. But that doesn't mean we can arbitrarily change behaviour without checking with people.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Paul Moore

I just checked and indeed pip/setuptools on PYTHONPATH cannot install sdists. It's because setuptools cannot find its own entry points when run from a non-site installation. This is a limitation of setuptools (I was going to say a "fundamental" limitation, but I don't know the design of setuptools well enough to say if it could be fixed - I've raised a setuptools bug

Paul Moore

Just updated this and rebased onto the 1.11rc1 codebase. I have also added documentation. I have not changed the behaviour of --extra-search-dirs compared to previous versions of the patch. I would prefer to get this patch merged so that we get a bit more end user feedback.

If the change in behaviour is problematic, I can develop a followup patch to address any issues. This gives us more opportunity to address actual user requirements in collaboration with someone who actually needs the functionality.

Paul Moore pfmoore merged commit 3729866 into from
Donald Stufft

It looks like this broke all of the pip tests.

Paul Moore

???!??!? How the heck can a non-released virtualenv change break the pip tests???

I'll look into this, but I'd assume that the pip tests are at fault here - they should use a known (or at least released) version of virtualenv.

Donald Stufft

The pip development tests test against the virtualenv develop branch.

Paul Moore

OK, I missed changing the to specify *.whl instead of *.egg *.tar.gz. Fix incoming. Sorry about that.

Donald Stufft

(This is actually why we test against virtualenv develop, makes a good smoke test for virtualenv :D)

Paul Moore

:-) Fair comment.

I've just pushed the change as well. To be honest, I think we're probably being too picky in I'd rather we just take whatever's in virtualenv_support rather than picking out specific files. But I don't want to start fiddling at the moment, maybe look at that later.

Donald Stufft

I retriggered the failed pip build to ensure this is working now will see what the results are.

Paul Moore

Ta. I'm watching it through now. Sorry for the issues with this.

Donald Stufft

No worries!

Paul Moore pfmoore deleted the branch
Marcus Smith

yea, the virtualenv tests are seriously lacking, having pip/develop use the virtualenv/develop branch for all it's scripttest functional testing is pretty valuable.

I switched to this when we made the awkward transition to the new merged setuptools.

Marcus Smith

@pfmoore @dstufft just stating the obvious so this doesn't get missed, this (and the other related merges that came after) needs to be pushed over to branch 1.11.X to get into the next RC.

Marcus Smith qwcode commented on the diff
((32 lines not shown))
911 936 if search_dirs is None:
912 937 search_dirs = file_search_dirs()
913   - found, sdist_path = _find_file(sdist, search_dirs)
Marcus Smith Owner
qwcode added a note

btw, the _find_file function is still left in the code, even though it's orphaned now.

Paul Moore Collaborator
pfmoore added a note

Yeah, I spotted that after I'd pushed :-( I'll do a tidy-up patch once 1.11 is released, don't want to confuse the release process over non-essentials at this point.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Paul Moore

Re the 1.11 branch, I'm not sure of the process - I'm working on the basis that @dstufft is in effect the release manager so he'll merge the change when he's ready (also I trust his git skills more than mine!) But I can merge it in if you want.

Paul Moore

@dstufft, @qwcode can either one of you tell me how I merge this onto the 1.11 release branch, to make sure it doesn't get missed?

Marcus Smith

since I think nothing has been merged to develop, that we don't want in 1.11, you can simply git merge develop from a 1.11.X checkout , and then push to 1.11.X. make sure to not overwrite the 1.11rc1 version.

Paul Moore

OK, I've updated 1.11.X. Can someone check I didn't mess up? Thanks.

Marcus Smith


Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
This page is out of date. Refresh to see the latest.
14 docs/news.rst
Source Rendered
@@ -14,6 +14,20 @@ Changes & News
14 14 ``$ENV/bin/python`` and re-running virtualenv on the same target directory
15 15 with the upgraded Python.
16 16
  17 +1.11 (2013-11-??)
  18 +~~~~~~~~~~~~~~~~~
  19 +
  20 +* **BACKWARDS INCOMPATIBLE** Switched to using wheels for the bundled copies of
  21 + setuptools and pip. Using sdists is no longer supported - users supplying
  22 + their own versions of pip/setuptools will need to provide wheels.
  23 +* **BACKWARDS INCOMPATIBLE** Modified the handling of ``--extra-search-dirs``.
  24 + This option now works like pip's ``--find-links`` option, in that it adds
  25 + extra directories to search for compatible wheels for pip and setuptools.
  26 + The actual wheel selected is chosen based on version and compatibility, using
  27 + the same algorithm as ``pip install setuptools``.
  28 +* Upgraded pip to v1.5
  29 +* Upgraded setuptools to v1.4
  30 +
17 31 1.10.1 (2013-08-07)
18 32 ~~~~~~~~~~~~~~~~~~~
19 33
9 docs/virtualenv.rst
Source Rendered
@@ -452,12 +452,13 @@ virtualenv like this::
452 452 $ virtualenv --extra-search-dir=/path/to/distributions ENV
453 453
454 454 The ``/path/to/distributions`` path should point to a directory that contains
455   -setuptools and/or pip `.tar.gz` source distributions.
  455 +setuptools and/or pip wheels.
456 456
457   -virtualenv will use the *first* distribution it finds, which won't
458   -necessarily be the latest version.
  457 +virtualenv will look for wheels in the specified directories, but will use
  458 +pip's standard algorithm for selecting the wheel to install, which looks for
  459 +the latest compatible wheel.
459 460
460   -After the extra directories, the default search order is like so:
  461 +As well as the extra directories, the search order includes:
461 462
462 463 #. The ``virtualenv_support`` directory relative to
463 464 #. The directory where is located.
@@ -906,34 +906,59 @@ def filter_install_output(line):
906 906 return Logger.INFO
907 907 return Logger.DEBUG
908 908
909   -def install_sdist(project_name, sdist, py_executable, search_dirs=None):
  909 +def find_wheels(projects, search_dirs):
  910 + """Find wheels from which we can import PROJECTS.
910 911
  912 + Scan through SEARCH_DIRS for a wheel for each PROJECT in turn. Return
  913 + a list of the first wheel found for each PROJECT
  914 + """
  915 +
  916 + wheels = []
  917 +
  918 + # Look through SEARCH_DIRS for the first suitable wheel. Don't bother
  919 + # about version checking here, as this is simply to get something we can
  920 + # then use to install the correct version.
  921 + for project in projects:
  922 + for dirname in search_dirs:
  923 + # This relies on only having "universal" wheels available.
  924 + # The pattern could be tightened to require -py2.py3-none-any.whl.
  925 + files = glob.glob(os.path.join(dirname, project + '-*.whl'))
  926 + if files:
  927 + wheels.append(os.path.abspath(files[0]))
  928 + break
  929 + else:
  930 + # We're out of luck, so quit with a suitable error
  931 + logger.fatal('Cannot find a wheel for %s' % (project,))
  932 +
  933 + return wheels
  934 +
  935 +def install_wheel(project_names, py_executable, search_dirs=None):
911 936 if search_dirs is None:
912 937 search_dirs = file_search_dirs()
913   - found, sdist_path = _find_file(sdist, search_dirs)
914   - if not found:
915   - logger.fatal("Cannot find sdist %s" % (sdist,))
916   - sys.exit(100)
917 938
918   - tmpdir = tempfile.mkdtemp()
  939 + wheels = find_wheels(['setuptools', 'pip'], search_dirs)
  940 + pythonpath = os.pathsep.join(wheels)
  941 + findlinks = ' '.join(search_dirs)
  942 +
  943 + cmd = [
  944 + py_executable, '-c',
  945 + 'import sys, pip; pip.main(["install"] + sys.argv[1:])',
  946 + ] + project_names
  947 + logger.start_progress('Installing %s...' % (', '.join(project_names)))
  948 + logger.indent += 2
919 949 try:
920   - tar =
921   - tar.extractall(tmpdir)
922   - tar.close()
923   - srcdir = os.path.join(tmpdir, os.listdir(tmpdir)[0])
924   - cmd = [py_executable, '', 'install',
925   - '--single-version-externally-managed',
926   - '--record', 'record']
927   - logger.start_progress('Installing %s...' % project_name)
928   - logger.indent += 2
929   - try:
930   - call_subprocess(cmd, show_stdout=False, cwd=srcdir,
931   - filter_stdout=filter_install_output)
932   - finally:
933   - logger.indent -= 2
934   - logger.end_progress()
  950 + call_subprocess(cmd, show_stdout=False,
  951 + extra_env = {
  952 + 'PYTHONPATH': pythonpath,
  953 + 'PIP_FIND_LINKS': findlinks,
  954 + 'PIP_USE_WHEEL': '1',
  955 + 'PIP_PRE': '1',
  956 + 'PIP_NO_INDEX': '1'
  957 + }
  958 + )
935 959 finally:
936   - shutil.rmtree(tmpdir)
  960 + logger.indent -= 2
  961 + logger.end_progress()
937 962
938 963 def create_environment(home_dir, site_packages=False, clear=False,
939 964 unzip_setuptools=False,
@@ -957,9 +982,10 @@ def create_environment(home_dir, site_packages=False, clear=False,
957 982 install_distutils(home_dir)
958 983
959 984 if not no_setuptools:
960   - install_sdist('Setuptools', 'setuptools-*.tar.gz', py_executable, search_dirs)
  985 + to_install = ['setuptools']
961 986 if not no_pip:
962   - install_sdist('Pip', 'pip-*.tar.gz', py_executable, search_dirs)
  987 + to_install.append('pip')
  988 + install_wheel(to_install, py_executable, search_dirs)
963 989
964 990 install_activate(home_dir, bin_dir, prompt)
965 991
BIN  virtualenv_support/pip-1.4.1.tar.gz
Binary file not shown
BIN  virtualenv_support/pip-1.5.dev1-py2.py3-none-any.whl
Binary file not shown
BIN  virtualenv_support/setuptools-1.4-py2.py3-none-any.whl
Binary file not shown
BIN  virtualenv_support/setuptools-1.4.tar.gz
Binary file not shown

Tip: You can add notes to lines in a file. Hover to the left of a line to make a note

Something went wrong with that request. Please try again.