PEP 453: add the ensurepip module #63605
Comments
|
Adding the initial ensurepip implementation and module docs (the "Installing Python Modules" updates will be handled in a separate issue) |
|
Attached is an initial rough draft of the ensurepip module. There are some issues still, but they largely need resolved in pip.
If you prefer to view it on github my branch for working on it is available at https://github.com/dstufft/cpython/compare/ensurepip |
|
Added a second draft that handles the case when the stdlib isn't directly browseable (e.g. it's zipped up or something). |
|
Added a third draft, this one adds the script to check if pip needs updated. I've removed the first two drafts to make it simpler. |
|
Draft docs patch attached. This splits out a new "Software Packaging and Distribution" section in the main docs index, with distutils, ensurepip and venv as the specific entries. There's a couple of questions I need answered regarding the behaviour of pip in order to document --user properly:
|
|
You cannot use --user in a virtual environment (well a venv, no idea about a pyvenv - but it should get a similar error message IMO if it doesn't). If you use --root and --user together it will install to the --root location, using the user layout, so instead of installing to /home/dstufft/.local/lib/python3.4/site-packages/ it will install to /value/of/root/home/dstufft/.local/lib/python3.4/site-packages/ |
|
Added most of the nosy list from the tracking bpo-19347. Updated draft docs. I believe this version of the module docs should be complete (since the installation guide updates are covered by bpo-19407). While Donald's attached patch is incomplete, the version on GitHub looks basically ready to go (aside from --root not working when running pip from a wheel file): https://github.com/dstufft/cpython/compare/ensurepip Shall we proceed with the commit based on the currently available pip, so Martin and Ned can get started on the installer updates? |
|
For what it's worth I can get --root ready to go shortly, I have a patch against pip for it (pypa/pip#1272) I just need to write some tests to ensure it keeps working. Let me go off and do that right now. |
|
I also need to update the bundled Wheel to one that was created with Python 3.4 instead of 2.7 (which matters until the fix for pypa/pip#1067 which is pypa/pip#1251 lands). That fixes the issue where the wheel will have pip2, and pip2.7 baked into it. |
|
Ok, merged in the --root fix to pip and created a Wheel using Python 3.4 (which I installed the Wheel distribution using an ensurepip installed pip ;) ). Updated on github and a patch added, all outstanding issues that affect this patch exist on the pip side of things. Primarily pypa/pip#1067 which, since I created the new pip wheel using 3.4, only affects this patch in that setuptools is still version specific. This'll be fixed soon in pip. |
|
Donald, could you upload an updated patch here that includes both your changes and my docs updates? We'll give people a day to look it over, and then go ahead and commit it and create the tracker issues to update pyvenv/venv and the Mac OS X and Windows installers. |
|
There you go Nick. |
|
Here are my review comments (sorry, no Rietveld): ensurepip/init.py Why have the separate _run_pip function? It would be simpler and less confusing to just merge run_pip into bootstrap and eliminate the extra for loop. Both depend on the _PROJECTS list anyway. -- Ah, I see that the reason for the separation is for mocking purposes in test_ensurepip. How about constructing the list of paths to insert in bootstrap and passing that as the new second argument to _run_pip. Then the knowledge about the wheel names is only in one place. test_ensurepip.py s/bootsraping/bootsrapping/g checkpip.py It would be useful if checkpip.py returned an non-zero exit status in the case any projects are out-of-date. Makefile.pre.in Need to add ensurepip and ensurepip/_bundled to LIBSUBDIRS so that they are installed. ensurepip/_bundeled The actual wheel files are not in the patch. For other reviewers, Donald pointed me to their source here: Using those wheels, which, as Donald notes, are preliminary (especially the setuptools one), I did some installs using a non-standard --prefix for python. Here are the contents of the bin directory (what would be installed into /usr/local/bin by default) after "make altinstall". $ ls -l bin
total 18760
-rwxr-xr-x 1 nad pyd 110 Nov 1 15:08 2to3-3.4*
-rwxr-xr-x 1 nad pyd 108 Nov 1 15:08 idle3.4*
-rwxr-xr-x 1 nad pyd 93 Nov 1 15:08 pydoc3.4*
-rwxr-xr-x 2 nad pyd 4791296 Nov 1 15:08 python3.4*
-rwxr-xr-x 2 nad pyd 4791296 Nov 1 15:08 python3.4dm*
-rwxr-xr-x 1 nad pyd 2041 Nov 1 15:08 python3.4dm-config*
-rwxr-xr-x 1 nad pyd 245 Nov 1 15:08 pyvenv-3.4*Now after python3.4 --ensurepip is run: -rwxr-xr-x 1 nad pyd 110 Nov 1 15:08 2to3-3.4* Now after "make install" is run: lrwxr-x--- 1 nad pyd 8 Nov 1 15:21 2to3@ -> 2to3-3.4 The problem I see here is that pip is unconditionally installing both a pip and a pip3 script along with the versioned script (pip3.4). Setuptools also has the same issue (along with the wonky .pya scripts which I think should be Windows only). For system installs, pip, setuptools, and ensurepip should support the equivalent of "altinstall" and "install" options, with the former only installing fully-versioned names and the latter adding "3" names and, to be consistent, neither should install the totally unversioned links for Python 3, so no "pip" or "easy_install". Third-party packagers solve the conflict between Py2 and Py3 scripts in various ways, like providing some administrative command (e.g. debian "update-alternatives" or Macports "port select"). For virtual environments, the user has complete control over the bin directory and the current behavior of pip and easy_install is OK there (and would be introduce an incompatibility if the default were to change). ensurepip should be conservative about this. Perhaps the way to implement it is to introduce something like a non-default --alt (name TBD) option to both setuptools and pip (for backwards compatibility) and have ensurepip by default always install setuptools and pip with --alt unless installing into a virtual environment or with a "--makedefault" (name TBD) option? It would be rather nasty for the default ensurepip to wipe out already installed versions of pip and easy_install for Python 2 (yes, you would need to have the appropriate privilege as well) in a shared bin directory, like /usr/local/bin or /usr/bin. FTR, the OS X installer build script could work around the current behavior. |
|
I suppose the changes could be isolated to just ensurepip if it used a temporary dir for the scripts and then moved the appropriate ones itself to the standard scripts directory. |
|
Just for the record, here's what the installed ./lib/python3.4/site-packages looks like after ensurepip has run: -rw-r--r-- 1 nad pyd 119 Nov 1 15:20 README And, within pip/_vendor, the following additional projects are installed: -rw-r----- 1 nad staff 266 Nov 1 15:19 __init__.py |
|
The .pya thing is an experimental extension type that setuptools added that just got missed during the new features to generate scripts during wheel install vs wheel build time. I opened a bug to remove that and it'll be gone before 1.5 is released. I can fix the typos and add those to the Makefile.pre.in. I agree the versioned scripts need some sort of resolution, I'm not entirely sure what that resolution is. This is coming from pip itself so it's not specific to ensurepip. So there are 4 cases that pip needs to handle.
So there's obviously a number of solutions that solve the problem in specifically the ensurepip case (adding a flag, temporary directory, etc). What I wonder is: A) Is it considered reasonable that if someone installs pip with ensurepip, but then later upgrades it with |
|
Oh one thing, I can't move anything out of _run_pip because the part you're referring to is actually modifying the sys.path. If I move it then I can't prevent the tests from having side effects. |
|
Oh nevermind, I understand now. I misread the statement. I can do that. |
|
Attached is the second combined2 patch with Ned's feedback incorporated. For anyone testing this the patch does not contain the binary files which can be found at https://github.com/dstufft/cpython/blob/ensurepip/Lib/ensurepip/_bundled/. |
|
The "install everything" approach is OK for Windows and virtual Perhaps pip itself could be updated such that it installs the bare "pip" Also, if we do the temp directory workaround then we need to copy and then |
|
Without digging deeper, I'd be a little cautious about using a test involving sys.executable and shutil.which. sys.executable in the past at least was not always what you might think in certain cases with OS X framework installs. |
|
I created pypa/pip#1294 for the script versioning issue on the pip side, as I'd prefer not to be doing the temporary directory dance if we don't need to. How do we want to handle this for beta 1? Is it OK to do an initial commit that has the installation of non-versioned scripts as a known defect so we can make progress on the rest of the changes in the meantime? Alternatively, we could perhaps tweak the embedded wheels as an interim fix until the problem is addressed upstream. |
|
Tweaking the Wheels won't work. The scripts are generated at install time. We need to fix it in pip, I was waiting on answers to http://bugs.python.org/issue19406#msg201954 before coming up with a solution and making a PR request as that will influence the proposal/PR I make to pip. Basically we need to figure out what is considered reasonable during the initial ensurepip, and subsequent pip install --upgrade pip. |
|
I'm OK with an unqualified "pip install --upgrade pip" retaining pip's current behaviour of always writing the unqualified script versions. However, running "pip3 install --upgrade pip" would ideally leave the "pip" and "easy_install" scripts alone. For "make altinstall" compatibility, we definitely need a "--altinstall" flag that only installs the fully qualified versions of pip and easy_install. Finally, there needs to be some way to explicitly request the "pip3" style behaviour when running via -m so we can use it from ensurepip. |
|
Donald, I know you've been busy with PyPI v2.0 the last few days, but I see the pull request to resolve pypa/pip#1294 has been merged. If we can get an updated patch that sets ENSUREPIP_OPTIONS appropriately in the process environment, it should be possible to commit this one and let Ned and Martin get started on the installers. |
|
|
Assigning to myself for final review and commit :) |
|
I'm also going to start creating the implementation issues for the installer and pyvenv updates. |
|
The inconsistency between altinstall and default_install bothered me, so I plan to change the spelling of the latter option to "default_pip" (since it directly controls whether or not the "pip" script gets installed). |
|
New changeset 6a6b1ee306e3 by Nick Coghlan in branch 'default': |
|
Aside from the default-install -> default-pip name change, the other fixes/changes between Donald's last patch and the committed version:
|
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: