PEP 453: "make install" and "make altinstall" integration

[ncoghlan]

BPO	19553
Nosy	@loewis, @ncoghlan, @larryhastings, @ned-deily, @dstufft
Dependencies	bpo-19406: PEP 453: add the ensurepip module
Files	issue19553.patch: patch for source build integration - run "autoreconf" first

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/ned-deily'
closed_at = <Date 2013-11-22.14:41:13.520>
created_at = <Date 2013-11-11.11:39:39.607>
labels = ['type-feature', 'build']
title = 'PEP 453: "make install" and "make altinstall" integration'
updated_at = <Date 2013-11-22.14:41:13.519>
user = 'https://github.com/ncoghlan'

bugs.python.org fields:

activity = <Date 2013-11-22.14:41:13.519>
actor = 'ncoghlan'
assignee = 'ned.deily'
closed = True
closed_date = <Date 2013-11-22.14:41:13.520>
closer = 'ncoghlan'
components = ['Build']
creation = <Date 2013-11-11.11:39:39.607>
creator = 'ncoghlan'
dependencies = ['19406']
files = ['32647']
hgrepos = []
issue_num = 19553
keywords = ['patch']
message_count = 13.0
messages = ['202616', '202674', '203013', '203027', '203028', '203054', '203055', '203060', '203069', '203725', '203729', '203746', '203773']
nosy_count = 6.0
nosy_names = ['loewis', 'ncoghlan', 'larry', 'ned.deily', 'python-dev', 'dstufft']
pr_nums = []
priority = None
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue19553'
versions = ['Python 3.4']

[ncoghlan]

Part of the PEP-453 implementation as tracked in bpo-19347.

This issue covers the integration of ensurepip with "make install" (running "python -m ensurepip") and "make altinstall" (running "python -m ensurepip --altinstall")

[ned-deily]

If nobody else gets to it first, I'll do this in the next couple of days.

[ned-deily]

Here's a patch for review. It implements:

• a new configure option:

  --with(out)-ensurepip=[=upgrade]
  "install" or "upgrade" using bundled pip

• a new makefile macro ENSUREPIP that can be supplied to a "make install" or "make altinstall" target to override the configure option:

  make [alt]install ENSUREPIP=[install|upgrade|no]

While testing the above I've identified a number of issues. I haven't had time to do more than a quick triage on most of them so the initial analyses may be wrong.

Wheel installation issues:

1. Files, but not directories, from the pip/setuptools wheel were installed with a different group id than when installed with pip from a source dist. (This was comparing the current checked-in wheel file versus a normal sdist pip install of pip, on OS X.) It appears that the files installed from wheels ended up with the effective gid of the installing process, while those from sdist files had the group id of the parent directory like other files installed by make [alt]install (POSIX allows either behavior but pip should be consistent).

2. It appears that .py files installed from wheels do not get compiled upon installation (no __pycache__ or .pyc) whereas non-wheel pip installed .py files do get compiled. (See also 7.)

3. The unversioned pip file name differs between wheel and non-wheel installs: pip3.4 vs pip-3.4.

4. No unversioned version of easy_install is created for wheel installs, only easy_install-3.4.

General pip issues:

5. Various ResourceWarnings about unclosed files and sockets were observed when used with a debug python.

6. Perhaps more a Distutils issue but the files and directories created under lib/python3.4/site-packages use the process default umask and thus may have permissions different from the files installed for the standard library. The latter get standard "install" permissions because the main Python setup.py subclasses and overrides the default Distutils install_lib class to accomplish this. Since the Makefile does not know exactly what files have been installed by ensurepip, the install and altinstall targets could only guess at what files need to be fixed up.

7. Other standard library .py files installed by "make [alt]install" are compiled to both .pyc and .pyo versions.

ensurepip issue:

8. "make install" is a superset of "make altinstall" and one would expect the results of (a) "make install" to be the same as (b) "make altinstall && make install". However (b) results in "python -m ensurepip --altinstall --upgrade && python -m ensurepip --upgrade" which results in no unversioned pip files being installed as the second call to pip does nothing:

Requirement already up-to-date: setuptools in /py/dev/3x/root/uxd/lib/python3.4/site-packages
Requirement already up-to-date: pip in /py/dev/3x/root/uxd/lib/python3.4/site-packages

Perhaps ensurepip module should also set "--ignore-installed" when calling pip?

Makefile:
9. I still need to create [alt]installunixtools links for pip* in Mac/Makefile.in.

[ncoghlan]

Thanks for the patch and the detailed feedback, Ned!

I've pinged the pip folks on pypa/pip#1322 to help triage these issues as existing pip upstream issues.

[ncoghlan]

For 8 in particular, I'm inclined to push that over to the pip side of the fence (as with the first 7 points). pip knows that ensurepip is involved (due to ENSUREPIP_OPTIONS being set in the environment), so it may be able to check if the unversioned scripts are missing and generate them appropriately in that case.

How do we want to proceed for the first beta? Given the integration timeline in the PEP (which doesn't require the shipped version of pip to be locked until just before beta 2), I'm inclined not to consider the additional quirks Ned found as blockers for beta 1, but we do need to get them fixed for beta 2.

[ned-deily]

I think the most serious problem is 3 since it directly affects the end user. It would be good to have that fixed someway for 3.4.0b1. Resolutions to the other items could wait.

[dstufft]

1. This is bound to be an issue that stems from the fact pip is doing the install instead of distutils. It probably makes sense to use the group id of the parent directory I think?

2. This is a side effect of Wheel being a whole new way to install, previously pip just did setup.py install which took care of this bit. It shouldn't be a problem to make this happen.

3. Which sdist were you testing? The one from PyPI? We switched the style of that in 1.5 in order to better match how Python itself handles it versioned commands.

4. You mean when you do --default-install? (Or whatever it's called if Nick changed it).

5. I don't think any of us have ever run pip under a debug build of Python so that's probably why. It should probably be fixed up.

6. I'm not sure what the right answer is here. We recently switched pip to install using the default umask because previously it was using whatever permissions where in the archive. If the make file wants to control the permissions of the files can't it simply adjust the umask before calling ensurepip?

7. See the answer to Rename README to README.rst and enhance formatting #2.

8. I'll need to think about this, we don't want to blindly add --ignore-installed because we don't want ensurepip to downgrade pip if a newer version is installed.

Things that I see as issues so far I've created upstream tickets for, things that I still have questions on I didn't.

pypa/pip#1324
pypa/pip#1325
pypa/pip#1326

[ncoghlan]

For the record: the current "--default-pip" option is the one that was previously "--default-install". I switched it because having the default installation behaviour be something other than the behaviour requested via the "--default-install" option really didn't sound right to me.

[ned-deily]

1. Yes, I think so.

2. OK

3. Ah, I was just using the default 1.4.1 from PyPI. With current dev from github, the result is now pip3.4 like the wheel install. So once 1.5.x is released, this shouldn't be a problem.

4. Sorry, I was imprecise. For "-m ensurepip --default-pip", both "easy_install" and "easy_install-3.4" are created. For just "-m ensurepip", only "easy_install-3.4". What differs from pip is that no "easy_install-3" is created in either case. But thinking about it, that's probably just fine if setuptools isn't creating it today; we're not trying to encourage its use! So, no action needed.

5. OK

6. Yes, the Makefile "install" and "altinstall" targets could set "umask 022" before calling ensurepip. That's sounds like an acceptable solution; I'll add that. There are other targets in the Makefile where permissions are currently not forced (for example, bpo-15890).

Things that I see as issues so far I've created upstream tickets for

Thanks!

[python-dev]

New changeset 90d4153728f6 by Ned Deily in branch 'default':
Issue bpo-19553: PEP-453 - "make install" and "make altinstall" now install or
http://hg.python.org/cpython/rev/90d4153728f6

[ned-deily]

The proposed patch with minor changes is now pushed. I believe all of the review points have either been resolved or are covered by separate pip issues with the exception of item 8. For item 6 (permissions not being forced), I decided that there are so many other files that are also not having permissions forced that it is silly to single out ensurepip. bpo-15890 should cover a comprehensive solution for all files. In the meantime, the solution remains to set umask appropriately (e.g. 022) before running "make install" or "make altinstall". I'm removing "release blocker" status and propose closing this issue once a decision about and/or home for item 8 is found.

[dstufft]

I'm honestly not sure what to do about #8 on your list. It's sort of a really wierd edge case as far as pip is concerned right now because the support for the versioned commands and differing them is sort of a hack job while we wait for proper support from a PEP.

Probably long term wise once there's support for this in a PEP pip will gain some sort of regenerate scripts commands that could handle this case better.

I'm struggling to come up with a good solution in the interim though :(

[ncoghlan]

I moved the "make altinstall && make install" problem out to its own issue (bpo-19693)