Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PEP 394: Allow the `python` command to not be installed, and other minor edits #630

Merged
merged 4 commits into from Apr 28, 2018

Conversation

@encukou
Copy link
Member

encukou commented Apr 25, 2018

Please do not merge yet. I'll send this for discussion on python-dev, but I think a concrete pull request makes the review easier.


In Fedora, I found that PEP 394's strict recommendation that python points to python2 is holding us back. I would like to officially relax this in several cases.

The problems are:

  • For developers that are not following the language's development, the fact that python invokes python2 sends a strong signal that 2 is somehow the preferred version, and it's OK to start new projects in it.
  • Users and sysadmins that do want to “live in the future” are switching the symlink to python3 themselves. We would like to give them a supported, documented way to do so -- and make surer they're aware of the caveats.
  • The python2 command is still not available out-of-the box on macOS, so it didn't completely live up to the expectation of being the cross-platform way to launch python2 scripts.
  • python in the shebang line can mean either that a script is carefully written to be 2/3 compatible, or that the author/maintainer is lazy or unaware of the recommendations. While Fedora guidelines have long banned the unversioned command, we feel that the only way to enforce that guidelines is to provide environments where the python command does not work (unless explicitly installed).

To help solve these, I would like to relax recommendations on the Unix python -> python2 symlink in these cases:

  • Users and administrators can, by a deliberate action, change python to invoke Python 3. (Activating a venv counts as such an action, but so would e.g. using alternates, installing a non-default overriding package, or replacing /usr/bin/python.)
  • In controlled environments where being explicit is valued more than user experience (test environments, build systems, etc.), distributions can omit the python command even when python2 is installed.

This PR also spells out several other things, which I felt were hidden between the lines -- but correct me if you disagree with my reading.

Relax recommendations on the Unix ``python`` command (which
should invoke Python 2) in these cases:

- Users and administrators can, by a deliberate action, change
  ``python`` to invoke Python 3. (Activating a venv counts as
  such an action.)
- Distributions can omit the ``python`` command even when
  ``python2`` is installed in test environments, build systems,
  and other controlled environments where being explicit is
  valued more than user experience.
@gvanrossum

This comment has been minimized.

Copy link
Member

gvanrossum commented Apr 25, 2018

The python command is still not available out-of-the box on macOS, so it didn't completely live up to the expectation of being the cross-platform way to launch 2/3 source compatile scripts.

Did you mean python2 there? In my experience macOS comes with python installed (and invoking Python 2) but no python2 link (hard or soft). In any case I'm not sure how this strengthens your argument.

I'm also still unhappy with any kind of endorsement of python pointing to python3. When a user gets bitten by this they should receive an apology from whoever changed that link, not a haughty "the PEP endorses this".

Regardless of what macOS does I think I would be happier in a future where python doesn't exist and one always has to specify python2 or python3. Quite possibly there will be an age where Python 2, 3 and 4 all overlap, and EIBTI.

@mcepl

This comment has been minimized.

Copy link

mcepl commented Apr 25, 2018

I'm also still unhappy with any kind of endorsement of python pointing to python3. When a user gets bitten by this they should receive an apology from whoever changed that link, not a haughty "the PEP endorses this".

And are you more happy with the current state where /usr/bin/python pointing to python2 effectively endorses that? I am afraid we are doomed to endorse something whatever we do,.

Regardless of what macOS does I think I would be happier in a future where python doesn't exist and one always has to specify python2 or python3. Quite possibly there will be an age where Python 2, 3 and 4 all overlap, and EIBTI.

Given python2 dies in less than two years, I am afraid that future of yourse will be really really short ;).

(still not yet speaking as the SUSE Python maintainer, but I expect to get there on May 2nd ;))

@gvanrossum

This comment has been minimized.

Copy link
Member

gvanrossum commented Apr 25, 2018

The endorsement of "python" -> "python2" exists regardless of what you put in a PEP.

You are hopelessly naive expecting that use of Python 2 will die at the time support is stopped. In fact IIRC Red Hat was at some point expecting to make money supporting it past that deadline (not sure if that's still their expectation).

@encukou

This comment has been minimized.

Copy link
Member Author

encukou commented Apr 25, 2018

@encukou

This comment has been minimized.

Copy link
Member Author

encukou commented Apr 25, 2018

You are hopelessly naive expecting that use of Python 2 will die at the time support is stopped. In fact IIRC Red Hat was at some point expecting to make money supporting it past that deadline (not sure if that's still their expectation).

No, I'm not expecting that.
However, I am expecting that to actually make Python 2 go away, (so that Red Hat doesn't have to support it anymore, and I can spend my time on better things), I need to actively discourage its use.
As long as things keep working smoothly, some people won't switch to Python 3 – and at some point things need to stop working smoothly. I want to avoid that being a world-breaking event. I want to move to a state where only things that actually need Python 2 use Python 2, so we can identify them and help port them.

@warsaw

This comment has been minimized.

Copy link
Member

warsaw commented Apr 25, 2018

FWIW Homebrew does include a python2 -> python symlink, so e.g. if you're installing 2.7.14 from brew, /usr/local/bin/python2 works as expected.

@encukou

This comment has been minimized.

Copy link
Member Author

encukou commented Apr 25, 2018

Yes, Homebrew found out the hard way that PEP 394 is actually relatively well thought-out and should be followed.

@warsaw

This comment has been minimized.

Copy link
Member

warsaw commented Apr 25, 2018

Forget about whatever Apple does - that shouldn't influence our decision here. Apple's out of the box distribution of open source tools isn't a model of modernity or utility. It's only whatever works for them. Homebrew (what I use) and maybe Fink or MacPorts (distributions I no longer use) are probably more receptive to whatever we decide.

I know for a fact that Debian/Ubuntu won't do anything about /usr/bin/python until at least after 2.7 is EOL'd. I'm also not sure that any change to the PEP will sway them even after that. It's even been difficult to get consensus on adding a python2 symlink and shebanging all CLIs with that.

Having said all that, I think it does make sense to allow distributions to make PEP-aligned policy as they see fit. They know their communities, policies, implications, etc. better than "we" do. OTOH, I'm also happy waiting until 2.7 EOL to make any changes to this PEP's policy. (Maybe we can say, here's where the PEP stands post-2020).

@gvanrossum

This comment has been minimized.

Copy link
Member

gvanrossum commented Apr 25, 2018

To be clear, what I would endorse is a world where python simply doesn't exist for most users. That seems compatible with what Fedora is striving for. I don't have any contacts inside Apple any more, so we'll have to exclude macOS.

I haven't memorized PEP 394 -- if it currently stipulates that python must exist we should definitely drop that requirement. But we shouldn't replace it with any kind of endorsement of people making python point to python3.

(I do think venv is wrong too, but it's too late, and perhaps it's the compromise we're all looking for.)

@encukou

This comment has been minimized.

Copy link
Member Author

encukou commented Apr 25, 2018

@encukou

This comment has been minimized.

Copy link
Member Author

encukou commented Apr 25, 2018

Also, the PEP currently explicitly says:

It is anticipated that there will eventually come a time where the third party ecosystem surrounding Python 3 is sufficiently mature for this recommendation to be updated to suggest that the python symlink refer to python3 rather than python2.

@mcepl

This comment has been minimized.

Copy link

mcepl commented Apr 25, 2018

You are hopelessly naive expecting that use of Python 2 will die at the time support is stopped. In fact IIRC Red Hat was at some point expecting to make money supporting it past that deadline (not sure if that's still their expectation).

I will let others argue over this, they seem to be smarter than me, but just let me note here that so far I am an employee of Red Hat, so I know something about the support of EOL'ed programs.

similar tool) is active, the ``python`` command should refer to the
virtual environment's interpreter. In other words, activating a virtual
environment counts as deliberate user action to change the default
``python`` interpreter.

This comment has been minimized.

Copy link
@pitrou

pitrou Apr 25, 2018

Member

+1 on this bullet point. Having python point to Python 3 when a Python 3 environment is activated is now established practice, and it would be nice for the PEP to acknowledge that. Many subparts of the community have already shifted away from the old idiom of relying on the system Python for user scripts.

This comment has been minimized.

Copy link
@gvanrossum

gvanrossum Apr 27, 2018

Member

Yeah, I won't ask venv to change, I just rm <env>/bin/python after creating the env. :-) I wonder if it should say "may refer" rather than "should refer"?

I do note that this makes using #!/usr/bin/env python quite unreliable to invoke Python 2.

This comment has been minimized.

Copy link
@pitrou

pitrou Apr 27, 2018

Member

I'd rather keep "should". It's quite an engrained expectation by now to have python point to the environment's Python. Doing otherwise would break those expectations.

This comment has been minimized.

Copy link
@ncoghlan

ncoghlan Apr 28, 2018

Contributor

Aye, this is definitely a should - we rely on it heavily in the Python Packaging User Guide.

This comment has been minimized.

Copy link
@gvanrossum
Copy link
Member

gvanrossum left a comment

Let me know if this doesn't answer all your questions from the main thread.

refers to the same target as ``python2``.
* for the time being, all distributions *should* ensure that ``python``,
if installed, refers to the same target as ``python2``, unless the system
administrator or user deliberately override this.

This comment has been minimized.

Copy link
@gvanrossum

gvanrossum Apr 25, 2018

Member

So this should probably change to ensuring that python, if it exists, runs Python 2 unless (a) the user explicitly overrides it, or (b) a venv is active.

Python as the ``python2`` command (however, note that some distributions
have already chosen to have ``python`` implement the ``python3``
command; see the `Rationale`_ and `Migration Notes`_ below).
command, and system administrators may be allowed to change the default;

This comment has been minimized.

Copy link
@gvanrossum

gvanrossum Apr 25, 2018

Member

I don't want sysadmins to feel empowered by this PEP, unless they are the only user of a system. Sysadmins often don't know what their users are doing. Changing what python does should be up to the user. The sysadmin should not have any dependencies on python -- sysadmin scripts should use python2 or python3.

command, and system administrators may be allowed to change the default;
see further recommendations and the `Rationale`_ and `Migration Notes`_
below).
* The ``python`` command should be available whenever ``python2`` is available,

This comment has been minimized.

Copy link
@gvanrossum

gvanrossum Apr 25, 2018

Member

I guess this should go.

* When packaging software that is source compatible with both versions,
distributions may change such ``python`` shebangs to ``python3`` (or
``python2``). This ensures software is used with the latest version of
Python available, and it can remove a dependency on Python 2.

This comment has been minimized.

Copy link
@gvanrossum

gvanrossum Apr 25, 2018

Member

(Then why the "or python2" ?)

(All software in such a controlled environment must use ``python3`` or
``python2`` rather than ``python``, which means scripts that deliberately
use ``python`` need to be modified for such environments.)
* System administrators and users may be empowered to change the meaning of

This comment has been minimized.

Copy link
@gvanrossum

gvanrossum Apr 25, 2018

Member

No on this bullet.

in March and July 2011 ([1]_, [2]_), February 2012 ([4]_) and
September 2014 ([6]_).
in March and July 2011 ([1]_, [2]_), February 2012 ([4]_),
September 2014 ([6]_), and April 2018 (XXX).

This comment has been minimized.

Copy link
@gvanrossum

gvanrossum Apr 25, 2018

Member

Likely this discussion isn't going to be on python-dev but just in this PR.

refer to ``python3`` rather than ``python2``.
party ecosystem surrounding Python 2 becomes irrelevant in most use cases,
at which point this recommendation should be updated to suggest that the
``python`` symlink refer to ``python3`` rather than ``python2``.

This comment has been minimized.

Copy link
@gvanrossum

gvanrossum Apr 25, 2018

Member

Honestly I would rather drop this paragraph -- it spreads false hope.

@@ -150,15 +175,15 @@ making such a change.
* When the ``pythonX.X`` binaries are provided by a distribution, the
``python2`` and ``python3`` commands should refer to one of those files
rather than being provided as a separate binary file.
* It is suggested that even distribution-specific packages follow the
``python2``/``python3`` convention, even in code that is not intended to
* It is suggested that distribution-specific packages use ``python2`` or

This comment has been minimized.

Copy link
@gvanrossum

gvanrossum Apr 25, 2018

Member

I'd say "strongly encouraged" or something like that.

convention by changing the ``python`` interpreter on a test box and checking
to see if anything breaks.
convention by changing or removing the ``python`` command on a test box
and checking to see if anything breaks.

This comment has been minimized.

Copy link
@gvanrossum

gvanrossum Apr 25, 2018

Member

Perhaps this fairly lame testing recommendation should just be deleted.

has ``python`` in its shebang line, is invoked on a system that does not have
Python 2 (and thus, the ``python`` command) installed.
This is mostly a non-issue -- as with any other case of a required interpreter
not being installed, the sysadmin can install the ``python`` command.

This comment has been minimized.

Copy link
@gvanrossum

gvanrossum Apr 25, 2018

Member

Again, I don't want sysadmins to feel empowered by this. Assuming the shebang line uses #!/bin/env python this can be remedied by activating a venv.

@ned-deily

This comment has been minimized.

Copy link
Member

ned-deily commented Apr 26, 2018

FWIW Homebrew does include a python2 -> python symlink, so e.g. if you're installing 2.7.14 from brew, /usr/local/bin/python2 works as expected.

As is the case for the Python 2.7.x's provided by python.org macOS installers. And the MacPorts Python 2.7.x.

Unfortunately, Apple support of Python in macOS releases has languished (for example, no Python 3, not up-to-date 2.7, no pip, no python2 link, seriously buggy outdated Tk, deficient SSL support) to the point that it is generally a disservice to users to recommend using it. So I don't think the Apple-supplied system Python should be a factor in discussions here, other than the fact that it does provide /usr/bin/python and /usr/bin/python2.7 links that most-third party macOS Python distributors shadow by $PATH manipulation (and will continue to need to do so as least as long as Apple keeps supplying them in /usr/bin).

Copy link
Member

gvanrossum left a comment

I'm personally fine with this weakened version. I wonder if we should post a summary of the changes to python-dev to put people on notice that we're changing the PEP. (IMO the most serious change is allowing python to not exist at all, and the second-most serious change is the removal of the paragraph about the anticipated future where python points to Python 3 -- the rest I see as minor edits.)

similar tool) is active, the ``python`` command should refer to the
virtual environment's interpreter. In other words, activating a virtual
environment counts as deliberate user action to change the default
``python`` interpreter.

This comment has been minimized.

Copy link
@gvanrossum

gvanrossum Apr 27, 2018

Member

Yeah, I won't ask venv to change, I just rm <env>/bin/python after creating the env. :-) I wonder if it should say "may refer" rather than "should refer"?

I do note that this makes using #!/usr/bin/env python quite unreliable to invoke Python 2.

@@ -267,6 +276,9 @@ References
.. [6] PEP 394 - Clarification of what "python" command should invoke
(https://mail.python.org/pipermail/python-dev/2014-September/136374.html)

.. [7] PEP 394: Allow changing the `python` command in some cases

This comment has been minimized.

Copy link
@gvanrossum

gvanrossum Apr 27, 2018

Member

I suggest changing the PR title to match the contents.

@holdenweb

This comment has been minimized.

Copy link
Member

holdenweb commented Apr 27, 2018

@encukou encukou changed the title [WIP] PEP 394: Allow changing the `python` command in some cases [WIP] PEP 394: Allow the `python` command to not be installed, and other minor edits Apr 27, 2018
@encukou

This comment has been minimized.

Copy link
Member Author

encukou commented Apr 27, 2018

Maybe the python command should just print "Did you mean python2 or python3?" to stderr and fail?

That tends to confuse tools like buildsystems which detect presence of the python command. I'm not saying they should be doing that, just that a failing command is quite different from the command not being available.

@holdenweb

This comment has been minimized.

Copy link
Member

holdenweb commented Apr 27, 2018

@gvanrossum

This comment has been minimized.

Copy link
Member

gvanrossum commented Apr 27, 2018

@encukou Let me know when you feel this is ready to commit. Please remind me to change commit topic line to match what you put in the PR. :-)

@ChrisBarker-NOAA

This comment has been minimized.

Copy link

ChrisBarker-NOAA commented Apr 27, 2018

This PEP was written a while back, so good to be revisiting.

But I also think we need to acknowledge common practice -- I doubt we will ever get to the day when there is no "python" command, and I have no idea what the numbers are, but there are certainly folks out there that are symlinking "python" to "python3" right now -- it's part of the "python 3 IS python now" approach.

So maybe the paragraph about the anticipated future where python points to Python 3 shouldn't be removed, but rather edited -- I suspect that will happen in some environments regardless of what this PEP says.

@gvanrossum

This comment has been minimized.

Copy link
Member

gvanrossum commented Apr 27, 2018

No, I really want that paragraph deleted. My point is that the PEP should not encourage hopes in that direction or endorse actions by rogue sysadmins (or distros :-), other than the established practice in virtual environments.

I want people who write 3rd party documentation to be quite aware of the repercussions when they suggest their readers type python at their shell: depending on a number of factors it may fail or invoke Python 2 or Python 3.

And I want sysadmins (and even users and distros) who "helpfully" link python to python3 be aware that this breaks Python 2 scripts that use #!/usr/bin/env python -- it doesn't just affect what the user gets when they type python at their shell.

Copy link
Contributor

ncoghlan left a comment

Reviewing the detailed changes, this update looks good to me.

@ncoghlan

This comment has been minimized.

Copy link
Contributor

ncoghlan commented Apr 28, 2018

@warsaw @kerrickstaley This version looks good to me, and I'm the only one that's commented on @encukou's python-dev thread so far, so unless one of you raises any objections, I'm going to suggest that Petr add himself as a co-author, and then we merge the update.

@gvanrossum

This comment has been minimized.

Copy link
Member

gvanrossum commented Apr 28, 2018

@ncoghlan

This comment has been minimized.

Copy link
Contributor

ncoghlan commented Apr 28, 2018

Right, the author credit here wouldn't just be about these particular updates - it's also about Petr being one of the primary drivers of how Fedora is handling the migration, so he is one of the folks having to stand by the recommendations it provides.

(I agree it can be a separate question, it just occurred to me to ask it while writing my earlier comment)

@encukou encukou changed the title [WIP] PEP 394: Allow the `python` command to not be installed, and other minor edits PEP 394: Allow the `python` command to not be installed, and other minor edits Apr 28, 2018
@warsaw

This comment has been minimized.

Copy link
Member

warsaw commented Apr 28, 2018

@ncoghlan Sorry, just been too busy to dive deeply into this, but I have lured on thread and this PR. JFDI! (And I'm find with adding @encukou as an author.)

The one "future" point I wholeheartedly agree with is that if we ever do a Python 4, I do not want to add /usr/bin/python4. I think by that time we probably can just reclaim /usr/bin/python because Python 2 will be long buried and there won't be anything backward incompatible about it (i.e. it'll just be a "normal" upgrade with the added bonus of a major version bump). That's of course if we don't move to a calver style versioning scheme <wink>.

@encukou

This comment has been minimized.

Copy link
Member Author

encukou commented Apr 28, 2018

I don't feel that great about taking new long-term responsibilities, but discussions on this PEP are probably something I'll get involved in anyway. Since the other authors don't object, let's make it explicit.

@gvanrossum

This comment has been minimized.

Copy link
Member

gvanrossum commented Apr 28, 2018

@encukou encukou merged commit cd59ec0 into python:master Apr 28, 2018
1 check passed
1 check passed
continuous-integration/travis-ci/pr The Travis CI build passed
Details
@encukou encukou deleted the encukou:pep394-relax branch Apr 28, 2018
@kerrickstaley

This comment has been minimized.

Copy link

kerrickstaley commented Apr 29, 2018

@ncoghlan

This comment has been minimized.

Copy link
Contributor

ncoghlan commented Apr 29, 2018

Thanks for working through this @encukou, and @gvanrossum for the initial review (I get the impression the original PR looked quite different from the final version I reviewed!)

@FRidh FRidh mentioned this pull request May 12, 2018
2 of 3 tasks complete
encukou added a commit to encukou/peps that referenced this pull request Feb 13, 2019
The intended future for the ``python`` command is:

> ``python`` doesn't exist and one always has to specify ``python2``
> or ``python3``.

( – Guido, python#630 (comment) )

There are three conflicting ideas around the ``python`` command,
longer-term:

1. The ``python`` command should continue to refer to Python 2 (if
   Python 2 is available).
2. ``python`` is a correct shebang for py2/py3 source-compatible scripts.
3. Python 2 should *not* be available (by default / after 2010).

One of these has to give.
It seems that (2) is the easiest to shed, so this proposal does just that.

* Make ``python3`` the preferred shebang for py2/py3 compatible
  scripts, as that's the only shebang that'll work on py2-less systems.
  (An exception is made for scripts targetting the *system* Python
  on e.g. macOS/RHEL.)
* Make the unversioned ``python`` command optional (along with
  ``idle``, ``pydoc``, and ``python-config``).
  Distributions now do not need to install it by default, even if
  Python 2 is installed. (But they should make it installable
  explicitly, as long as they ship Python 2.)
  This should introduce more people to systems without the
  "legacy" ``python`` command, encourage the use of explicit
  ``python2``/``python3``, and ease switching to systems that don't
  have Python 2 at all.
* Clarify that distributions *should not* make unversioned
  ``python`` configurable. (That might work for carefully managed
  systems, but the ecosystem should converge on ``python3``, not
  "your ``python`` needs to be set properly".)
* Remove mentions of choosing to link ``python`` to ``python3`` in the
  future, as we don't expect to start recommending that.

* Use the term **"unversioned"** ``python`` when contrasting it with
  ``python3``/``python2``.
  I found that this makes the message much clearer.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
You can’t perform that action at this time.