New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Links to current Python are lost after Homebrew update #15746

Closed
orome opened this Issue Jul 18, 2017 · 55 comments

Comments

Projects
None yet
@orome

orome commented Jul 18, 2017

Pior to my latest update of Homebrew (1.2.4) I had Python located, as indicated by brew --config, with a link to the latest Homebrew version:

Python: /usr/local/bin/python => /usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/bin/python2.7

and both this and the macOS system-provided Python were visible to which -a.

Following the update, I've lost access to this Python (though it still exists, and is available as python2) and just the old system-supplied Python is indicated by listed by brew --config as

Python: /usr/bin/python

and by which -a.

Is this the intended behavior of the latest Homebrew? If not, is there a fix for restoring the access I once had to the latest Homebrew Python?


Note that I also get the following results (which means I can't use brew to fix itself):

$ brew link --overwrite --dry-run python2
Warning: Already linked: /usr/local/Cellar/python/2.7.13_1
To relink: brew unlink python && brew link python
$ brew link --overwrite --dry-run python
Warning: Already linked: /usr/local/Cellar/python/2.7.13_1
To relink: brew unlink python && brew link python
$ brew link --overwrite --dry-run python3
Warning: Already linked: /usr/local/Cellar/python3/3.6.2
To relink: brew unlink python3 && brew link python3

Naively I assume I can fix this "by hand" with

redacted

But I wonder, given that Homebrew has made such a profound change to the Python configuration, whether more is not going on (that such linking will break). After all, the latest update chose to remove this link when it was present.

@kevwil

This comment has been minimized.

Contributor

kevwil commented Jul 18, 2017

Following, as my "python" and "pip" are also missing.

@DomT4

This comment has been minimized.

Contributor

DomT4 commented Jul 18, 2017

Is this the intended behavior of the latest Homebrew?

Yes. See brew info python:

==> Caveats
This formula installs a python2 executable to /usr/local/bin.
If you wish to have this formula's python executable in your PATH then add
the following to ~/.zshrc:
  export PATH="/usr/local/opt/python/libexec/bin:$PATH"

Pip and setuptools have been installed. To update them
  pip2 install --upgrade pip setuptools

You can install Python packages with
  pip2 install <package>

They will install into the site-package directory
  /usr/local/lib/python2.7/site-packages

See: http://docs.brew.sh/Homebrew-and-Python.html
@xeron

This comment has been minimized.

xeron commented Jul 19, 2017

I can confirm that adding /opt/brew/opt/python/libexec/bin into $PATH fixes this issue for me. However I also missed this notes after the update and thought that it's some bug.

@ghazel

This comment has been minimized.

ghazel commented Jul 19, 2017

This intended change does seem like a mistake to me. Python and pip used to work, and now they do not.

Can we have the old formula back?

midchildan added a commit to midchildan/dotfiles that referenced this issue Jul 19, 2017

@slhck

This comment has been minimized.

Contributor

slhck commented Jul 19, 2017

This also got me very confused – seems like a quite big change.

Ideally, this page should (have) be(en) updated: http://docs.brew.sh/Homebrew-and-Python.html

@jbub

This comment has been minimized.

jbub commented Jul 19, 2017

Seems like a pretty huge breaking change

i was using VIRTUALENVWRAPPER_PYTHON=/usr/local/bin/python, where should it point now ?

@MikeMcQuaid

This comment has been minimized.

Member

MikeMcQuaid commented Jul 19, 2017

This is an intentional change (#14408) because for years people have been complaining about things that need to depends_on "python" or depends_on :python being installed without their explicit request which overrides their which python. I appreciate this is frustrating if you're in the opposite camp (you liked that behaviour) but there was no workaround for the prior group. Additionally, the way Python is being handled now is more consistent how we should be handling other formulae that shadow system versions.

Note from the discussion in #14408 that the long-term plan is to have brew install python install Python 3 and Python 2 be a keg-only install with brew install python@2.

Ideally, this page should (have) be(en) updated: http://docs.brew.sh/Homebrew-and-Python.html

Done in Homebrew/brew#2912. That said, you were told this by the caveats when Python was installed.

I can confirm that adding $(brew --prefix)/opt/python/libexec/bin into $PATH fixes this issue for me.

This is the correct workaround if you wish to have python and pip in your $PATH. It's a one-time change for you.

@MikeMcQuaid

This comment has been minimized.

Member

MikeMcQuaid commented Jul 19, 2017

i was using VIRTUALENVWRAPPER_PYTHON=/usr/local/bin/python, where should it point now ?

/usr/local/bin/python2?

@jbub

This comment has been minimized.

jbub commented Jul 19, 2017

Thanks for the info, i feel like homebrew should prompt user for confirmation when upgrading formulas with such breaking changes, so that people avoid scenarios like these. I run brew update/upgrade everyday and i did not really spot any warning :) Anyway seems like this is a nice cleanup. Keep up the great work! Thank you

@ilovezfs

This comment has been minimized.

Contributor

ilovezfs commented Jul 19, 2017

@jbub that is a nice idea, but nothing in Homebrew is currently interactive, so instead we print caveats.

@jbub

This comment has been minimized.

jbub commented Jul 19, 2017

Yeah i mean its my responsibility to check for compatibility problems, its just i have never really had any problems like these so i never really checked the caveats, thats the nice thing of homebrew, its been really stable for years.

@ilovezfs

This comment has been minimized.

Contributor

ilovezfs commented Jul 19, 2017

so i never really checked the caveats,

Tsk, tsk :)

We're aware that they aren't always read, or are under-read, so we do make an effort to keep them as brief as possible to avoid having people tune out and never read them.

that's the nice thing of homebrew, its been really stable for years.

Thanks! We appreciate the kind words.

@orome

This comment has been minimized.

orome commented Jul 19, 2017

And I assume that adding /usr/local/opt/python/libexec/bin to /private/etc/paths will also do the trick. Correct?

@ilovezfs

This comment has been minimized.

Contributor

ilovezfs commented Jul 19, 2017

@orome yes, or /etc/paths.d/foo

@orome

This comment has been minimized.

orome commented Jul 19, 2017

FWIW, IMV, the mistake here is not in the change of configuration (which makes good sense) but in the way it was (not) communicated (which was quite inadequate).

Minimally there should have been a clear message added to the docs; but even better would have been a much more prominent message in the "caveats". I don't know whether I'm a typical user, but in my experience, these have been entirely ignorable (as the message itself states): in years of using Homebrew I've had to attend them perhaps once (or not at all). The implication and experience is that doing anything they suggest is mostly optional. But in a case where critical and frequently used tools have suddenly become inaccessible, a more prominent and more complete message is required.

@orome

This comment has been minimized.

orome commented Jul 19, 2017

@ilovezfs

yes, or /etc/paths.d/foo

Except that that will put the Homebrew Python after any system Python, which is not the same as the suggested modification to PATH or using /private/etc/path. No?

@DomT4

This comment has been minimized.

Contributor

DomT4 commented Jul 19, 2017

The issue of framing caveats prominently enough is something that has been discussed a lot over the years without much agreement on how to advance there. Certainly if you upgrade not incredibly regularly you can pile up upgrades in a way that spotting individual caveats, perhaps highlighting breaking changes, becomes rather difficult.

@slhck

This comment has been minimized.

Contributor

slhck commented Jul 19, 2017

It's all “would've” and “could've”, but certainly, if the Homebrew Python webpage had been updated at the same time, I would've been able to confirm that this behavior was intentional. The description on the website is now much better, and it all makes sense now, thanks!

Perhaps the formula caveat should've said:

Homebrew does not add a python executable to $PATH (anymore), so as not to override the macOS system python. Instead, this formula installs a python2 executable to #{HOMEBREW_PREFIX}/bin.

Then it would've been ultra-clear to me.

But isn't the real issue about changing existing behavior? It really depends on whether you:

  1. already had the formula installed (in which case it broke things or at least introduced significant changes), and it got updated, or
  2. you installed that particular formula the first time, or
  3. it got installed as a side-effect

My assumption would be that the importance of showing such caveats decreases in that order; probably most people don't care in the latter case, and – of course – would be surprised if their system python suddenly changed.

@ilovezfs

This comment has been minimized.

Contributor

ilovezfs commented Jul 19, 2017

Let's try to focus on actual technical issues with the current setup please.

@MikeMcQuaid

This comment has been minimized.

Member

MikeMcQuaid commented Jul 19, 2017

way it was (not) communicated (which was quite inadequate)

It was communicated in multiple ways:

  • there was a pull request open detailing the change for a while
  • brew update and brew upgrade showed a Python change
  • the caveats changed
  • the docs changed within 24h

I'm not really interested at this stage in doing a post-mortem on whether it was or wasn't communicated well enough or being told (repeatedly) about our failings as a project. A gentle reminder: Homebrew is run by volunteers and you use it for free. If you run a similarly scale package manager and you can point to your better implementation of handling changes like this: that would be helpful.

As for the change itself: people were complaining about how it was before, people will be complaining about how it is now. It's literally impossible to keep everyone happy or informed with a change like this. We did our best, next time we may do differently. If you want to help: consider watching the project and providing feedback like this on pull requests before they are merged.

Note this issue is being left open to avoid duplicate issues for a few days. I don't consider there anything actionable to do based on this thread.

@orome

This comment has been minimized.

orome commented Jul 19, 2017

I'm not really interested at this stage in doing a post-mortem on whether it was or wasn't communicated well enough

I'm not sure what your objection to improving communications is. It's clear what the problem here is, so it's really not helpful to anyone that you don't ant to address it. Simply wording the "caveat" better would have done the trick ("Note that this is a change from earlier configurations. Previous users of Python should ... to restore the access to Homebrew Python they have had in the past.").

or being told (repeatedly) about our failings as a project

If that's how you choose to interpret feedback, perhaps monitoring issues is not a good use of your time.

@MikeMcQuaid

This comment has been minimized.

Member

MikeMcQuaid commented Jul 19, 2017

Simply wording the "caveat" better would have done the trick

Given multiple people above have said they don't read the caveats: unfortunately, it wouldn't.

Pull requests to Homebrew are welcome, that's the best way to discuss an improvement.

If that's how you choose to interpret feedback, perhaps monitoring issues is not a good use of your time.

Just to repeat what you've said back at you: you're suggesting the lead maintainer of Homebrew stop reading Homebrew issues. Clearly that isn't going to happen.

@orome

This comment has been minimized.

orome commented Jul 19, 2017

@MikeMcQuaid

Just to repeat what you've said back at you: you're suggesting the lead maintainer of Homebrew stop reading Homebrew issues.

I think you see the logic error there: there's another adjustment that could be made (to the premise).

Given multiple people above have said they don't read the caveats: unfortunately, it wouldn't.

I can only speak from my own experience. If the caveat had been worded in any of the ways that have been suggested, that would have made all the difference. And it's worth taking this to heart (again, IMV) as a general rule: if something as dramatic as access to installed tools has changed, it's worth the few words it takes to simply say that right in the caveat.

@MikeMcQuaid

This comment has been minimized.

Member

MikeMcQuaid commented Jul 19, 2017

it's worth the few words it takes to simply say that right in the caveat.

Ok, to try and be even more explicit: we will review a pull request from you on this matter but we do not want any more comments or suggestions from you in this issue, thanks.

@orome

This comment has been minimized.

orome commented Jul 19, 2017

@MikeMcQuaid : Can you point me to where this the relevant message is generated in the code?

@slhck

This comment has been minimized.

Contributor

slhck commented Jul 19, 2017

@orome:

def caveats; <<-EOS.undent

(This is the latest commit as of posting this message. When editing, make sure to use the latest version of the file, and check if there is no open PR already.)

@micahculpepper

This comment has been minimized.

micahculpepper commented Jul 19, 2017

Hey folks,

In my organization, we use Macs with brew-installed Python. We depend on brew shadowing the system Python in order for a lot of our internal Python scripts to work.

Based on the number of people that seem to be having problems with this change, it seems like it would make sense to keep the old way of doing things around, just not as the default way. I would like to see an install option or a brew link option to create the links the old way.

I'll see if I can come up with a PR to do this so we can have something more concrete to discuss, but does this sound like a reasonable option? I would prefer not to edit my users' bash profiles, or require them to do so, in order to get things to work the way they did yesterday.

Edit:

So, based on comments from @ilovezfs and @MikeMcQuaid, I will not be making a PR for this. The long-term solution for my use-case involves a more involved setup script for installing our scripts, that creates virtual environments that satisfy our requirements and then just link the executables into the user's path. Thank you for pointing out PEP 394 -- The "python" Command on Unix-Like Systems, that was especially helpful.

@Homebrew Homebrew deleted a comment from someposer Jul 19, 2017

@Homebrew Homebrew deleted a comment from jnozsc Jul 19, 2017

@ilovezfs

This comment has been minimized.

Contributor

ilovezfs commented Jul 19, 2017

I'll see if I can come up with a PR to do this so we can have something more concrete to discuss, but does this sound like a reasonable option?

No, because this will trigger a source build. Please prepend $(brew --prefix)/opt/python/libexec/bin to your PATH if you need an unversioned Homebrew python executable.

@MikeMcQuaid

This comment has been minimized.

Member

MikeMcQuaid commented Jul 20, 2017

I would prefer not to edit my users' bash profiles, or require them to do so, in order to get things to work the way they did yesterday.

You could also adjust your scripts to use python2 or even Homebrew's full path or use the system Python 2.

@ilovezfs

This comment has been minimized.

Contributor

ilovezfs commented Jul 20, 2017

You could also adjust your scripts to use python2

Also, note that this is the recommendation of PEP 394 -- The "python" Command on Unix-Like Systems.

@orome

This comment has been minimized.

orome commented Jul 20, 2017

@ilovezfs :

Also, note that this is the recommendation of PEP 394 -- The "python" Command on Unix-Like Systems.

But doesn't the new formula also break this PEP without some user intervention, even on systems that had been following it?

The more general python command should be installed whenever any version of Python 2 is installed and should invoke the same version of Python as the python2 command.

Following execution of this formula (at least for me and the others posting here) this is not what happens: python and python2 invoke different Pythons, until the path changes suggested in the caveat are made.

@ilovezfs

This comment has been minimized.

Contributor

ilovezfs commented Jul 20, 2017

No.

@therealmarv

This comment has been minimized.

therealmarv commented Jul 20, 2017

When you are quoting PEP 394 recommendation let me quote some more:

The more general python command should be installed whenever any version of Python 2 is installed and should invoke the same version of Python as the python2 command

So with your installation method you are against the PEP 394 recommendation because python is not python2 (Homebrew) by default!

@orome

This comment has been minimized.

orome commented Jul 20, 2017

@ilovezfs :

No

Can you elaborate? Following the update, unless I change my path, python runs a different Python than python2 (against PEP 394). Prior to the update python ran the same Python as python2 (as in PEP 394).

@ilovezfs

This comment has been minimized.

Contributor

ilovezfs commented Jul 20, 2017

I suggest you re-read the caveat.

@MikeMcQuaid

This comment has been minimized.

Member

MikeMcQuaid commented Jul 20, 2017

@orome I quote myself above:

we do not want any more comments or suggestions from you in this issue, thanks.

As a result, you're getting blocked from the Homebrew organisation for now.

@MikeMcQuaid

This comment has been minimized.

Member

MikeMcQuaid commented Jul 20, 2017

Another reminder to everyone here: I was strongly in favour of having brew install python install Python 3 entirely and was convinced to compromise. I'm now increasingly wondering what the point of compromise is (mostly) with people who don't seem to be willing to expend any work to make Homebrew better but instead continue to snark and 👎 at the people who spend a lot of their free time to provide you with software for free.

@kkirsche

This comment has been minimized.

kkirsche commented Jul 20, 2017

For me it's just a matter of finding a way to better communicate the change to individuals who have it installed. I don't think the issue is necessarily the change, but rather that when doing something like brew update and updating a large number of packages at once, this can easily get lost in the output.

This is what bothered me personally. It's not to say that the change itself is a problem, just that I wonder if there is a better way to communicate these kinds of changes. Not your fault at the end of the day that I missed it, just frustrating when things get missed

@MikeMcQuaid

This comment has been minimized.

Member

MikeMcQuaid commented Jul 20, 2017

I wonder if there is a better way to communicate these kinds of changes.

I'm 100% sure there's a better way to communicate changes. The question is: what does the implementation look like and who is willing and able to implement it?

@leobalter

This comment has been minimized.

leobalter commented Jul 20, 2017

This issue is another case I appreciate a good use of semver. So bad homebrew could not prevent me from that and I had to try many things until I found this issue.

It was disappointing to figure out what was happening.

It does not make much sense when the installed pip2 goes to the usual path but not python itself.

I really like to use homebrew, it's a necessary tool, but the communication can be improved on breaking changes.

@therealmarv

This comment has been minimized.

therealmarv commented Jul 20, 2017

How about a mail list for announcements and/or a twitter account when breaking changes appear?

Maybe sites like https://brew.sh/blog/ and https://twitter.com/machomebrew are too less used? But would like to see software related breaking changes on some other place.

Or is there even a command before doing brew upgrade to see what will change? <- Actually it would be a good to read all new info texts before doing a brew upgrade for the not so adventurous upgrader 🤔

@MikeMcQuaid

This comment has been minimized.

Member

MikeMcQuaid commented Jul 20, 2017

but the communication can be improved on breaking changes.

Please, please stop just posting the same comment again and again, people. If you have something new and incredibly insightful to add that hasn't been said before. I've blocked one person and I don't want to but I will block more if I have to. Again:

The question is: what does the implementation look like and who is willing and able to implement it?

@DomT4

This comment has been minimized.

Contributor

DomT4 commented Jul 20, 2017

The question is: what does the implementation look like and who is willing and able to implement it?

Something I was thinking about earlier in terms of ideas is in the case where there's multiple upgrades not dumping the caveats until everything has finished upgrading, but I'm not sure how many people that would benefit in this particular discussion.

@kkirsche

This comment has been minimized.

kkirsche commented Jul 20, 2017

In terms of implementation I like @DomT4's suggestion of dumping the caveats at the end, grouped by package.

Another thought (just trying to brainstorm ideas) would be to have a formula implement a variable that's last_compatible_version potentially and use that to compare against the version being upgraded from. Then we could do something like if installed_version < last_compatible_version we could then prompt the user with a dialogue or something that explains that you are about to perform an update that will require manual updates (e.g. updating your path), and ask if they want to accept it.

@MikeMcQuaid

This comment has been minimized.

Member

MikeMcQuaid commented Jul 20, 2017

Another thought (just trying to brainstorm ideas) would be to have a formula implement a variable that's last_compatible_version potentially and use that to compare against the version being upgraded from. Then we could do something like if installed_version < last_compatible_version we could then prompt the user with a dialogue or something that explains that you are about to perform an update that will require manual updates (e.g. updating your path), and ask if they want to accept it.

brew outdated does pretty similar already. We're never going to prompt with a dialogue.

@kkirsche

This comment has been minimized.

kkirsche commented Jul 20, 2017

Out of curiosity, why not? Yum and apt do it at times. It's not uncommon for package managers to do this type of thing

@MikeMcQuaid

This comment has been minimized.

Member

MikeMcQuaid commented Jul 21, 2017

@kkirsche Homebrew never prompts users; it was an original design decision and at this point the consistency is internalised and not worth overriding for a single case.

@micahculpepper

This comment has been minimized.

micahculpepper commented Jul 21, 2017

PSA: If you need to call python from inside an already-running python script, and, because of this change, #!/usr/bin/env python no longer works for you, you can find the executable that got you to where you are with sys.executable.

@kkirsche

This comment has been minimized.

kkirsche commented Jul 21, 2017

Fair enough @MikeMcQuaid — while it differs from my opinion on the situation, I respect the decision. Thanks for explaining that 👍

@genebean

This comment has been minimized.

genebean commented Jul 24, 2017

Just my two cents that it would have saved me a lot of time if this had been noted as a breaking change in the output of brew upgrade or brew reinstall python or on the http://docs.brew.sh/Homebrew-and-Python.html. This bug report is the only reason I understand why my stuff just all of a sudden stopped working. I have no issue with the change, just the way in which it came about.

@acdha

This comment has been minimized.

Contributor

acdha commented Jul 25, 2017

I wanted to third support for @DomT4's idea about grouping caveats at the end, at least for upgrades. In our case, the report is “brew upgrade breaks python” because everything output was well into the scrollback buffer behind the output from the remaining packages and if you're configuring virtualenvwrapper in your shell init files it's easy not to notice the change until you open a new shell window.

@ilovezfs

This comment has been minimized.

Contributor

ilovezfs commented Jul 25, 2017

This issue is about Python not where caveats go.

@kkirsche

This comment has been minimized.

kkirsche commented Jul 25, 2017

@MikeMcQuaid to address what seems to be a communication concern is that something that should get a new issue to attempt to track and hopefully address to reduce the impact from changes such as this?

@MikeMcQuaid

This comment has been minimized.

Member

MikeMcQuaid commented Jul 25, 2017

@kkirsche No. If people want to submit a pull request to propose concrete changes we'll review it but we're not interested in any further post-mortem of this change.

@Homebrew Homebrew deleted a comment from kkirsche Jul 25, 2017

@Homebrew Homebrew locked and limited conversation to collaborators Jul 25, 2017

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.