Obsolete pip2014.com

[dstufft]

So @glyph made a thing, it's called pip2014.com and it installs pip, virtualenv, virtualenvwrapper, and adds a bunch of configuration around them all in order to make the defaults much much saner.

So, we have bad defaults, lets make the defaults good so pip2014.com is no longer a thing.

Things to do:

• Make --user the default when it makes sense to use it (Default to --user  #1668)
• Download cache by default (Cache Downloads by default #1732)
• Wheel/Package Cache. ("--package-cache" #878)
  • Simple way is to just make defauts for find-links and pip wheel to point someplace so people can prime their own cache
  • Better way is to make pip always build Wheels and always cache those built Wheels. ([WIP] internal wheel cache #1572)
• Standardize File Locations (Logs, Configs, etc) (Standardize our Paths #1733)
• Fix older version of setuptools being present and taking priority on OSX (Unprivileged installations onto OS X don't support wheels even if a newer setuptools is installed #1480)

[glyph]

pip2014.com also sets your $PATH by editing your ~/.bash_profile.

This is critically important, because the advertised behavior of get-pip.py is that, after you run it, you type pip and that runs pip.

Especially given that sometimes pip will end up in ~/.local/bin and sometimes pip will end up in ~/Library/Python/2.7/bin/pip, and you have to tell users what to type, giving them a way to manipulate their $PATH is critically important. It would be nice to do this in a more portable way.

Some users will say that this is too "invasive", and it should be opt-in. Let me just head those folks off at the pass and say definitively that it should be opt-out, because the sort of user who cares about that sort of thing is the sort of user who can figure out that they need to opt out; the user who stares, uncomprehending, at yet another -bash: pip: command not found is the one who needs the tool to help them a bit more.

I wish there were a better way to do this; a per-user equivalent of OS X's /etc/paths.d or something. But a hacky, special-case-filled platform-specific way of doing this is better than just being (from the user's perspective anyway) broken by default.

[alex]

@glyph All I will say about that is that please get which file you edit correctly. I once had a .bashrc and some tool (RVM if I recall) created a .bash_profile for me, which resulted in my .bashrc being ignored and me being /very/ confused as to why everything broke.

[glyph]

@alex I emphatically agree with one particular interpretation of that comment ;-). Basically - yes, it should be careful, it should be well-designed, and any configuration that a user has set up which breaks should be considered a serious bug in the path-manipulation tool. Perhaps this would actually warrant an entirely separate PyPA project whose entire job was being a library for managing shell environment stuff. I can see that this would be super handy and it would be nice to isolate that.

Such a tool should also pay particular attention to being able to update the configuration that it performs when bugs are fixed, so that users who install it at different times don't have a zillion different versions of environment configuraiton.

However, this should not mean that because this problem is hard to get completely correct that pip shouldn't try to do anything at all. It's broken for everyone by default right now, this is just about making it less broken for more people.

[alex]

Totally agree, I suppose the point of my comment was to info about a very
specific, and very obnoxious case to be aware of.

On Fri, Apr 18, 2014 at 10:14 PM, Glyph notifications@github.com wrote:

@alex https://github.com/alex I emphatically agree with one particular
interpretation of that comment ;-). Basically - yes, it should be careful,
it should be well-designed, and any configuration that a user has set up
which breaks should be considered a serious bug in the path-manipulation
tool. Perhaps this would actually warrant an entirely separate PyPA project
whose entire job was being a library for managing shell environment stuff.
I can see that this would be super handy and it would be nice to isolate
that.

Such a tool should also pay particular attention to being able to update
the configuration that it performs when bugs are fixed, so that users who
install it at different times don't have a zillion different versions of
environment configuraiton.

However, this should not mean that because this problem is hard to get
completely correct that pip shouldn't try to do anything at all. It's
broken for everyone by default right now, this is just about making it less
broken for more people.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/1731#issuecomment-40861138
.

"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
GPG Key fingerprint: 125F 5C67 DFE9 4084

[Ivoz]

Some users will say that this is too "invasive", and it should be opt-in. Let me just head those folks off at the pass and say definitively that it should be opt-out, because the sort of user who cares about that sort of thing is the sort of user who can figure out that they need to opt out; the user who stares, uncomprehending, at yet another -bash: pip: command not found is the one who needs the tool to help them a bit more.

I've experienced overwhelmingly the opposite. Especially pythons & rubies; they'll install one that puts itself on the path without their knowledge, and then later try to use another and meet absolute frustration and perplexion when they have an executable but its not the one they want and things start breaking.

[pfmoore]

While I understand that this is essentially a Unix proposal, I'd just like to add a data point from my experience. I tend to specifically not have (python and) pip on my path, so that I don't accidentally run the system pip and install into the system Python. So if I type pip install X without an active virtualenv, I get "command not found" rather than something installed in the system Python.

So I'm against path hacking by default on principle as the path is the user's domain.

Utilities like pip2014 whose defined purpose is to set up the user's environment are of course exempt - the user just doesn't use the utility if they don't want to. But get-pip's defined purpose is to get pip I'd dispute that adding it to PATH is part of the defined purpose, but I would accept that the docs don't make that clear. Maybe the docs should simply make it clear that after getting pip, the user needs to make sure pip is on PATH before they can use it. (After all, whatever happens on Unix, that exception will still need to be made clear for Windows, so the docs won't get any clearer...)

[dstufft]

@pfmoore Isn't that what PIP_REQUIRE_VIRTUALENV is for?

[glyph]

@Ivoz @pfmoore There are other ways to solve these problems than making pip broken by default.

To address the issue raised by @Ivoz, the problem caused by these tools is not that they manipulate the user's PATH, but that they blindly and buggily manipulate the path, and then throw up their hands when there's a bug. Pip shouldn't do that. Pip should inspect the path carefully, always select the correct binary explicitly itself and not rely on $PATH for everything, and provide a clear explanation of what it's going to do when things might not work or there could be a conflict. One thing it could do is put a stub pip on your path that just tells you how to activate the relevant virtualenv. Another would be to add a message to the shell startup where the PATH gets added so the user knows that it's been added and how to turn it off if it's not what they want.

But all the layers of indirection that let you work on multiple disparate projects are predicated on the notion that you have worked on one project to get to the point that you know what Python is ;-).

[alex]

I hate to say it, but python -m pip really solves these $PATH issues :/

On Sat, Apr 19, 2014 at 11:18 AM, Glyph notifications@github.com wrote:

@Ivoz https://github.com/Ivoz @pfmoore https://github.com/pfmooreThere are other ways to solve these problems than making pip broken by
default.

To address the issue raised by @Ivoz https://github.com/Ivoz, the
problem caused by these tools is not that they manipulate the user's PATH,
but that they blindly and buggily manipulate the path, and then throw up
their hands when there's a bug. Pip shouldn't do that. Pip should inspect
the path carefully, always select the correct binary explicitly itself and
not rely on $PATH for everything, and provide a clear explanation of what
it's going to do when things might not work or there could be a conflict.
One thing it could do is put a stub pip on your path that just tells you
how to activate the relevant virtualenv. Another would be to add a message
to the shell startup where the PATH gets added so the user knows that it's
been added and how to turn it off if it's not what they want.

But all the layers of indirection that let you work on multiple disparate
projects are predicated on the notion that you have worked on _one_project to get to the point that you know what Python is ;-).

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/1731#issuecomment-40876628
.

"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
GPG Key fingerprint: 125F 5C67 DFE9 4084

[glyph]

@alex That's one possible answer, but that also involves changing every Python project's documentation :). I think that's a bit annoying to type, but I want the average user to be using the GUI anyway.

[glyph]

Also, remember that pip is not the only tool that Pip will install. What about all the other, non -m scripts installed by other packages? How will the user find those?

[dstufft]

Well we support python -m pip so people who want to do that can do it already. I think it's annoying to type and I agree with @glyph that it doesn't help projects who are not pip. I also think it makes it harder overall for non pip things where people might not even know that a particular script on their machine is a python script.

I think at the very least we should have a big obvious warning if ~/.local/bin isn't on the $PATH during installs, and probably getpip.py should add that to $PATH. There's also the question of what we should do with Python 3.4+ and the ensurepip case to.

[dstufft]

I made a milestone for Improve our User Experience.

[glyph]

I also think it makes it harder overall for non pip things where people might not even know that a particular script on their machine is a python script.

Yes, this, precisely. Python is at its best when it can be completely submerged, an implementation detail that nobody is even aware of, a boon to the maintainers but completely invisible to end-users. The fact that Python can do this is one of its major advantages over, for example, Java.

[glyph]

There's also the question of what we should do with Python 3.4+ and the ensurepip case to.

There was some discussion at the language summit about possibly including ensurepip in a future 2.7 installer as well.

[pfmoore]

@pfmoore Isn't that what PIP_REQUIRE_VIRTUALENV is for?

@dstufft wow, I'd never heard of that. That's cool. (I just checked and it seems like it's on the PyPI page but not in the help or the manual...)

There are other ways to solve these problems than making pip broken by default.

@glyph I don't like using the term "broken by default" here. Particularly about pip. It seems to me that you are actually talking about getpip.py but it's hard to tell among the rhetoric. Can we confirm if we're talking about pip or getpip please? I emphatically do not consider it reasonable for pip to change PATH. Whether it is appropriate for getpip.py is something I'm willing to debate, though.

Specifically, I'd like to understand exactly what is being proposed here:

1. It seems that everything @dstufft originally included in this issue now has its own issue. @glyph brought up PATH changes in the first comment - but only in terms of describing what pip2014 does. Things then got a bit excited (some of which was probably my fault - I apologise if so).

2. Are any changes being proposed for pip in terms of changing PATH?

3. Is the suggestion that getpip.py ensure that pip is on the user's PATH? If so, are we talking about just for the current session, or permanently?

4. To clarify what the PATH changes will mean, when do people imagine getpip.py will be run?

   a. In a system Python? On Unix, surely not, the distro packagers would scream, but on Windows quite possibly for pre-3.4 Python.
   b. In a virtualenv? Why? pip is installed there anyway, except in a pre-3,4 venv?
   c. In a Python built from source? Surely anyone building Python from source can be trusted to handle PATH themselves?
   d. In some other situation that I'm not aware of, or which is purely for Unix?

5. What documentation changes will be needed to describe the resulting situation? Both pip and 3rd party documentation.

6. How bad would it be to just add a note to the pip documentation saying that when the command pip is referred to (both here and in external documentation) it's assumed that the user has ensured that pip is on his PATH, or that they are using python -m pip which is always a suitable (if verbose) way of invoking pip? (I'm sure someone can word that better than I just did).

[glyph]

@pfmoore I do not really have religion about an implementation strategy here. It would be absolutely fine with me if this were getpip.py's job. It could also be pip and I wouldn't mind. I would much rather if there were an entirely new installer that did not require usage of a command-line to get started, but a working command-line installer would be a good start :-). Notice that pip2014.com includes a GUI app for the mac for the setup step, and is not simply a script. This avoids the "your current terminal doesn't work yet" post-install problem.

However, if we're going to be realistic about the state of pip on an average user's computer after installation, I absolutely insist that it is broken by default, and that unless we all acknowledge that this is a functional impairment and not simply the user's responsibility to understand all the nuances of their shell and their environment and their configuration before it works, then "let's not fix anything" remains a viable option.

[dstufft]

@pfmoore

• Any $PATH changes would be permanent.
• I think it would be getpip.py's job (or any other theoretical non CLI based installer). It's an installation of pip task.
• Changes should be reduced documentation for the most part, especially in tutorials. People would no longer be responsible for adding pip to their $PATH any more.
• As far as how bad it is, I think it lowers new users ability to effectively use pip, especially on Windows where prior to 3.4 (IIRC?) even the normal Scripts folder isn't added to $PATH.

[glyph]

As far as how bad it is, I think it lowers new users ability to effectively use pip, especially on Windows where prior to 3.4 (IIRC?) even the normal Scripts folder isn't added to $PATH.

And we haven't even started talking about %PATHEXT% :-\

[Ivoz]

@pfmoore It probably doesn't help if you're not aware of all the $PATH subtletlies that go on in *nix. (not that I blame you whatsoever, because you don't deal with them)

Windows isn't actually horrible here, it's just that command line tools have never been a major occurance or culture in Windows. Windows does have both System-Wide and User-specific $PATH instances (both settable in the Environment Variables dialogue). Python 3 adds to the System one when you tick the box in its Windows Installer.

In linux, this is set from various heirarchies. $PATH is often modified by everyone except for linux newbies, and they most often do this in .bashrc, .bash_profile, .zshrc, etc for their own user. Usually a case of adding a line, that adds to the existing path, e.g. I have this to configure my go install:

# Configure Go
export GOPATH=~/code/go
export PATH="$PATH:$GOPATH/bin"

So $PATH usually only gets added to. Of course the order matters for how things are picked up if there are multiple tools of the same name in different $PATH folders.

The Windows $PATH problems are funny; indeed it has been sad for me that for so many years Python hasn't added itself to Windows' PATH so that people can't just run python from their cmd.exe after installing, until very recently if they know to tick the checkbox in Python 3 (and I guess most still install Python 2 which has never had the checkbox). I don't consider this an insane thing to do for python.org installers. And as you say with Python 3.3 you get the main Python directory on path, but not its Scripts folder (so pip still doesn't work after get-pip.py...)

A win_add2path.py script has existed in Windows' python's Tools\Scripts\ folder for ages (since Python 2) and is a great tool but is extremely under-advertised as well. (It adds Python and the Scripts folder to Windows' PATH).

In the case of Windows, a decent solution for get-pip.py might just be to offer to run win_add2path.py if it detects pip isn't on the path after installing.

In linux, since customizing $PATH is of far wider occurance for users fo *nix, I'd say just warn that pip's executable script folder (~/.local/bin) isn't currently on the PATH, and give an example line to add to a .bashrc.

[pfmoore]

Thanks to all for clarifications.

As an "installing pip" issue, I'm happy to see things improved. And yes, I don't know enough about the Unix side of things to criticise - I'm happy with whatever the Unix folks feel is appropriate there.

I'm somewhat concerned over the idea that manipulating PATH is the pip installer's issue. I can see that on Unix where there is typically no Python installer (is that correct?) but on Windows it seems to me that getting PATH right is the Python installer's job. The fact that it's taken all this time to get anything in the installer reflects the endless rounds of arguments on python-dev over the "correct" behaviour. (Seriously, it may be worth someone searching out and reading the discussions on python-dev for some perspective, although I suspect @glyph has a good idea from his work on pip2014).

By the way @glyph - PATHEXT should no longer be an issue for pip to worry about, as entry point scripts are now wrapped as exe files on Windows. That was another long debate - where I argued for .py and PATHEXT for a long time before being convinced that using .py scripts will always have surprising consequences in some situations. So if you care about the experience of Windows users, the rule nowadays is use entry points and get generated .exe wrappers. PATHEXT remains for personal use (usually via the launcher for Windows) but not for wide distribution.

[dstufft]

@pfmoore I think the debate about whose job it should be is an OK one (Probably the Python installer should handle it) but in the meantime if we can "patch" over bad behavior on the Python side (especially in the already released versions) I don't see any reason why we shouldn't on any platform.

[glyph]

There are always going to be interesting quirks on various platforms; for example, ancient setuptools on OS X is one thing that pip2014 specifically works around. Pip or get-pip should not just give up and fail when presented with these challenges though; its release cycle is considerably faster than any of the things with the quirks in them and it can adjust.

[glyph]

@pfmoore Thanks for the info about %PATHEXT%. I guess we should switch to entry points on Twisted :).

[pfmoore]

@glyph yeah, that was a hard-fought battle. We're not very good at capturing the details of why some of these recommendations are the way they are (I'm useless at it - as you may have noticed I always end up writing too much ;-))

I've just added pypa/packaging.python.org#51 as a reminder that we should cover this in the PUG.

[paulproteus]

I would just like to say:

❤️

My name is Asheesh Laroia, and I approve of this thread. To the extent that my help in the pip2014.com creation and/or publicity helped move this forward, I'm honored.

[glyph]

See also #1480 - this should probably be added to the description. (Or… the first comment, I guess?)

[dstufft]

So pip caches by default now, one thing marked off the list.

[glyph]

Great!

[glyph]

Just reminding everyone that it is 3/4 of the way through 2014 and in a few months in 2015 I am still going to be recommending a website called pip2014 to get the pip of the future … ;-)

[dstufft]

Up until a few minutes ago there was one less item done :(

[dstufft]

I'm going to close this issue, the only remaining thing is defaulting to --user which has it's own dedicated issue. Anything not covered by that should also get its own dedicated issue.

[pradyunsg]

@glyph By any chance, could you link to this issue from http://pip2014.com? :)

[glyph]

@pradyunsg A new baby took priority for a while but https://pip2014.com now has a link to this issue, as well as to pipenv.

[pradyunsg]

Awesome! Thank you!