Add documentation on pip's role as an installer

[pfmoore]

Posting a very initial draft here for comments. This was prompted by this comment on discourse, which suggests that it would be useful to have somewhere that explains the logic behind not including workflow capabilities in pip.

In order to avoid an extended debate over pip's long-term role (which we might need to have, but I don't think here is the place) I'm taking the view that this PR is documenting the current situation. If at some future date we decide that pip should become a workflow tool, competing with the likes of hatch, PDM and poetry, then we can revisit and modify this document. But for the moment, having something that explains the current state of affairs is IMO useful.

I've kept this relatively short. I think the basic points are fairly easy to cover, and I don't think it helps to spend a lot of text trying to justify our position - it simply invites arguments which typically end up going nowhere. But if people feel otherwise, I can expand this to cover why adding things like environment management and locking would be a significant scope change that we can't currently sustain¹.

Footnotes

1. Sneak preview there of the arguments I'd make 😉 ↩

[potiuk]

Fully agree, should be short and importantly it is great that it explains that this is an absolutely conscious decision made by pip maintainers to do just what you describe.

[notatallshaw]

This is great, the question of why Pip doesn't take into account of the existing environment often comes up, and the answer to how a user could have found the answer or motivation at the moment is "read the giant history of GitHub issues and discussion".

Maybe not in this PR, but in the future might it make sense to suggest some popular other tools? Not including conda tools, I think PDM, Poetry, and Hatch have all reached a base level of popularity that currently they would all be worth mentioning.

[pfmoore]

Maybe not in this PR, but in the future might it make sense to suggest some popular other tools?

No, that's the job of the packaging user guide, not the pip manual, IMO.

[notatallshaw]

No, that's the job of the packaging user guide, not the pip manual, IMO.

Then perhaps a link to the packaging guide? I'm thinking of the user experience of "okay, this explains Pip isn't the right tool for me, what now?".

[chrysle]

Then perhaps a link to the packaging guide?

You could link to the tool recommendations page (which is currently being refurbished).

[chrysle]

Suggested change

	In particular, this includes a number of workflow tools.
	In particular, this includes a number of [workflow management tools](https://packaging.python.org/en/latest/tutorials/managing-dependencies/).

I'd find this worth mentioning. Feel free to discard.

[pfmoore]

Thanks for the suggestion, but I don't personally think "managing dependencies" is the whole story here - there's environment management, project templates, running tests and other project-specific actions, and a bunch of other stuff that "workflow tools" do which complicate the whole issue. Also, I thought the PUG was being updated at the moment, so I'm reluctant to try to link to specific sections while that's going on. If pressed, I'd be willing to link to the top level of the PUG at the end of the first paragraph:

... one where there are many views on the "correct approach". The Packaging User Guide covers this in more detail.

but I honestly think that distracts from the core message here.

I'm going to stick with my instinct that pip shouldn't get involved in all this, and keep it simple and descriptive.

Having said that, I'd still like to hear from the other @pypa/pip-committers before merging. I think we should at least have rough consensus before making statements about pip's "core purpose" in the official documentation.

[chrysle]

I don't personally think "managing dependencies" is the whole story here

Yes, I agree. Only this was the most fitting source I could find (at the moment). Linking to the tool recommendations would be better.

If pressed,

Surely not ;-). I'm just a passer-by.

I'd be willing to link to the top level of the PUG at the end of the first paragraph:

A PUG link would definitely be helpful, IMHO.

[sbidoul]

This reminded me that I wrote (not publicly) about the pip scope a few years ago, and after re-reading today, I think it is still valid.

My view is that pip [is a CLI to] manage distributions installed in an existing environment. This includes installing and resolving dependencies, of course, but also uninstalling, and upgrading. This also includes “querying” what is installed (inspect, show, list, freeze, check fall under that umbrella).

Since pip can install from source, this implies building (the wheel command). And since installing requires obtaining the distributions from somewhere, the download command is quite natural.

On top of that developers want a lot of different things in their workflow.

Managing the environments themselves (creating them, deleting them) is one of those things. It is not in scope for pip at the moment (and may never be), likely because there is so much diversity in the ways python environments can be defined and/or come into existence.

Managing the “project” is another thing people want (for instance, adding or removing top level dependencies). This is also not currently in scope for pip, if only because what constitutes a project does not seem to gather that much consensus today. Pip also has no notion at all of what a project is, beyond the fact that some directories can be built into wheels or installed as editables.

Installing python itself also comes to mind as a fairly new feature people have come to expect from workflow tools. That also is obviously not in scope for pip at the moment.

And many more.

All this to say I'm a little uncomfortable with limiting the pip scope to be an installer only, but I do agree it's a good thing to have some sort of scope statement in the documentation.

[pfmoore]

Thanks. I like the framing of pip's role as "managing the packages in an environment". That seems better to me than simply an "installer" (I remember reading your original post now that you mention it, and I agreed with it then as well).

I'll try to re-word the statement around the idea of a package manager.

Personally, I see build and download as steps in an install. We expose them as a convenience for people who want to break down what pip does into parts, but I wouldn't want to position pip as a full-fledged project builder (for example). Does that sound reasonable?

[notatallshaw]

I'll try to re-word the statement around the idea of a package manager.

I'm a bit confused, isn't the point of this to exactly say Pip isn't a package manager? I would expect a package manage to manage the lifecycle of a package.

E.g if I install foo and then later installed bar with a package manager I would expect the package manager to take into account the transitive/dependencies of foo and not break it with the installation of bar. Or if I installed foo with a package manager then uninstalled foo immediately afterward I would expect a package manager to uninstall all the dependencies.

While I agree with @sbidoul that Pip provides more functionality than an installer, I would expect the word "manager" to mean that something is managed, which by my understanding of the word Pip explicitly doesn't do.

[sbidoul]

@pfmoore

I'll try to re-word the statement around the idea of a package manager.

👍

Personally, I see build and download as steps in an install. We expose them as a convenience for people who want to break down what pip does into parts, but I wouldn't want to position pip as a full-fledged project builder (for example). Does that sound reasonable?

I tend to agree that build and download are second-order, convenience features. However I think pip has to be (or include?) a correct and complete (whatever that means) build front end to achieve its installer purpose.

@notatallshaw I tend to see the limitation you mention as an implementation choice. In my understanding this does not undermine pip's qualification as a package manager. I don't know if a better name exists to summarize what pip is, but at least it helps clarifying that pip is not meant to manage environments nor python projects.

[pfmoore]

I tend to agree that build and download are second-order, convenience features. However I think pip has to be (or include?) a correct and complete (whatever that means) build front end to achieve its installer purpose.

This leads into an interesting question. We don't currently have a command to build a sdist. Based on the principle that pip's role is managing installed packages, that makes perfect sense, as you don't need to build a sdist at any point in the process of installing a package. So I'd say that while pip must include a correct build front end, it doesn't need to be complete, in the sense that we have no need for a command reflecting build_sdist. Users who want a complete build frontend should be using build, which is explicitly targetted at being a complete build front end tool.

My current plan is to add this to the PR, as a concrete example of where pip's "project management" or "workflow" features are limited to those needed to support installation (package management). Do you have any objection to that?

(Note that if we want to add a pip sdist command at some later point, this doesn't stop us. The existence of this new page doesn't mean we can't change the scope of pip - it simply means that we need to be conscious of why we're doing so, and make sure we can document our reasons for wanting to make the change).

[sbidoul]

@pfmoore I agree. I did not think about building sdists when I wrote that.

[pfmoore]

Discussion seems to have died down. I'm going to take this out of draft mode, and leave it a few days for any final comments before merging.