pip needs an API that can be used by build automation

[nanonyme]

Currently pip doesn't have a public API so a developer using pip to automatically create deployable environments must use pip as a subprocess. It would be better if pip had a simple public API for most common things. I'm proposing these

• installations (setting PyPI address, requirements as list; requirements as file most likely unnecessary as build script can just read requirements from file and pass as list)
• freeze (gathering already installed items, output should be an iterable)
• uninstalling (input most likely iterable so can be passed eg what is given by freeze)

This combination would allow a flow clean+build through a Python build script.

[cowlicks]

I want this too. Specifically a way to get dependencies and check a packages availability pypi.

[andy-maier]

I want this too.
In addition to the list mentioned above, I'd like to be able to search for packages on PyPI, and to retrieve information about a specific package / package version.

Edit: I am using pip.commands.search.SearchCommand() currently, and it just changed something the other day so my code broke.

[pfmoore]

Most of the requested APIs can be handled in a supported manner by calling pip as a command using the subprocess module. If that's not sufficient, then for specific tasks there are other modules like packaging and distlib that provide a Python-callable API for packaging related tasks.

We understand the desire for a pip API, but "just providing it" isn't as simple as it seems. Pip is designed as a command line program, not as a library, so the internal APIs are not really designed to be exposed. Also, there are risks to installing/uninstalling packages in a running Python process (the Python import machinery isn't designed with that scenario in mind) so even if we did provide an API, it would probably come with a lot of warnings and caveats anyway...

[cowlicks]

@pfmoore the behavior @andy-maier and I requested should not un/install packages. We want an api to package meta-data.

I also wanted an api for resolving package dependencies, but IIRC this isn't possible with pip's current architecture because it isn't determined before install time.

[RonnyPfannschmidt]

local metadata is available via packaging and/or setuptools pkg_ressources

[pfmoore]

... and the original request on this issue was for install/uninstall, which @andy-maier said he wanted too, hence my comment.

Searching on PyPI is pretty trivial using the XMLRPC API directly, and distlib provides functions for that if you prefer a "pre-packaged" version. As @RonnyPfannschmidt says, for dependency info, packaging and pkg_resources will do that for you locally.

For remote files, dependency data isn't exposed. That's a limitation of current metadata. The only way of getting dependency information for an uninstalled package is to download it, unpack it, and run setup.py egg_info (or for an egg, read the egg metadata files). Pip doesn't, and will never, expose the results of that. What will happen, in the future, is that metadata standards will be extended to expose dependency information (it's still an open question whether dependency data can be exposed statically, some people consider that certain aspects can only be decided at build time) - at that point, PyPI (or hopefully by then, Warehouse) will expose that data via the web interface. Probably there will be wrapper APIs for that, but likely not from pip but from other 3rd party projects (that's the benefit of a defined standard, you're not reliant on getting everything from one provider!)

[nanonyme]

@pfmoore Yes, I get that you want to keep the current API's internal and not have stable API through them. This request wasn't a matter of doing that but instead formulating an API something that works with the regular commands, potentially with better error handling. Eg distinguishing between fatal and non-fatal uninstallation errors currently involves parsing console output

[pfmoore]

OK, then I suggest that you check the setuptools, distlib and packaging APIs. And possibly elsewhere on PyPI. They may have what you want. If not, then it should be possible to write a module (and publish it on PyPI) that does what you want. I'm not clear why this has to be in pip.

If you do have a reason why it must be in pip, then you could create a PR proposing the API you require, It wouldn't be easy to get it accepted, as you'd need to address all the existing reservations the pip devs have about exposing a stable API, as well as explaining how the proposed API would avoid acting as the "thin end of the wedge", prompting more and more requests for APIs in other areas. But if you feel that the effort is worth it, then by all means submit a PR.

Personally, I doubt there's any real need for something like this to be in pip, as opposed to a separate project.

[nanonyme]

I still disagree with above since imo such a library would as far as I see it basically be re-implementation of pip internals and if it would actually be high quality, pip would have little room for existing other than as a shim on top of it. However, I'm not going to proceed with the PR against pip and will withdraw my feature request since obviously developers disagree with the idea of the API in pip despite the severe limitations using pip in any automations currently has

[pfmoore]

Note that as a user, I'd be interested in such a library, I wasn't just suggesting a theoretical option. I don't see how it would re-implement pip, as it's specifically been noted that it won't do installs, which is a huge part of the complexity of pip. But if you don't want to pursue the idea, that's fine. Thanks for the suggestion in any case.

[nanonyme]

@pfmoore See my first comment. This feature request's scope was API to do installs, upgrades, uninstalls and getting list of installed packages. I don't get where you got the idea of not doing installs. I don't personally care about those other use cases mentioned here

[pfmoore]

Sorry, I was confused - it was @cowlicks who commented

@pfmoore the behavior @andy-maier and I requested should not un/install packages.

Agreed, if you're asking for full pip functionality then yes it would need to be an API exposed by pip (or you;d have to rewrite most of pip). In which case, the original comment stands - sorry, we don't expose a supported API.

So:

• If you only want metadata enquiry APIs, you're probably best writing a new package for that.
• If you want full pip functionality (install/uninstall) we only support command line use (you can run pip via subprocess.Popen, of course, if you want to invoke it from another program).
• The pip.main entry point, while not officially supported, is probably stable enough - but it doesn't offer much more than command line use does.
• There are no plans at the moment for a supported pip API. We wouldn't reject a PR that exposed one, but you could expect a lot of pushback and questions - there are many subtle issues we've encountered over time with people using the internal APIs, and any supported APIs would have to cover those scenarios. There would also need to be some pretty significant documentation and tests as well, so it's a distinctly non-trivial undertaking.

As an alternative, projects like distlib, wheel, setuptools, packaging and probably others provide many of the building blocks you may need to roll your own solution, if you have a specific and sufficiently well-defined use case. One of the points of moving towards making pip work based on defined standards is to allow people to write their own code like this, if they prefer, and still interoperate correctly with pip.

[nanonyme]

Yes, I do use it through subprocess currently. The problem is sometimes there's bogus errors on uninstallation but I already figured out a workaround so I guess I don't need this at all. Basically:
freeze (check if packages installed)
uninstall (ignore return value)
freeze (check if packages installed)
fail if packages installed