Need recommended tool to build project using PEP 517 / 518

[jaraco]

Historically, I've had packages that depend on setuptools_scm to load metadata. Due to issues with setup_requires/easy_install and SSL support, I've tried dropping the setup_requires dependency to rely on PEP 518 and pyproject.toml to resolve the build-time dependencies.

However, in that mode, there's no command I can run to run "setup.py sdist" in the context of the build dependencies.

I know @benoit-pierre is working on a patch to allow pip to install dependencies as declared in pyproject.toml, which with in conjunction with pip-run could create a command that would do the build.

[njsmith]

I feel like @takluyver was working on a tool for calling PEP 517 hooks at some point? I might be misremembering.

[pfmoore]

There's a library https://github.com/pypa/pep517 which is probably what you're thinking of. It has PEP 518 isolated environment functionality, but it's separate from pip's implementation, so probably not as battle-tested yet.

It would also be possible to add a pip sdist command, parallel to pip wheel that used the build_sdist hook to create sdists. I'm not 100% sure if that's something that we'd consider core pip functionality (it's about building packages more than installing them) but a proposal to add such a command (particularly if accompanied by a PR ;-)) could certainly be considered.

[gaborbernat]

I think I've proposed to have pip build wheel/sdist command quite a while back. To be fair neither pip wheel is really about installing packages, so I would argue having sdist is just as valid. Furthermore, I think pip is more like the de facto build frontend available out there so having pep 517 described build frontend capabilities should be added/considered. And considering this build wheel and build sdist should be supported.

[njsmith]

Donald has also mentioned the idea of having twine handle building sdists/wheels.

I still don't understand why twine and pip should be different tools, but if they're different then putting this stuff on the twine side does of the split does have some logic to it.

[gaborbernat]

twine is mostly about upload, pip is download and install🤔 I can see making packages belonging to both. I mean to make a wheel to upload you need build. To install and sdist you need the build logic.

[jaraco]

Personally, I've never liked the pip wheel command. First, unlike freeze and install and download, wheel isn't a verb (the build- prefix is implicit). More importantly, for separation of concerns, I'd much prefer for there to be separate tools for building, installing, publishing or one tool that wraps them all. With separate tools, because they have shared responsibilities (installer needs to build sometimes), they would have dependencies. It feels right and natural to me that pep517 is a build tool, or that there could be a higher level interface that performs the build. However, when pep517.build is about 100 lines of code, it barely justifies a separate library.

That said, if pip were to also expose something like pep517.build as pip build (eventually obviating wheel), I could get on board with that.

[jaraco]

In this commit, by installing pip@master and pep517 at the feature/build-command branch, cut releases of wheel and sdist (in Travis-CI) for any PEP 517 complaint package and avoiding invocation of setup_requires/easy_install.

[pfmoore]

I'm not a huge fan of bundling everything packaging-related into pip either. But it seems like a lot of people want it...

[ncoghlan]

The "why not one tool to rule them all?" perspective is the already mentioned one: there's no reason to tightly couple the publishing toolchain to the installation toolchain and plenty of reasons not to (it's literally taken years to undo the original tight coupling between setuptools and easy_install and that work still isn't done yet).

For-profit corporations anchoring their own ecosystems have commercial reasons to pursue an all-in-one approach, but those reasons don't apply to PyPA.

The one UX argument in favour of all-in-one is that it means you don't need to declare any build dependencies by default, but honestly, there are a lot of projects where the most appropriate state for them is to not be published on PyPI: releasing a package means making a non-trivial future commitment to anyone that depends on it, and if needing to explicitly declare their publishing toolchain is enough to put people off publishing at all, that's not necessarily a bad thing (PyPI is already growing at a faster rate than humans are discovering interesting new problems to solve with software, so most of that growth has to be coming from functional duplication - it can easily be interpreted as a sign of ecosystem UX failure on the software discovery and re-use side of things, rather than UX success on the eaae of publication side)

The ambiguous status of wheel building as both a publishing and installation task stems from allowing both source and binary inputs to the install process, and generating the binary artifact from the source one: if you install from source, then wheel building is an installation task, while if you install from a wheel, then making the wheel is a publication task.

Whether building an sdist is an installation task or not then depends on how you define installing from a source tree: if that necessarily builds an sdist first, then building an sdist is a required installation task.

PEP 517 isn't written that way, though: when installing from a source tree the installer just asks the build backend for a wheel archive, and the backend decides how to make it.

[pfmoore]

One open question for pip is whether we should build source trees via sdists (currently we don't). The above comment adds an interesting perspective - if we build via sdist, a pip sdist command falls out of that decision logically, whereas if we build direct from the source tree, there's less justification for having pip sdist.

[dstufft]

I don't think the question of whether pip should handle uploads or not is anything like the setuptools issue. Setuptools was so "locked in" mostly because of the fact that the interface that tools like pip was defacto just "whatever setuptools did", so any other tool had to emulate an ill defined interface, coupled with the fact that there was nothing like PEP 518 that would allow a competitor to setuptools to be preinstalled, so it had to gain some level of critical pass before it would work as a setuptools replacement.

The thing with pip is that coupling these two things doesn't represent a major risk to the ecosystem itself, the way pip talks to PyPI is standardized and documented. Anyone can make a competitor that does the same thing and we've been careful to avoid special casing pip (except as a default option) so that such a competitor could be written.

The reasons, both for and against, doing it are largely down to deciding what kind of UX makes the most sense for people, whether combining them into a singular tool where trade offs would have to be made at the command level provides the best experience, or whether keeping them split apart and thus allowing the command level to be more focused, but require two tools, provides a better experience.

[pfmoore]

The thing with pip is that coupling these two things doesn't represent a major risk to the ecosystem itself, the way pip talks to PyPI is standardized and documented

While that's true in theory, in practice PEP 517 puts a significant load on other tools, because of the need to create isolated environments, which in turn means installing prerequisites, which carries a significant burden because pip has so many options controlling the install process, that users could reasonably expect to be exposed by any tool that implements PEP 517.

I agree that by keeping things standards-based, we make sure that pip isn't in a privileged position of being "the only option" - but we need to remain aware of some of the practical issues and not glibly assume that the standards make everything fine.

Maybe what we need is a few more standards:

• A standard for mechanisms to control how to configure the way packaging tools connect to repositories (covering things like pip's --proxy, --trusted-host, --extra-index-url etc flags in a tool-agnostic way).
• A standard for configuring package selection choices (similar, for --pre, --only-binary, ...)

Upload doesn't really suffer from these issues, but sdist building and any generic interfacing to PEP 517 hooks does, because of the build environment issue. We had this problem once before, with setup_requires metadata not being installed in a way that respected pip's configuration. Let's not ignore the risk that we could find ourselves in a similar situation in future.

Note: I don't think this is a huge risk, I just think we need to be careful not to assume that having standards solves all our ecosystem diversity problems. After all, one of the reasons pipenv vendored pip was because it was the most practical way for them to avoid bug reports saying they behave differently (excuse any misinterpretation of a half-remembered comment I might be making here) - if interop standards were all the answer we need, that shouldn't have been necessary.

[jaraco]

Those challenges you mention remind me of another use-case that I found surprising and challenging when attempting to remove vendored dependencies in setuptools, the desire of other package management tools to be able to build from source, preferably without pip, demanding that the build dependencies form an acyclic graph. I'm not sure if that's necessarily relevant to this consideration, but I wanted to highlight it in case it is.

[dstufft]

I don't think there is a risk? Things like --proxy, --trusted-host, --only-binary make sense for pip, and make pip more usable in more situations, I also think it's perfectly reasonable to have a tool that doesn't have those things. The most obvious one would be I think it wouldn't be unreasonable for someone to try to make a pip-like that only supported installing from wheels, and thus the entire --only-binary and --no-binary set of ptions make no sense what so ever.

Importantly, while pip can handle both producer/consumer side, there's no requirement that a competitor has to implement both sides of that. If someone wanted to integrate uploading into their tool (a la flit) there is no sort of lock in or similar making them also implement the rest of things as well or needing to conform their upload command to be the same "shape" as pip's.

Likewise if someone wanted to just implement the consumer side of things, they can completely ignore the fact pip has uploading built into it. They can make a competitor to pip as an installer.

Of course there is a lot of battle tested code in pip, so trying to replicate pip may require a significant amount of coding, but just because pip is complex and if someone is writing a competitor they might have to duplicate code doesn't mean that there's lock in here. The closest thing to lock in is honestly probably the requirements.txt format, since that is used as a defacto standard by a lot of tools (like Heroku and such), but beyond that, pip should be mostly able to be swapped out for something else regardless, if something else were written to take it's place.

[pfmoore]

The most obvious one would be

Would be what?

thus the entire --only-binary and --no-binary set of ptions make no sense what so ever.

Probably. The options that I think are hardest to ignore are all the various networking ones - proxy, trusted hosts, extra index URLs, etc. Corporate environments will often have specific requirements here, which they will typically put in pip configuration. So maybe it's OK when creating build environments to use pip but simply not pass any options. But I can see people expecting tools to have "the same options as pip". To be honest, though, at the moment, we don't really have any experience of actual use cases, so it's all a bit theoretical. That's one reason I'm pushing people to try pep517, to get some actual experience.

There's a lot of differing points here, BTW. I think you're mainly talking about twine/pip publish, whereas I'm talking about building sdists, and setting up build environments. The trade-offs are likely different in those two use cases.

[dstufft]

Er, ignore the fact I left in a fragment of a sentence I deleted ;)

Things like --proxy and such are things that are useful and most tools are likely going to want, but I think it's also reasonable for a tool to say they don't support environments where proxies are required or multi index support or anything like that and just opt out of them. Of course that decision reduces the cases their tool is useful in, but I don't see that as any different than flit deciding it doesn't want to support C extensions.

I think the same things apply to setting up build environments and the like. For instance pip is likely going to support some stuff for legacy reasons or because we're trying to cater to as wide of an audience as we reasonably can. However someone else might decide they want to be Python 3 only, and they're just going to always use a Python 3 style venv to implement their build isolation.

Unfortunately there is likely always going to be some level of "users expect X, because pip does X", but the best thing we can do I think is to standardize the bits that actually need to be shared between tools, and then the rest try to design our standards and the way things work so that different tools can make different decisions about how to implement something.

[pfmoore]

the desire of other package management tools to be able to build from source,

There's a valid argument for minimising dependencies, and I get that. But the desire to build everything from original sources, with no reliance on already-available builds, seems to me to be more driven by a philosophical preference than by practical concerns. And I'm not sure I have much sympathy with that position. Sure, the build process for pip or setuptools depends on having a six wheel, and six needs pip or setuptools to build that wheel. But who cares, it's more convenient for the maintainers. If you want a "from scratch" build process, you can create a six wheel by hand - no-one is stopping you. But apart from self-imposed constraints and armageddon scenarios (every wheel on PyPI gets destroyed), there's no point.

So no, I don't think it's directly relevant here. Vendoring dependencies is something that tools like pip and setuptools have to consider, to avoid a chicken and egg issue for users, but I don't think it affects what we're talking about here.

[dstufft]

As an example, I don't think that it's unreasonable at all for a tool to only build sdists, and to say that it expects the environment to have already been set up prior to invoking it. So that this hypothetical tool doesn't get involved in PEP 518 dependencies at all, and just expects that the user has already set up environment. Likewise I think that other tools will get involved in PEP 518 and will want to set up a build environment prior to actually invoking the PEP 517 hooks.

[pfmoore]

I think the same things apply to setting up build environments and the like. For instance pip is likely going to support some stuff for legacy reasons or because we're trying to cater to as wide of an audience as we reasonably can. However someone else might decide they want to be Python 3 only, and they're just going to always use a Python 3 style venv to implement their build isolation.

Ouch. That's a very good point, and one that's directly relevant here, as this issue is about "what options exist to build a sdist with PEP 517/518 support". At the moment there are two obvious possibilities, pip and the pep517 library. Pip isn't an option because it doesn't build sdists. pep517 is, but it's made some as-yet untested assumptions about how it builds isolated environments.

I think the build environment side of PEP 517 is potentially the hardest part for tools to implement for themselves¹, and the one that's crying out for library support. But it's not at all clear what the right choices here would be for a general library.

On that basis, what @jaraco is doing here, trying to battle test the new standards, is really useful, even if we don't yet have good answers to a lot of the questions raised.

¹ Getting things like cyclic build dependencies and forkbomb prevention right in pip's implementation took some time, for instance.

[dstufft]

I think ideally the PEP 517 invocation code is independent from the build isolation code, even if they live in the same library.

[pfmoore]

just because pip is complex and if someone is writing a competitor they might have to duplicate code doesn't mean that there's lock in here

Agreed. At this point the question is more about "splitting useful parts of pip's internals into a library" than about "writing standards that allow people to create their own implementations". Which is something of a digression from the point of this issue.

the PEP 517 invocation code is independent from the build isolation code

It is, in the pep517 library.

[njsmith]

the desire to build everything from original sources, with no reliance on already-available builds, seems to me to be more driven by a philosophical preference than by practical concerns.

There are corporate environments where this is policy, because they're worried that someone might upload a wheel that doesn't match the source. (E.g. in the recent event-stream mess that npm had to deal with, the malicious code was added to the built artifact only, not the source.) I think the logic is that while they can't audit every line of code they run, they want to know that they are least have the source code that they're running, in case they need to do forensics or emergency bug fixes.

[pfmoore]

So that's the sort of environment where putting some investment into a dedicated, custom toolchain to build the baseline wheels from source is not unreasonable. After all, it's not exactly hard to create a wheel for a pure Python library "from scratch", using nothing more than the stdlib zipfile module.

Anyway, this is a digression, and it's only my opinion anyway :-)

[ncoghlan]

FWIW, Fedora/RH have a separate "bootstrapping" phase for each new Python version where they create deliberately broken cpython/setuptools/pip/wheel RPMs from the plain source archives that are nevertheless usable enough to build complete and correct RPMs for those projects on the second pass. (Part of that bootstrapping is skipping the docs builds, since there isn't a working version of sphinx available while bootstrapping a new Python release, but the other aspect is resolving the circular dependencies between those components).

Python's far from the only ecosystem to have that problem - it comes up as soon as an ecosystem is mature enough to start self-hosting its own tooling.

(The situation in pypa/setuptools#980 arises from the fact that setuptools itself didn't historically have any circular dependencies involving it, and trying to get it away from vendoring everything introduces some)

[pradyunsg]

I'm late to the discussion so I'll just pitch in with my 2 cents on topics in this discussion:

Regarding of pep517 and pip's isolation, I genuinely think we should pull out pip's isolation out and make it either part of pep517 in some form or bring them both in line with each other. That's probably the best solution to this situation moving forward.

To me, having both pip install and pip build looks great -- once the build mechanism / interfaces are well documented, this is not the same problem as the one with setuptools / easy_install. I'm feel there's merit to approach it such that this interface can share some common implementation as well -- pip using twine for pip publish (if we do that) / pip using pep517 (or a can-handle-all-builds library) for pip build. The lower the barrier for a different tool to do builds like pip, the better. If that approach also allows doing things slightly differently, even better. :)

TBH, I think there's even merit in actually making more of pip into separated "fairly" independent libraries; making it possible to have shared implementation for common packaging things (building packages, finding packages, resolving, handling installing, etc) -- then pip just becomes one tool that has its own trade-offs/choices for how it uses these libraries (and something like pipenv can trivially replicate the behavior of pip or implement its own). This is a lot of work obviously but the pipenv folks have shown interest in this too.

The lower the barrier for a different tool to do builds etc the same way as pip, the better.

Regarding pip publish, I'm on the fence. [snip big paragraph] Most of what I'd pointed out has already been stated in #60 in some form, so I'll not digress here.

[jaraco]

Now that pep517 0.5 is out and implements the pep517.build command, I've re-named this ticket to focus on where the recommended functionality should reside.

[takluyver]

To record my position on the new issue: I'm inclined to recommend pip. It's going to need the ability to act as a pep 517 frontend, including the complexities of finding and installing build dependencies, in any case. On Mon, 17 Dec 2018, 6:37 p.m. Jason R. Coombs <notifications@github.com wrote:

Now that pep517 0.5 is out and implements the pep517.build command, I've re-named this ticket to focus on where the recommended functionality should reside. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#219 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAUA9UjqmZDWWr7X7PnvZ8kXv8WKdLwjks5u59ZygaJpZM4ZKUUs> .

On 17 Dec 2018 6:37 p.m., "Jason R. Coombs" <notifications@github.com> wrote: Now that pep517 0.5 is out and implements the pep517.build command, I've re-named this ticket to focus on where the recommended functionality should reside. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#219 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAUA9UjqmZDWWr7X7PnvZ8kXv8WKdLwjks5u59ZygaJpZM4ZKUUs> .

[pfmoore]

I'm happy to support pip build sdist as the correct longer-term recommendation.

But it's not a simple case of implementing that command. There's a lot of rationalisation that would (IMO) be needed to pip's command structure before implementing pip build. For example, there's a lot of overlap and inconsistency between pip wheel, pip download --only-binary :all: and the potential pip build wheel, which I think we need to sort out first. But that's a discussion for the pip tracker, not here.

[pradyunsg]

There's a lot of rationalisation that would (IMO) be needed

Agreed.

that's a discussion for the pip tracker, not here.

Indeed.

[pradyunsg]

Apparently I forgot to cross link this issue. :/

There's significantly more discussion on this issue, here: https://discuss.python.org/t/building-distributions-and-drawing-the-platypus/2062

[uranusjr]

There is now a library exposing the same functionality as pep517.build. See https://discuss.python.org/t/moving-python-build-to-pypa/4390.

[pganssle]

There is now a library exposing the same functionality as pep517.build. See https://discuss.python.org/t/moving-python-build-to-pypa/4390.

As it is currently scoped, that project does not expose the same functionality as pep517.build, and I don't think it would be a suitable recommended build front-end, unfortunately. I've commented in the thread to that effect, and maybe its scope can be expanded to be something we can reasonably recommend.

[henryiii]

This has been moved over and I think the docs here: https://setuptools.readthedocs.io/en/latest/build_meta.html probably should recommend python -m pip install build && python -m build; they still recommend using pep517.build!

[pradyunsg]

For anyone wondering, https://github.com/pypa/build is the "final" answer to this issue.

We're mainly transitioning away from pep517 as a CLI tool to build as the main user-facing library + CLI. :)