Adopt distutils

[ghost]

Originally reported by: jaraco (Bitbucket: jaraco, GitHub: jaraco)

---

Setuptools builds on the distutils framework of the Python Stdlib. Doing so has benefits of depending on a standard, extensible framework, but also has downsides, such as limiting the advancement of distutils (across many Python versions) but simultaneously adding monkey patches to patch bugs in distutils.

I'd like to consider if Setuptools should adopt distutils (or a version of it). Doing so would decouple Setuptools from distutils, freeing Setuptools from the constraints of distutils and freeing distutils from the expectations of Setuptools.

Doing so would allow distutils to adapt at the same pace as Setuptools, addressing issues and vetting patches that could be ported back to distutils in CPython.

I've once proposed this idea and it was promptly dismissed by @RonnyPfannschmidt. Why is vendoring distutils in Setuptools not an option?

---

• Bitbucket: https://bitbucket.org/pypa/setuptools/issue/417

[ghost]

Original comment by ncoghlan (Bitbucket: ncoghlan, GitHub: ncoghlan):

---

I think this could work well with an eventual approach like the following:

1. By default, importing setuptools would leave the standard distutils namespace alone
2. setuptools would gain a "setuptools.override_distutils()" command that injected the vendored distutils into sys.modules (and complained if the standard library one had already been imported)
3. pip was updated to both import setuptools and tell it to explicitly override distutils

There'd need to be a transition period where setuptools did the injection by default, as the monkeypatching of distutils is currently implicit in importing setuptools

Longer term, we could eventually aim to move distutils out of the normal standard library update cycle, and instead install it via ensurepip (such that it evolves at the pace of the Python packaging ecosystem, rather than at the pace of the underlying language runtime)

[ghost]

Original comment by dstufft (Bitbucket: dstufft, GitHub: dstufft):

---

@ncoghlan Would that work? Doesn't Python depend on distutils to build itself?

[ghost]

Original comment by qwcode (Bitbucket: qwcode, GitHub: qwcode):

---

I love the possibility. btw, I recall this being discussed a couple years back. At the time, I guess it was too radical to gain any traction.

as for Nick's 1/2/3 points, wouldn't vendoring it, effectively give it a new namespace, and avoid any concerns with conflicts?

[ghost]

Original comment by jaraco (Bitbucket: jaraco, GitHub: jaraco):

---

My thinking was more along the lines with what Marcus is suggesting.

Thanks to Nick's comments, I'm reminded that pip install (of non-binary distributions) and also setup.py install (where setuptools is somehow imported) are expected to run with setuptools' overrides. Maintaining that expectation could be tricky, but is probably achievable.

[ghost]

Original comment by ncoghlan (Bitbucket: ncoghlan, GitHub: ncoghlan):

---

Python does depend on distutils to build some extension modules, but not the core interpreter itself.

However, unbundling per se probably isn't worth the hassle, as implementing the 1/2/3 approach I suggested above based on a "setuptool._distutils" vendored copy should provide most of the benefits with a fraction of the pain. The key will be ensuring the stdlib distutils is kept in sync with the setuptools vendored copy for bug fixes (all versions) and new features (future versions).

[ghost]

Original comment by RonnyPfannschmidt (Bitbucket: RonnyPfannschmidt, GitHub: RonnyPfannschmidt):

---

i fear this will result in a certain hell, there is no reasonable way to get rid of the current legacy within setuptools without creating many lines of compatibility breaks

[ghost]

Original comment by ralfgommers (Bitbucket: ralfgommers, GitHub: Unknown):

---

I don't quite understand how the "vendoring and put in a new namespace" idea would help. There are other packages that extend (and monkeypatch) distutils. So they'd then all have to change to instead extend/monkeypatch setuptools.distutils?

[ghost]

Original comment by andi0661 (Bitbucket: andi0661, GitHub: Unknown):

---

I like the idea of separating distutils and setuptools, but given @RonnyPfannschmidt 's and @ralfgommers 's comments, it seems to me that there is so much new legacy on top of setuptools that it would require a mechanism that allows for incompatible changes. Maybe a new major version is sufficient, or a different package name. That would allow packages to transition to it as they have a need for the better functions, less bugs, etc.

[jaraco]

@xoviat : To answer your question from the other ticket, I'd very much appreciate a PR.

[ghost]

Okay, I'll send one your way sometime soon.

[ghost]

@jaraco Please create a new distutils branch:

Ref #1191.

[jaraco]

Distutils branch now exists, created against the latest master. Is that what you had in mind?

[ghost]

Yes.

[ghost]

Is there anything left to do other than merging distutils to master?

[ghost]

Theoretically more testing is needed though.

[ncoghlan]

@xoviat @jaraco Is setuptools going to be adopting the still relevant parts of the legacy distutils documentation as part of this change?

We've kept https://docs.python.org/3/distutils/ and https://docs.python.org/3/install/ around and linked from https://docs.python.org/3/library/distutils.html because there are things documented in there that aren't documented anywhere else, but it would be nice to see that content migrated somewhere that's more actively maintained.

The other key question I would have relates to the setuptools injection in pip and other tools: existing versions of pip assume that simply importing setuptools into a process will be enough to migrate a "plain distutils" setup.py script to using setuptools instead.

While I know I suggested making that override opt-in in the original BitBucket discussion, I'm now thinking that you're going to have to keeping doing it implicitly as a side effect of import to maintain backwards compatibility with the expectations of installation tools.

[ghost]

So the current plan is to always attempt to override the builtin distutils package with the version provided by setuptools if setuptools is installed. In addition, importing setuptools should guarantee this behavior. Related to this, setuptools will preserve compatibility will all existing projects (numpy distutils included) as it has always done. Setuptools is not currently providing any documentation for distutils, but I see no reason why that would not be moved over (one step at a time).

[ncoghlan]

Cool, sounds like a good plan to me.

Given that, I think the one specific thing I would request is to have a dedicated page somewhere in the setuptools documentation about "Overriding distutils with setuptools". The premise behind that is to have a specific thing we can link to and/or update as folks find potential migration issues (similar to what we have for the pypi.org migration in https://packaging.python.org/guides/migrating-to-pypi-org/ ).

[benoit-pierre]

Looking at the code, setuptools will now patch it's own internal version of distutils, how will this work with pip? Pip expects setuptools to patch distutils so pure distutils projects are supported.

[ghost]

pip should continue to import setuptools to guarantee that distutils is patched. The monkeypatches will be migrated to to the internal distutils over time.

[benoit-pierre]

Right, provided setuptools is imported first, its version of distutils will be used.

[jaraco]

@seawolf42 @Zelgius FYI

[jaraco]

Substantial progress has been made on this effort in the distutils branch, though I haven't reviewed it.

[pganssle]

Looking at the distutils branch, it seems that we're just vendoring distutils there.

My thinking on the matter is that we should actually unify the codebases and just move the code from distutils directly into setuptools, rather than attempting to maintain distutils as a package within setuptools.

Basically I think the pypa/packaging-problems#127 model of making setuptools the reference implementation for the distutils API is better than maintaining an updated version of the distutils code. I think this is a good idea regardless of whether CPython uses an ensurepip style model to adopt setuptools as the reference implementation.

[RonnyPfannschmidt]

@pganssle i beleive in part that is entirely manpower dependen - i recall that @jaraco currently is usually only able to spark very little volunteer time every now and then, and hes pretty much the only active contributor im aware of

[pganssle]

@RonnyPfannschmidt He is not the only one working on setuptools, but in any case it's important to get the right decision.

In fact, I think it would be much worse if distutils moved into setuptools and there was only one person who doesn't have much time to work on it maintaining the project. CPython has a long release cadence but also has many contributors.

Most of the setuptools API is already subclassing and/or patching the base distutils classes as it is. I think we can start moving over those classes in their entirety rather than subclassing, and implement the patches directly in setuptools.

The main problems I could foresee this causing are people doing isinstance checks against distutils and people possibly accidentally relying on the setuptools monkey patches. Both of these may be fixable (at least to the degree that they are fixed by the "move distutils into the setuptools repo" solution) with a deprecation period.

[pganssle]

It's also worth noting that we can almost certainly do the "move all of the distutils components into setuptools" incrementally. Start with distutils.Distribution, for example, then Command, etc. Moving the entire package means that it all has to be done at once.

[jaraco]

I agree we don’t want setuptools to present distutils (which that Branch currently does). Rather, for the sake of incrementalism, I’d suggest that it include distutils as a private package (e.g. setuptools._distutils)... and eventually integrate the functionality into one unified implementation.

Perhaps it makes sense to have an even less ambitious adoption of only a portion of distutils.

I welcome any efforts, but compatibility (including the current expectation that some packages rely on the monkey patching of distutils) needs to be addressed, probably by deprecating distutils.

[ncoghlan]

While CPython as a whole has many contributors, we don't have many folks contributing to distutils any more - folks need build tools that let them target currently deployed versions of Python, which setuptools provides, but distutils really doesn't.

So the challenge for pypa/packaging-problems#127 is as follows:

• assume that in some future version of Python (3.9? 3.10? 4.0? We dunno yet) distutils is no longer in the standard library per se, but is instead bundled the way we bundle pip (i.e. by installing it from a wheel at Python installation time).
• eventually, for new Python versions (where distutils isn't in the standard library), we'd like the "real" distutils code to live in setuptools, and the code you get when you do import distutils to just be a backward compatibility shim that aliases the setuptools components into the right place
• however, for old Python versions (where distutils is still in the standard library), then setuptools will still need to do its monkeypatching magic to allow plain distutils projects to emit the installation database metadata and to be built as wheel archives

The scope of the eventual distutils-compat shim probably wouldn't need to be the full distutils API, but it wouldn't be the empty set either.

So @jaraco's suggestion sounds like a reasonable starting point to me: give setuptools its own pre-patched copy of distutils so it can work on a distutils-free copy of Python, without providing an importable distutils facade.

Building a suitable facade (with a setuptools dependency) would then be part of removing distutils from the standard library.

[pganssle]

As an aside about implementing this, I could be way over-estimating the abilities of the average student, but this feel like approximately the right size project for a GSoC student. Too late for this year I'm sure, but if this isn't done by next year we might want to consider applying.

[RonnyPfannschmidt]

@pganssle while you consider the abilities of a student right, i believe there is an underestimation on the complexity of legacy not so nice apis and their interaction in setuptools - unfortunately both codebases are already hard to navigate, and their interaction makes things even more tricky

[pradyunsg]

unfortunately both codebases are already hard to navigate

As a past GSoC student, the thing that hits hard "right in the face" is the technical debt -- working with and around it, is very much not something that formal education prepares students for.

Anyway, now would be a good time to get started on figuring out GSoC-sized projects within the PyPA and identifying mentors for them, if we're interested in having projects under PSF's GSoC umbrella for 2020 with decent amount of preparation/planning.

[jaraco]

In #2143, I've drafted a PR that adopts Setuptools from CPython master, retaining history and attribution. This is essentially the second step outlined in pypa/packaging-problems#127. I've asked a few key contributors to review, but I welcome review or comments or questions from anyone.

[hugovk]

This is essentially the second step outlined in pypa/packaging-problems#217.

Typo, it's really: pypa/packaging-problems#127 :)

[jaraco]

The Setuptools 48 release incorporates the fix and closes this issue, although there is still a lot of work to do to actually merge the functionality and remove the monkey patching.

[77FN7]

tidelift.com/funding/github/pypi/setuptools