How to bootstrap setuptools from source

[adamjstewart]

Hi, I'm a developer for the Spack package manager. We've had a setuptools package that has worked fine for a while now, and the vast majority of our Python packages depend on it. However, I went to update to the latest version of setuptools and noticed that it now depends on six, packaging, and appdirs. Unfortunately all 3 of these packages also depend on setuptools. six and packaging can fall back on distutils.core if necessary, but the setup.py for appdirs has a hard setuptools import.

It seems to me like it is no longer possible to build setuptools from source. Is this correct??

[adamjstewart]

An easy solution to this problem would be to replace:

from setuptools import setup

with:

try:                                                                            
    from setuptools import setup                                                
except ImportError:                                                             
    from distutils.core import setup   

in the appdirs setup.py. That's what six and packaging do. If you agree, I can reach out to the appdirs devs and see what they think.

[adamjstewart]

Ok, I hacked our package manager so that it can still install setuptools. spack/spack#3198 contains all of the changes necessary to build setuptools from source now that the dependencies are no longer vendored. I understand the frustration that comes with having to maintain vendors of all of your dependencies; I just hope that you don't add any dependencies that cannot be built with distutils.core instead of setuptools.

[jaraco]

Can you use pip to install the dependencies from wheels (which don't require setuptools to install)? That's the recommended procedure. Or can your package manager install the wheels?

[adamjstewart]

Unfortunately, a package manager that requires another package manager kind of defeats the purpose, don't you think? It looks like other developers have already contributed patches to fix appdirs and pyparsing so that they can be built without setuptools. Assuming the developers approve those patches, we should be good for now.

[jaraco]

Yes, I sort of see that. I worry this approach may conflict with the long-term strategy for setuptools, which is that it might require arbitrary dependencies, some of which require setuptools. In fact, an early goal is to move pkg_resources into its own packages, and that package would probably require setuptools to install. Is there a reason your package manager can't have pre-built versions of these packages available or vendor them itself (into the setuptools build recipe)?

[adamjstewart]

Is there a reason your package manager can't have pre-built versions of these packages available or vendor them itself (into the setuptools build recipe)?

In general, yes, there is a reason. Our package manager is designed to install software in the world of HPC, where you may need to use exotic compilers like xlc to install something on a bluegene-q or cray supercomputer. Since we need to support hundreds of operating systems and dozens of compilers, we haven't spent much time on supporting binary package installation like Python's wheels or Homebrew's taps. We have plans to do so in the future, but it has been low priority. Obviously, the compiler/OS doesn't matter much for non-compiled packages like setuptools, but the mechanism would have to be the same.

We have dealt with circular dependencies before. For example, pkg-config depends on glib and vice versa. Luckily the developers realized this, and pkg-config comes with it's own internal copy of glib, much like setuptools used to come with its dependencies. This can be annoying since we end up having to add the same patches for glib to both packages, but it prevents a circular dependency which is nice.

We could theoretically vendor the dependencies ourselves. That seems like the easiest solution to me if we ever run into a setuptools dependency where distutils.core is insufficient to handle the installation. I see a lot of packages that use distutils.core as a fallback. Out of curiousity, how common is it to find a package that uses features of setuptools that are not available in distutils.core?

[jaraco]

I'd say it's fairly common. And some packagers might be depending on those features and not realizing it, because pip will pre-import setuptools and setuptools monkey-patches distutils, so even a package that only imports distutils.core will be installed with all the setuptools hooks when installed under pip. Of course, this all should change when PEP 518 lands and makes it possible to declare build dependencies and thus enable dropping the implicit invocation of setuptools.

[adamjstewart]

@tgamblin @svenevs may also want to get in on this discussion.

[svenevs]

Muahahahaha! @adamjstewart I hope this discussion pushes spack over the edge and you ditch the efforts of replicating PyPi. Things like numpy and scipy -- totally, keep them, if people pip install them then they are at fault (spack has never been easy, you can make minimal assumptions).

For things like ipdb or sphinx, there's literally no reason for spack to be providing those.

The below may not be appropriate for this thread, and some of it gets convoluted quickly. Please disregard that which you feel is not appropriate here, apologies in advance!

@jaraco is it correct that the general expectation will be

1. A user installs python (aka their OS, which came with python in most cases)
2. setuptools will never be packaged directly into this (the whole distutils split -- different conversation), but with the changes in #581 are now expected to install a pre-built version.
3. This pre-built version should be independent of anything managed by pip in the ideal case.
   • AKA frown upon ye who pip install setuptools?

Is this correct, or should pip be used to install setuptools as the preferred approach on systems that have it available in both their native package manager and they received pip out of the box? The conversation got a little confusing on the other thread, setuptools is fundamental to so many things on pip, it seems reasonable to just assume it exists after a certain point.

Edit: I was browsing related issues, things seem clearer now. When PEP 518 is standardized, the pyproject.toml is the thing spack should ultimately be checking for setuptools? e.g. when pkg_resources splits, it will then be listed in the .toml for setuptools?

Do you happen to know if there is a "safe" way of configuring pip to not install / touch anything underneath the site-packages directory? E.g. only supporting virtualenv or ~/.pip? Adam and I had both looked into doing this, but it sounds like you are saying even if we found a way to prevent pip from behaving this way, that might cause problems (the "monkeypatching" business).

[tgamblin]

@jaraco: how do the plans for setuptools affect efforts by OS distro folks to create reproducible builds (podcast) of packages? It seems like these changes will make it very hard to validate a Python package entirely from source, and there are good reasons to make that possible.

Even if you have build dependencies and can install the build deps from wheels, that is still a binary in the chain that someone has to trust. So I worry a bit about this direction. I suppose that we could implement some bootstrapping logic that goes back to older versions of setuptools, if we need to, or we could rely on a baseline set of "trusted" wheels for setuptools (adamjstewart is right that we don't have binary installs yet, but they're not that hard to add). But I'd rather reproduce from source, and I suspect other distros would too.

@svenevs: other than reproducibility, the main issue with relying on pip in spack is that we need to be able to mirror builds on airgapped networks. Spack traverses a dependency DAG, downloads all the sources, and lets you build that on networks that aren't connected to the internet. If we rely on pip, we can't do that.

[adamjstewart]

In general, requiring yourself as a dependency to bootstrap yourself leads to a lot of headaches. For example, we are having a lot of nightmares trying to package openjdk/icedtea right now. If you need a java compiler to build a java compiler, how do you install the first java compiler? C/C++ compilers generally come with the operating system, but I don't think java compilers do.

[jaraco]

suppose that we could implement some bootstrapping logic that goes back to older versions of setuptools, if we need to, or we could rely on a baseline set of "trusted" wheels for setuptools

I wouldn't recommend using older versions of setuptools. That's an unsustainable approach in the long run.

You could rely on trusted wheels, or you could even vendor your own bootstrapping logic. That is, you could write your own routine that resolves the setuptools build dependencies (or hard-codes them), builds them into a viable form for import, and injects them onto sys.path before building setuptools.

Hmm. This makes me wonder if setuptools should bundle its build dependencies. Rather than vendor its dependencies in general, it could bundle its build dependencies. I'll give that a go.

[jaraco]

I've drafted a new release in the feature/bundle-build-deps branch and released b1. Install it with pip install -i https://devpi.net/jaraco/dev/+simple/ setuptools==34.3.0b1 or grab the sdist and test it in your build environment.

I notice that this change won't affect the most common use-case, that of pip installing setuptools from source when dependencies aren't met. It fails because the setuptools shim is invoked before setuptools has a chance to inject its build dependencies.

[adamjstewart]

This makes me wonder if setuptools should bundle its build dependencies. Rather than vendor its dependencies in general, it could bundle its build dependencies.

Can you elaborate on the distinction between the two options?

[jaraco]

Prior to Setuptools 34, setuptools would vendor its dependencies, including them as hacked-in copies of the dependencies. Thus import pkg_resources.extern.six would give the vendored version of six if present as pkg_resources._vendor.six or would fall back to the top-level six otherwise. It would do this even when installed. In this model, the dependencies were undeclared (nothing in install_requires).

With this new proposed technique, the dependencies are bundled into the sdist only to facilitate building and installing from source... but the dependencies are still declared as install_requires, thus relying pip or easy_install or whatever install those requirements, but providing a zip of those dependencies to use to build the package prior to installation. In the installed environment, there are no import hacks and the packages must have been installed properly and naturally.

[jakirkham]

To summarize, this would allow us to build setuptools from an sdist on PyPI without other dependencies, correct?

[tgamblin]

With this new proposed technique, the dependencies are bundled into the sdist only to facilitate building and installing from source... but the dependencies are still declared as install_requires, thus relying pip or easy_install or whatever install those requirements, but providing a zip of those dependencies to use to build the package prior to installation. In the installed environment, there are no import hacks and the packages must have been installed properly and naturally.

@jaraco: Ok, so if understand this correctly, we could then declare setuptools to have no (circular) build dependencies, and it would install fine from source. That is good. Then we could use the installed setuptools to install other packages as needed.

One question: are any of the build dependencies also run dependencies? If they are, it seems like we'll still need to make sure we install the bundled run dependencies along with setuptools when bootstrapping, or the installed setuptools won't run.

[jaraco]

are any of the build dependencies also run dependencies?

Yes. At the moment, the build dependencies and install dependencies are identical. So you would need those installed to invoke setuptools... which I can see is problemmatic if you then want to install the setuptools dependencies from source and they require setuptools.

I guess if you're using setup.py install, that will invoke easy_install, which will grab the installed dependencies and install them, but I don't recommend that.

I guess what it's coming down to is that whatever installs from source needs a way to install setuptools and its dependencies in one operation (as pip does with wheels or the setuptools did when the packages were vendored).

[adamjstewart]

I guess if you're using setup.py install, that will invoke easy_install, which will grab the installed dependencies and install them

Won't work on air-gapped clusters that have no connection to the outside world. We've been very careful about making sure that all of our packages can be installed without an internet connection.

[tgamblin]

I guess what it's coming down to is that whatever installs from source needs a way to install setuptools and its dependencies in one operation (as pip does with wheels or the setuptools did when the packages were vendored).

Ok, so I guess I have two questions:

1. Could we have an option to install the bundled dependencies along with setuptools? That would get us started.

2. Is that not basically the same as vendoring the dependencies? If so, what was so bad about vendoring, especially since setuptools is effectively a leaf in every DAG? I'm trying to figure out what problem is being solved by vendoring (feel free to point me at some other discussion...)

[dstufft]

It's different than just vendoring because it only takes place at build time, thus it allows people to say, upgrade the packaging that setuptools is using without having to also upgrade setuptools itself.

[tgamblin]

@dstufft: ok, so if we can get the bundled dependencies installed from source initially, so that we can use setuptools to install the other packages from source, does that fit the model, assuming the installed setuptools will use the newer versions when it finds them?

[kenreitz42]

This (depending on six, appdirs, pyparsing, etc) really complicates packaging on Heroku, FWIW.

[jaraco]

@kennethreitz: Is that because Heroku builds these packages from source? Why not install from wheels using pip? What is the complication?

[dstufft]

@jaraco Heroku installs setuptools (and pip) by default into those environments, but people tend to also depend on libraries like six, appdirs, etc in their own projects hosted on Heroku. The unvendoring means that setuptools and the project code are now competing over who gets to define what an acceptable version is for these libraries to be installed with.

[FRidh]

This is an issue every other distribution is going to face eventually as well. Therefore, I recommend that each build tool vendors its dependencies. At least setuptools should because to install a setuptools-based package one needs to import from setuptools.

[kenreitz42]

@jaraco people do pip freeze > requirements.txt from older setuptools installations, which then conflicts with our installation.

Here's what I'm doing about it. heroku/heroku-buildpack-python#397

[jakirkham]

Sorry if this has already been said, this discussion is quite long and have unfortunately lost the time to read it fully. That said, am still very much struggling with this issue on many fronts.

What if setuptools provided a mechanism to vendor in-place and then used that functionality for managing these dependencies? Given the discussion here it seems that vendoring needs to happen for setuptools, but also this needs to be done in a way where dependencies can be tracked and easily update. This could also be a feature for those that also need vendoring for other reasons. Admittedly vendoring is not a great thing to use, but some cases (like setuptools) seem to need it. Might as well make it less painful.

[jaraco]

With Setuptools 36, it once again vendors its dependencies.

[kenreitz42]

\o/

[reinout]

If I'm not too overenthusiastic... "when will it be released?" (I don't see it on pypi yet).

[jaraco]

Well, it should have been released automatically when I tagged the commit. I'll have to investigate why.

[jaraco]

Aah, there was an error in the deploy step due to the removal of requirements.txt.

[jaraco]

It should now be on PyPI.

[jakirkham]

Seems to not have been deployed as that change was not on a tag. Would it make sense to add a 36.0.1 tag?

ref: https://travis-ci.org/pypa/setuptools/jobs/237980870#L381

[nicoddemus]

IMHO it would make sense to just upload the release manually using twine, as it was a deployment problem and not a problem with the package.

[jaraco]

That's weird. I definitely ran the commands to cut a release. Oh, I ran setup.py release which built the dists but didn't upload them. :/

[jaraco]

Using twine, as recommended, the dists are now (verifiably) in PyPI.

[jakirkham]

Great thanks for doing this.