[BUG] Having setuptools installed causes cpython stdlib build to fail

[jmroot]

setuptools version

setuptools>=60

Python version

Python 3.9.9 (and probably others)

OS

Probably affects all; reported on macOS

Additional environment information

Originally reported in the context of MacPorts' python39 and py39-setuptools ports.

Description

If you have setuptools installed and attempt to build cpython from source, setuptools' copy of distutils may be used when building the stdlib modules and cause the build to fail due to an incompatibility.

This is easily cured by setting SETUPTOOLS_USE_DISTUTILS=stdlib if you understand the problem, but if you don't, it's a pretty baffling failure mode.

Downstream report: https://trac.macports.org/ticket/64352

Expected behavior

Cpython build uses its own distutils and succeeds.

How to Reproduce

1. Install setuptools into a convenient site-packages directory.
2. Configure cpython to install in a prefix such that it will be using the site-packages directory from step 1.
3. Build cpython.

Output

Traceback (most recent call last):
  File "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_python39/python39/work/Python-3.9.9/./setup.py", line 2598, in <module>
    main()
  File "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_python39/python39/work/Python-3.9.9/./setup.py", line 2568, in main
    setup(# PyPI Metadata (PEP 301)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 148, in setup
    return run_commands(dist)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 163, in run_commands
    dist.run_commands()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 967, in run_commands
    self.run_command(cmd)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 986, in run_command
    cmd_obj.run()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/setuptools/_distutils/command/build.py", line 135, in run
    self.run_command(cmd_name)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/setuptools/_distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 986, in run_command
    cmd_obj.run()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/setuptools/_distutils/command/build_ext.py", line 339, in run
    self.build_extensions()
  File "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_python39/python39/work/Python-3.9.9/./setup.py", line 456, in build_extensions
    build_ext.build_extensions(self)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/setuptools/_distutils/command/build_ext.py", line 446, in build_extensions
    self._build_extensions_parallel()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/setuptools/_distutils/command/build_ext.py", line 468, in _build_extensions_parallel
    fut.result()
  File "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_python39/python39/work/Python-3.9.9/Lib/concurrent/futures/_base.py", line 445, in result
    return self.__get_result()
  File "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_python39/python39/work/Python-3.9.9/Lib/concurrent/futures/_base.py", line 390, in __get_result
    raise self._exception
  File "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_python39/python39/work/Python-3.9.9/Lib/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_python39/python39/work/Python-3.9.9/./setup.py", line 554, in build_extension
    build_ext.build_extension(self, ext)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/setuptools/_distutils/command/build_ext.py", line 528, in build_extension
    objects = self.compiler.compile(sources,
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/setuptools/_distutils/ccompiler.py", line 574, in compile
    self._compile(obj, src, ext, cc_args, extra_postargs, pp_opts)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/setuptools/_distutils/unixccompiler.py", line 117, in _compile
    self.spawn(compiler_so + cc_args + [src, '-o', obj] +
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/setuptools/_distutils/ccompiler.py", line 917, in spawn
    spawn(cmd, dry_run=self.dry_run, **kwargs)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/setuptools/_distutils/spawn.py", line 38, in spawn
    log.info(subprocess.list2cmdline(cmd))
AttributeError: module '_bootsubprocess' has no attribute 'list2cmdline'
make: *** [sharedmods] Error 1
make: Leaving directory `/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_python39/python39/work/Python-3.9.9'
Command failed:  cd "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_python39/python39/work/Python-3.9.9" && /usr/bin/make -j2 -w all

Code of Conduct

• I agree to follow the PSF Code of Conduct

[jaraco]

I'm not sure what should happen here. Setuptools could potentially add another exclusion to the hack if there's a reliable signal to detect building CPython. Or maybe users should simply not add Setuptools to the site packages while building CPython. What's the motivation for having Setuptools present at this stage?

[jmroot]

There's no particular reason to want to be in this situation, it just happens sometimes when building python, usually a new version, after already having installed python and setuptools.

If the issue can be worked around in setuptools without too much effort, great; otherwise I guess it's sufficient to just document it somewhere. Presumably it will eventually go away when distutils is dropped from the stdlib.

[jaraco]

I took a look at the traceback above, but by the time import distutils is encountered in the setup.py, it's too late, so there's very little signal for Setuptools to detect that it's Python. It could theoretically detect that 'setup.py' is in the call stack and open that file and parse it for setup(name="Python"), but that would be expensive and would have to be done anytime any project imports distutils. I'd say it's too messy to try that.

It's possible there is other context in the environment if for example, the Makefile sets some environment variable like PYTHON_IS_BUILDING. Setuptools could honor that. I scanned through the makefile and didn't find anything that looked relevant.

In which case, "document it somewhere" seems like a good approach, and from my perspective, this report provides that documentation (anyone searching for that error message will find this bug). If you can imagine another place where documentation would be more visible, we could consider that.

In the meantime, I'll close this issue, and thanks for raising it.

[jaraco]

@tiran had this to contribute:

If there is a pybuilddir.txt next to setup.py then it's a CPython Unix build. All the other factors are not reliable.

[tiran]

It's possible there is other context in the environment if for example, the Makefile sets some environment variable like PYTHON_IS_BUILDING. Setuptools could honor that. I scanned through the makefile and didn't find anything that looked relevant.

Any kind of workaround of CPython's setup.py will only land in future releases of CPython 3.9 and newer. Python 3.6 and earlier won't get any new releases because they are EOL. 3.7 and 3.8 are in security-only mode.

[tiran]

https://bugs.python.org/issue46401 is another case of the same issue.

[vstinner]

Is it possible that the bug is back in setuptools 60.9.3? Please have a look at this Python regression: https://bugs.python.org/issue47013

[jaraco]

Is it possible that the bug is back in setuptools 60.9.3? Please have a look at this Python regression: https://bugs.python.org/issue47013

In that issue, CPython is attempting to update from Setuptools 58.x. Setuptools 60 introduced the drastic shift to supplying distutils by default. So it’s probably safe to say that the approach applied above didn’t have the intended effect.

Setuptools detects the presence of ./pybuilddir.txt to disable the distutils replacement. The “installed” tests probably don’t have this characteristic. In those environments, my recommendation is to set SETUPTOOLS_USE_DISTUTILS=stdlib when running tests.

[tiran]

SETUPTOOLS_USE_DISTUTILS is not an option. We cannot reliable inject the env var into our tests harness. The problem has to be solved in setuptools.

[nre-ableton]

I ran into this behavior recently while trying to build Python 3.10.4 on Ubuntu 22.04, which resulted in a confusing error, ModuleNotFoundError: No module named 'binascii'. Thankfully exporting SETUPTOOLS_USE_DISTUTILS=stdlib worked, but I agree it's not an ideal workaround.

This issue was closed, having been fixed by #3031. I'm a bit confused as to whether the issue is actually fixed or not, though.

[jaraco]

I'm a bit confused as to whether the issue is actually fixed or not, though.

The issue reported here was addressed (stdlib builds do work for the most part). It appears, however, that there are some select cases where the CPython build detection isn't taking effect, so continues to lead to trouble.

Let's continue the conversation in python/cpython#91169, which is the open issue describing the defect.