distutils.command.build_ext raises FileNotFoundError

[jaraco]

BPO	41207
Nosy	@jaraco, @vstinner, @merwok, @ambv, @dstufft, @miss-islington
PRs	bpo-41207 In distutils.spawn, rewrite FileNotFound #21359 [3.9] bpo-41207 In distutils.spawn, rewrite FileNotFound (GH-21359) #21373

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/ambv'
closed_at = <Date 2020-07-07.11:58:53.621>
created_at = <Date 2020-07-04.01:25:52.705>
labels = ['3.10', 'library', '3.9', 'type-crash']
title = 'distutils.command.build_ext raises FileNotFoundError'
updated_at = <Date 2020-07-07.11:58:53.620>
user = 'https://github.com/jaraco'

bugs.python.org fields:

activity = <Date 2020-07-07.11:58:53.620>
actor = 'vstinner'
assignee = 'lukasz.langa'
closed = True
closed_date = <Date 2020-07-07.11:58:53.621>
closer = 'vstinner'
components = ['Distutils']
creation = <Date 2020-07-04.01:25:52.705>
creator = 'jaraco'
dependencies = []
files = []
hgrepos = []
issue_num = 41207
keywords = ['patch', '3.9regression']
message_count = 10.0
messages = ['372973', '373045', '373047', '373100', '373104', '373170', '373172', '373218', '373219', '373220']
nosy_count = 6.0
nosy_names = ['jaraco', 'vstinner', 'eric.araujo', 'lukasz.langa', 'dstufft', 'miss-islington']
pr_nums = ['21359', '21373']
priority = None
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'crash'
url = 'https://bugs.python.org/issue41207'
versions = ['Python 3.9', 'Python 3.10']

[jaraco]

In pypa/setuptools#2228, by adopting the distutils codebase from a late release of CPython, Setuptools stumbled onto an API-breaking change in distutils rooted at bpo-39763.

Details are in the Setuptools investigation, but to summarize:

• distutils.ccompiler.CCompiler.compile declares "raises CompileError on failure" and calls self._compile, implemented by subclasses.
• In at least distutils.unixcompiler.UnixCCompiler._compile, distutils.spawn.spawn is called (through CCompiler.spawn).
• Since References to non-existant "StringIO" module #62943, distutils.spawn.spawn calls subprocess.Popen which raises FileNotFoundError when the target executable doesn't exist.
• Programs trapping CompileError but not FileNotFoundError will crash where once they had error handling.

Setuptools discovered this behavior in the 48.0 release when it incorporated these distutils changes into a vendored release of Setuptools, but the failures exhibited will apply to all builds (including pyyaml) on Python 3.9.

[jaraco]

In pypa/distutils@7aa5abeafc, I've pushed a fix (with a repro test in the parent commit). I recommend this fix be applied to CPython 3.9.

[jaraco]

CPython should also consider this change, which unifies the DEBUG handling, consolidates the exception trapping, and uses subprocess.check_call to re-use exit code checking.

[ambv]

Good catch! The unification change is 3.10 only but I'm totally fine accepting the _compile fix in Python 3.9.

[vstinner]

Programs trapping CompileError but not FileNotFoundError will crash where once they had error handling.

Do you mean mean that spawn() of distutils.spawn should catch FileNotFoundError and raise a CompilerError instead?

It seems like before my commit 1ec63b6, any OSError was catched and replaced with DistutilsExecError, not only FileNotFoundError.

Do you want to propose a fix?

[jaraco]

Sure. I'll submit patches.

[jaraco]

I learned the magical incantation to port commits from pypa/distutils to CPython:

cpython bugfix/41207-rewrite-filenotfound $ git -C ~/p/pypa/distutils format-patch HEAD~2 --stdout | git am --directory Lib                                
Applying: Add test for spawn when exe is missing. Ref pypa/distutils#3.
Applying: Replace OSError with DistutilsExecError. Fixes pypa/distutils#3 and pypa/setuptools#2228 and bpo-41207.

[miss-islington]

New changeset 6ae2780 by Jason R. Coombs in branch 'master':
bpo-41207 In distutils.spawn, rewrite FileNotFound (GH-21359)
6ae2780

[miss-islington]

New changeset 2c82628 by Miss Islington (bot) in branch '3.9':
bpo-41207 In distutils.spawn, rewrite FileNotFound (GH-21359)
2c82628

[vstinner]

I close the issue since it's fixed. I also reset the priority.