Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

make sharedinstall does not return failure if install commands fail #100220

Closed
mgorny opened this issue Dec 13, 2022 · 2 comments
Closed

make sharedinstall does not return failure if install commands fail #100220

mgorny opened this issue Dec 13, 2022 · 2 comments
Labels
build The build process and cross-build type-bug An unexpected behavior, bug, or error

Comments

@mgorny
Copy link
Contributor

mgorny commented Dec 13, 2022

Bug report

If make sharedinstall fails to install some Python extensions, the make target wrongly succeeds. For example, I'm seeing:

/usr/bin/install -c -m 755 Modules/array.cpython-312-x86_64-linux-gnu.so /usr/lib/python3.12/lib-dynload/array.cpython-312-x86_64-linux-gnu.so
/usr/bin/install: cannot create regular file '/var/tmp/portage/dev-lang/python-3.12.0_alpha3/image/usr/lib/python3.12/lib-dynload/array.cpython-312-x86_64-linux-gnu.so': No such file or directory
/usr/bin/install -c -m 755 Modules/_asyncio.cpython-312-x86_64-linux-gnu.so /usr/lib/python3.12/lib-dynload/_asyncio.cpython-312-x86_64-linux-gnu.so
/usr/bin/install: cannot create regular file '/var/tmp/portage/dev-lang/python-3.12.0_alpha3/image/usr/lib/python3.12/lib-dynload/_asyncio.cpython-312-x86_64-linux-gnu.so': No such file or directory
/usr/bin/install -c -m 755 Modules/_bisect.cpython-312-x86_64-linux-gnu.so /usr/lib/python3.12/lib-dynload/_bisect.cpython-312-x86_64-linux-gnu.so
[...]

Nevertheless, make install returns successfully in this case. This causes major problems for automated builds since they end up with broken Python installs when the make target should have failed.

I need to investigate why it's failing but that's a separate issue.

Complete build log (1.2M): dev-lang:python-3.12.0_alpha3:20221207-142002.log

The problem seems to be that the sharedinstall targets runs a single shell command and make doesn't check the exit status until its very end.

I suspect the same problem may apply to other install rules.

Your environment

  • CPython versions tested on: 3.12.0a3
  • Operating system and architecture: Gentoo Linux/amd64

Linked PRs

@mgorny mgorny added the type-bug An unexpected behavior, bug, or error label Dec 13, 2022
@mgorny mgorny changed the title make sharedinstall no longer returns failure if install commands fail make sharedinstall does not retrn failure if install commands fail Dec 13, 2022
@mgorny mgorny changed the title make sharedinstall does not retrn failure if install commands fail make sharedinstall does not return failure if install commands fail Dec 13, 2022
mgorny added a commit to mgorny/cpython that referenced this issue Dec 18, 2022
Use `set -e` before compound shell commands in order to ensure that make
targets fail correctly when at least one of the subcommands fail.

This is necessary since make considers a target failed only if one of
the shell invocations returns with unsuccessful exit status.  If a shell
script does not exit explicitly, the shell uses the exit status
of the *last* executed command.  This means that when multiple commands
are executed (e.g. through a `for` loop), the exit statuses of prior
command invocations are ignored.

This can be either resolved by adding an explicit `|| exit 1` to every
command that is expected to succeed, or by running the whole script
with `set -e`.  The latter was used here as it the rules seem to be
written with the assumption that individual commands were supposed
to cause the make rules to fail.
@erlend-aasland erlend-aasland added the build The build process and cross-build label Jan 11, 2023
mgorny added a commit to mgorny/cpython that referenced this issue Feb 9, 2023
Use `set -e` before compound shell commands in order to ensure that make
targets fail correctly when at least one of the subcommands fail.

This is necessary since make considers a target failed only if one of
the shell invocations returns with unsuccessful exit status.  If a shell
script does not exit explicitly, the shell uses the exit status
of the *last* executed command.  This means that when multiple commands
are executed (e.g. through a `for` loop), the exit statuses of prior
command invocations are ignored.

This can be either resolved by adding an explicit `|| exit 1` to every
command that is expected to succeed, or by running the whole script
with `set -e`.  The latter was used here as it the rules seem to be
written with the assumption that individual commands were supposed
to cause the make rules to fail.
zware added a commit that referenced this issue Apr 7, 2023
Set `SHELL = /bin/sh -e` to ensure that complex recipes fail on the first error rather than incorrectly reporting success.

Co-authored-by: Zachary Ware <zach@python.org>
@zware
Copy link
Member

zware commented Apr 7, 2023

GH-100328 merged, thanks for the report and patch!

I think we'll pass on backporting this; there haven't been reported problems in this vein before (though the recipes haven't changed much, so the underlying issue does exist in older branches), and there could be some lesser-used recipes that shouldn't use -e that we missed.

@zware zware closed this as completed Apr 7, 2023
@mgorny
Copy link
Contributor Author

mgorny commented Apr 8, 2023

Yeah, I suppose the risk is relatively small, as it affects only a few less frequently used rules.

warsaw pushed a commit to warsaw/cpython that referenced this issue Apr 11, 2023
Set `SHELL = /bin/sh -e` to ensure that complex recipes fail on the first error rather than incorrectly reporting success.

Co-authored-by: Zachary Ware <zach@python.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
build The build process and cross-build type-bug An unexpected behavior, bug, or error
Projects
None yet
Development

No branches or pull requests

3 participants