Bootloader: forward signals to bundled application #3515

brettdh · 2018-05-15T20:15:28Z

Forward more signals

All the signals, to be precise. (Well, all the signals that can be handled.)

If the caller of pyinstaller specifies the --bootloader-ignore-signals CLI flag, all signals will be ignored instead. This is useful if e.g. a supervisor process signals the bootloader's process group; otherwise, the child process would be signaled twice.

Note: docs forthcoming for the new flag (and perhaps this overall behavior), pending feedback on the overall approach.

New functional test

This test checks whether signals sent to an pyinstaller-built executable can be handled
using signal.signal(...) in the Python code.

Summary of my manual testing so far (before this test):

Linux
- Fails in --onedir and --onefile modes.
macOS
- Fails only in --onefile mode
Windows
- Not tested - and I suspect I should skip this test on Windows, since signals aren't the same there.
- (Also, I don't really have any idea how signals, or whatever semi-equivalent, work on Windows.)

At present, I'm assuming that an application using pyinstaller could reasonably want to handle any handleable signal (i.e. not SIGKILL or SIGSTOP). If there are any exceptions to this, or if this needs to be handled more selectively for reasons I'm not aware of, I'd be glad to hear about it.

Closes #3514.

brettdh · 2018-05-16T18:59:29Z

CI build is looking good so far. I'm optimistically declaring this ready for review.

brettdh · 2018-05-17T14:18:35Z

Added @skipif_win for the signal tests, 'cause I have no idea how to handle the Windows case, and I kinda doubt that it's common to do much with signals and PyInstaller on Windows.

Other Appveyor errors seem unrelated to this PR.

brettdh · 2018-05-24T15:45:25Z

There's a subtle problem with this PR: messing with SIGCHLD makes the boot loader unable to wait() for the child and capture its exit status. Since the exit code of wait() is not checked, this makes it appear that the child always exits 0.

I have a fix in flight; will hopefully push soon, along with a related piece of functionality that I think is required here (the ability to ignore signals in the boot loader if desired).

htgoebel · 2018-05-26T16:12:18Z

In the meanwhile: thanks for working on this :-) I'm expecting our updates.

brettdh · 2018-05-26T16:26:23Z

I'm currently stuck on this issue: I have a test that breaks with py.test -n4 (or -n1) but succeeds without it. I'm wondering if pytest-xdist or execnet is getting thrown off by the boot loader ignoring all signals. Does that happen to raise any red flags as to what might be going wrong?

I've just pushed the branch in its current broken state to show what I mean. Suggestions welcome.

brettdh · 2018-05-26T16:30:11Z

If the failure in Travis looks like my local testing, it'll show that the call to PI_Py_Finalize appears to crash the child, as there's no output after that. Also, the parent exits with a negative integer value, indicating that the signal was not ignored as it should have been.

But, I've also been seeing strange results where debug prints that I insert in the parent don't show up in the output, even though the child process is still forked and runs the test. So there might be something wonky with my test environment. I'm very curious to see what the same test in Travis looks like.

brettdh · 2018-05-26T16:38:50Z

...and of course, since I wasn't thinking and pushed commits to another PR before this one, it's going to be a while before this test runs. 🤦‍♂️

brettdh · 2018-05-26T16:40:30Z

Lol and as I write that, the build kicked off after all. Pleasantly surprised.

brettdh · 2018-05-26T17:48:16Z

Ok, so - the failures I see locally don’t show up in Travis. I need to fix the SIGCLD case (treat it like SIGCHLD), but otherwise I guess this is almost done.

Figuring out the issue with my local test env can be a part of #3519.

brettdh · 2018-05-26T19:10:18Z

@htgoebel this should be good for review; I just pushed the SIGCLD fix.

brettdh · 2018-06-05T20:56:23Z

Rebased, resolved conflicts, squashed a bit.

brettdh · 2018-06-14T16:56:33Z

@htgoebel any news on this? Anything else you need me to do?

htgoebel

Thanks a lot for this pull-request. Sorry for not answering earlier, I'm busy with my day-job.

This looks good for me and I like the test-suite you've created. Nevertheless I added some comments, most of which are nit-picking to keep the code maintainable for another 10 years.

htgoebel · 2018-07-06T13:34:19Z

PyInstaller/building/api.py

@@ -56,7 +56,7 @@ def __init__(self, *tocs, **kwargs):
                ModuleGraph.

        kwargs
-            Possible keywork arguments:
+            Possible keyword arguments:


This is unrelated to this change and should go into a commit of its own. Same for typo fixed below. I'd even abstain from fixing then, since it is still understandable.

htgoebel · 2018-07-06T17:01:09Z

tests/functional/test_signals.py

+
+            target = child
+
+        eprint('signalling {{}} ({{}})'.format(target.name(), target.pid))


By using %s interpolation, this becomes easer to read. Same for all other cases below. and above.

Also: Why not passing several args to eprint:
``
eprint('signaling', target.name(), '(%s)' % target.pid)

This is why print is designed as it is :-) And you can get rid of the formatting at all in many cases.

htgoebel · 2018-07-06T17:01:11Z

tests/functional/test_signals.py

+                print('Bootloader did not fork; test is invalid')
+                sys.exit(0)
+
+            target = child


This trailing target = child looks lost. What about this structure:

if parent.name ... ... elif ignore: ... else: ...

htgoebel · 2018-07-06T17:01:14Z

tests/functional/test_signals.py

+    if signame in ['SIGCHLD', 'SIGCLD']:
+        pytest.xfail(
+            'Messing with {} interferes with bootloader'.format(signame)
+        )


Please add a short comment. I assume these signals are not filtered out when building the signals list to spot whether the bootloader actually does not forward these to the child?!

htgoebel · 2018-07-06T17:01:18Z

bootloader/src/pyi_utils.c

+    VS("LOADER: Restoring signal handlers\n");
+    for (signum = 0; signum < num_signals; ++signum) {
+        if (handlers[signum] != SIG_ERR) {
+            signal(signum, handlers[signum]);


Not setting the signal back to SIG_DFL (as the old code did) is worth a commit of its own. Thus when reading the commit messages one can easily spot this relevant change.

@htgoebel It's pretty insignificant, IMO, particularly since the bootloader (parent) is about to exit after this. I'd be more inclined to switch back to using SIG_DFL.

htgoebel · 2018-07-06T17:01:35Z

bootloader/src/pyi_utils.c

+
+    #ifndef sighandler_t
+    typedef void (*sighandler_t)(int);
+    #endif


Please move this near the top of the file, where other type-defs are.

htgoebel · 2018-07-06T17:01:37Z

bootloader/src/pyi_utils.c

 {
-    kill(child_pid, signal);
+    VS("LOADER: Forwarding signal %d to child pid %d\n", signum, child_pid);
+    kill(child_pid, signum);


Renaming parameters is a non-functional change, please revert this. This will increase the patch-sice and the time to review and understand what this change is doing.

The previous name, signal, was shadowing a function name. Like you, I'm interested in the long-term maintainability of the code I write, so I fixed this issue. I've already split this tiny function into two handlers; even calling this a rename is a stretch.

htgoebel · 2018-07-06T17:01:40Z

bootloader/src/pyi_utils.c

@@ -850,21 +855,44 @@ set_systemd_env()
 pid_t child_pid = 0;

 static void
-_signal_handler(int signal)
+_ignore_signal(int signum)


Rename to something telling this is a handler, too. E.g. ignoring_signal_handler. Also a comment would be nice.

What would the comment say that isn't already obvious from the function name and log message?

htgoebel · 2018-07-06T17:01:44Z

bootloader/src/pyi_archive.c

@@ -553,7 +553,7 @@ pyi_arch_status_free_memory(ARCHIVE_STATUS *archive_status)
 * for freeing it.
 */
 char *
-pyi_arch_get_option(ARCHIVE_STATUS * status, char * optname)
+pyi_arch_get_option(const ARCHIVE_STATUS * status, char * optname)


These changes (adding "const") are unrelated to this pull-request and should go into a commit of its own.

In fact, these were required, because I added an argument of type const ARCHIVE_STATUS * to pyi_utils_create_child below.

I can either make these existing methods const-correct or I can make the new argument const-incorrect. I chose the option that I felt was a net improvement. Please let me know if you disagree.

htgoebel · 2018-07-06T17:02:34Z

PyInstaller/building/makespec.py

+                         "Useful in situations where e.g. a supervisor "
+                         "process signals both the bootloader and child "
+                         "(e.g. via a process group) to avoid signalling "
+                         "the child twice."))


What do you think about moving this below --noupx, or even into the "Rarely used special options:" section?

Sure; though I'm not really sure how common this use case is, I lean towards the latter.

brettdh · 2018-07-10T17:23:46Z

Addressed feedback and pushed new version.

brettdh changed the title ~~WIP: Bootloader: forward signals to~~ WIP: Bootloader: forward signals to bundled application May 15, 2018

brettdh force-pushed the develop branch from 737de93 to 28ba8df Compare May 16, 2018 18:38

brettdh changed the title ~~WIP: Bootloader: forward signals to bundled application~~ Bootloader: forward signals to bundled application May 16, 2018

brettdh force-pushed the develop branch from 28ba8df to 5e37595 Compare May 17, 2018 13:59

brettdh force-pushed the develop branch from ca85ca7 to 6b0e74e Compare June 5, 2018 19:15

htgoebel added the area:bootloader Caused be or effecting the bootloader label Jun 27, 2018

htgoebel self-requested a review June 27, 2018 10:30

htgoebel added this to the PyInstaller 3.4 milestone Jun 27, 2018

htgoebel suggested changes Jul 6, 2018

View reviewed changes

brettdh added 2 commits July 10, 2018 13:22

Tests: Add functional test for signal handling

842a13a

Bootloader: forward all signals to child process

2ddb80d

brettdh force-pushed the develop branch from 6b0e74e to 2ddb80d Compare July 10, 2018 17:23

htgoebel mentioned this pull request Jul 28, 2018

KeyboardInterrupt input #3646

Open

htgoebel added 3 commits August 16, 2018 11:37

Fixup tests: Remove .format() when using wprint

1aebd5f

Merge branch 'develop' into develop

15006f2

Fixup syntax error.

07612c8

htgoebel merged commit a1ecdfb into pyinstaller:develop Aug 25, 2018

htgoebel mentioned this pull request Aug 28, 2018

patch: Interactive frozen script goes to background on SIGINT #208

Closed

CSammy mentioned this pull request Oct 28, 2018

[1.7] CTRL+C fails to stop containers when running compose in the foreground docker/compose#3347

Closed

half1red mentioned this pull request Feb 7, 2019

Issue with using signal 20 in terminal (ctrl-z) #4057

Closed

alchem1ster mentioned this pull request Jan 17, 2022

Nuitka does not let os signals pass to python code Nuitka/Nuitka#1383

Closed

hinqiwame mentioned this pull request Jul 14, 2022

[BUG] pls suppress this hinqiwame/rice-clock#1

Closed

github-actions bot locked as resolved and limited conversation to collaborators Nov 16, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bootloader: forward signals to bundled application #3515

Bootloader: forward signals to bundled application #3515

brettdh commented May 15, 2018 •

edited

brettdh commented May 16, 2018

brettdh commented May 17, 2018

brettdh commented May 24, 2018

htgoebel commented May 26, 2018

brettdh commented May 26, 2018

brettdh commented May 26, 2018

brettdh commented May 26, 2018

brettdh commented May 26, 2018

brettdh commented May 26, 2018

brettdh commented May 26, 2018

brettdh commented Jun 5, 2018

brettdh commented Jun 14, 2018

htgoebel left a comment

htgoebel Jul 6, 2018

htgoebel Jul 6, 2018

htgoebel Jul 6, 2018

htgoebel Jul 6, 2018

htgoebel Jul 6, 2018

brettdh Jul 10, 2018

htgoebel Jul 6, 2018

htgoebel Jul 6, 2018

brettdh Jul 10, 2018

htgoebel Jul 6, 2018

brettdh Jul 10, 2018

htgoebel Jul 6, 2018

brettdh Jul 10, 2018

htgoebel Jul 6, 2018

brettdh Jul 10, 2018

brettdh commented Jul 10, 2018


		target = child

		eprint('signalling {{}} ({{}})'.format(target.name(), target.pid))

Bootloader: forward signals to bundled application #3515

Bootloader: forward signals to bundled application #3515

Conversation

brettdh commented May 15, 2018 • edited

Forward more signals

New functional test

brettdh commented May 16, 2018

brettdh commented May 17, 2018

brettdh commented May 24, 2018

htgoebel commented May 26, 2018

brettdh commented May 26, 2018

brettdh commented May 26, 2018

brettdh commented May 26, 2018

brettdh commented May 26, 2018

brettdh commented May 26, 2018

brettdh commented May 26, 2018

brettdh commented Jun 5, 2018

brettdh commented Jun 14, 2018

htgoebel left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

brettdh commented Jul 10, 2018

brettdh commented May 15, 2018 •

edited