-
-
Notifications
You must be signed in to change notification settings - Fork 30.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
sigaddset() can fail on some signal numbers #77510
Comments
Originally reported in https://bugs.python.org/issue31463 This started to bother us in Fedora rawhide for various Python versions, so chances are something changed on the system level. # python3.7 -m test.regrtest test_multiprocessing_fork
Run tests sequentially
0:00:00 load avg: 1.24 [1/1] test_multiprocessing_fork
/usr/lib64/python3.7/multiprocessing/semaphore_tracker.py:55: UserWarning: semaphore_tracker: process died unexpectedly, relaunching. Some semaphores might leak.
warnings.warn('semaphore_tracker: process died unexpectedly, '
Exception in thread Thread-26:
Traceback (most recent call last):
File "/usr/lib64/python3.7/threading.py", line 917, in _bootstrap_inner
self.run()
File "/usr/lib64/python3.7/threading.py", line 865, in run
self._target(*self._args, **self._kwargs)
File "/usr/lib64/python3.7/multiprocessing/resource_sharer.py", line 139, in _serve
signal.pthread_sigmask(signal.SIG_BLOCK, range(1, signal.NSIG))
File "/usr/lib64/python3.7/signal.py", line 60, in pthread_sigmask
sigs_set = _signal.pthread_sigmask(how, mask)
ValueError: signal number 32 out of range (hangs...) ^CProcess Process-184:
Traceback (most recent call last):
File "/usr/lib64/python3.7/multiprocessing/process.py", line 297, in _bootstrap
self.run()
File "/usr/lib64/python3.7/multiprocessing/process.py", line 99, in run
self._target(*self._args, **self._kwargs)
File "/usr/lib64/python3.7/test/_test_multiprocessing.py", line 3328, in child_access
w = conn.recv()
File "/usr/lib64/python3.7/multiprocessing/connection.py", line 251, in recv
return _ForkingPickler.loads(buf.getbuffer())
File "/usr/lib64/python3.7/multiprocessing/connection.py", line 960, in rebuild_connection
fd = df.detach()
File "/usr/lib64/python3.7/multiprocessing/resource_sharer.py", line 57, in detach
with _resource_sharer.get_connection(self._id) as conn:
File "/usr/lib64/python3.7/multiprocessing/resource_sharer.py", line 87, in get_connection
c = Client(address, authkey=process.current_process().authkey)
File "/usr/lib64/python3.7/multiprocessing/connection.py", line 498, in Client
answer_challenge(c, authkey)
File "/usr/lib64/python3.7/multiprocessing/connection.py", line 741, in answer_challenge
message = connection.recv_bytes(256) # reject large message
File "/usr/lib64/python3.7/multiprocessing/connection.py", line 216, in recv_bytes
buf = self._recv_bytes(maxlength)
File "/usr/lib64/python3.7/multiprocessing/connection.py", line 407, in _recv_bytes
buf = self._recv(4)
File "/usr/lib64/python3.7/multiprocessing/connection.py", line 379, in _recv
chunk = read(handle, remaining)
KeyboardInterrupt
Warning -- multiprocessing.process._dangling was modified by test_multiprocessing_fork
Before: <_weakrefset.WeakSet object at 0x7f2abd5f6d30>
After: <_weakrefset.WeakSet object at 0x7f2abd5f6128>
Warning -- threading._dangling was modified by test_multiprocessing_fork
Before: <_weakrefset.WeakSet object at 0x7f2abd5f66d8>
After: <_weakrefset.WeakSet object at 0x7f2abc835048> Test suite interrupted by signal SIGINT. Total duration: 3 min 43 sec -------------------------------- # python3.7 -m test.regrtest test_multiprocessing_forkserver
Run tests sequentially
0:00:00 load avg: 1.00 [1/1] test_multiprocessing_forkserver
Exception in thread Thread-26:
Traceback (most recent call last):
File "/usr/lib64/python3.7/threading.py", line 917, in _bootstrap_inner
self.run()
File "/usr/lib64/python3.7/threading.py", line 865, in run
self._target(*self._args, **self._kwargs)
File "/usr/lib64/python3.7/multiprocessing/resource_sharer.py", line 139, in _serve
signal.pthread_sigmask(signal.SIG_BLOCK, range(1, signal.NSIG))
File "/usr/lib64/python3.7/signal.py", line 60, in pthread_sigmask
sigs_set = _signal.pthread_sigmask(how, mask)
ValueError: signal number 32 out of range (hangs...) ^CProcess Process-184:
Traceback (most recent call last):
...
KeyboardInterrupt
Warning -- multiprocessing.process._dangling was modified by test_multiprocessing_forkserver
Before: <_weakrefset.WeakSet object at 0x7f0323ecde10>
After: <_weakrefset.WeakSet object at 0x7f0323ecd208>
Warning -- threading._dangling was modified by test_multiprocessing_forkserver
Before: <_weakrefset.WeakSet object at 0x7f0323ecd7b8>
After: <_weakrefset.WeakSet object at 0x7f0323ecddd8> Test suite interrupted by signal SIGINT. Total duration: 55 sec -------------------------------- # python3.7 -m test.regrtest test_multiprocessing_spawn
Run tests sequentially
0:00:00 load avg: 1.49 [1/1] test_multiprocessing_spawn
Exception in thread Thread-26:
Traceback (most recent call last):
File "/usr/lib64/python3.7/threading.py", line 917, in _bootstrap_inner
self.run()
File "/usr/lib64/python3.7/threading.py", line 865, in run
self._target(*self._args, **self._kwargs)
File "/usr/lib64/python3.7/multiprocessing/resource_sharer.py", line 139, in _serve
signal.pthread_sigmask(signal.SIG_BLOCK, range(1, signal.NSIG))
File "/usr/lib64/python3.7/signal.py", line 60, in pthread_sigmask
sigs_set = _signal.pthread_sigmask(how, mask)
ValueError: signal number 32 out of range (hangs...) ^CProcess Process-184:
Traceback (most recent call last):
...
KeyboardInterrupt
Warning -- multiprocessing.process._dangling was modified by test_multiprocessing_spawn
Before: <_weakrefset.WeakSet object at 0x7fb2134a9dd8>
After: <_weakrefset.WeakSet object at 0x7fb2134a91d0>
Warning -- threading._dangling was modified by test_multiprocessing_spawn
Before: <_weakrefset.WeakSet object at 0x7fb2134a9780>
After: <_weakrefset.WeakSet object at 0x7fb2134a9da0> Test suite interrupted by signal SIGINT. Total duration: 54 sec -------------------------------- Happens with all 3 tests on 3.4 to 3.7. To test in docker, one can do: $ docker run -ti fedora:rawhide /bin/bash
# dnf update
# dnf install python37 # or python3-test for 3.6, or python35, python34 Note that without |
Perhaps you can compile and run this C program, before and after the system changes, and post the output: #include <errno.h>
#include <signal.h>
#include <stdio.h>
int main(int argc, char** argv)
{
sigset_t set;
int i, ret;
printf("NSIG = %d\n", NSIG);
for (i = 1; i < NSIG; i++) {
errno = 0;
ret = sigaddset(&set, i);
printf("sigaddset(%d) returned %d, errno =%d\n", i, ret, errno);
}
return 0;
} (on Ubuntu 16.04, I get NSIG = 65, and all signal numbers work fine) |
I'll try. In the meantime, I've checked and it's glibc update that makes the difference: glibc-2.27.9000-7.fc29 -> glibc-2.27.9000-14.fc29 I'll see what was changed there and whether is was intentional, or bug in Fedora. |
I was able to compile the program above after the system upgrade using your Docker image, and the following stands out: sigaddset(31) returned 0, errno = 0 However, I do not know how to compile it *before* the system upgrade ("dnf install gcc" seems to install gcc). |
glibc-2.27.9000-14.fc29: NSIG = 65
sigaddset(32) returned -1, errno =22
sigaddset(33) returned -1, errno =22 |
"dnf install gcc" upgrades glibc, yes. OK, I'll dig into it a bit more, see why and where this change happened. Thanks for hints. |
dnf install https://kojipkgs.fedoraproject.org/packages/glibc/2.27.9000/7.fc29/x86_64/glibc-headers-2.27.9000-7.fc29.x86_64.rpm https://kojipkgs.fedoraproject.org/packages/glibc/2.27.9000/7.fc29/x86_64/glibc-devel-2.27.9000-7.fc29.x86_64.rpm NSIG = 65
sigaddset(1) returned 0, errno =0
... all ok |
glibc-2.27.9000-13.fc29 ... all ok 13 to 14 is this in upstream commits: d39c0a459ef32a41daac4840859bf304d931adab to 583a27d525ae189bdfaa6784021b92a9a1dae12e |
Ok, this is due to: Short explanation here: |
Miro, could you check whether 2.7 is affected too? |
I cannot find a corresponding test in 2.7. Running the entire build (incl. tests) to see what happens. Is there a command or a couple I could try instead? I'm afraid I don't understand how does that test work (I seems a bit complicated) |
Running the whole test suite sounds fine. |
Python 2 testsuite runs fine. |
Setting as release blocker as it impacts functionality of the multiprocessing module. |
Should the fix be to exclude 32 and 33 from multiprocessing.resource_sharer:_serve? if hasattr(signal, 'pthread_sigmask'):
signal.pthread_sigmask(signal.SIG_BLOCK, range(1, signal.NSIG)) |
We could do that, but I'd rather something not multiprocessing-specific. I think we have to ignore the return value of sigaddset() until bpo-33332 allows us to be smarter. |
... and also not Linux-specific. |
Miro, can you give the following patch a try? diff --git a/Modules/signalmodule.c b/Modules/signalmodule.c
index 7916160..ca76a20 100644
--- a/Modules/signalmodule.c
+++ b/Modules/signalmodule.c
@@ -819,7 +819,6 @@ iterable_to_sigset(PyObject *iterable, sigset_t *mask)
int result = -1;
PyObject *iterator, *item;
long signum;
- int err;
sigemptyset(mask);
@@ -841,11 +840,10 @@ iterable_to_sigset(PyObject *iterable, sigset_t *mask)
Py_DECREF(item);
if (signum == -1 && PyErr_Occurred())
goto error;
- if (0 < signum && signum < NSIG)
- err = sigaddset(mask, (int)signum);
- else
- err = 1;
- if (err) {
+ if (0 < signum && signum < NSIG) {
+ (void) sigaddset(mask, (int)signum);
+ }
+ else {
PyErr_Format(PyExc_ValueError,
"signal number %ld out of range", signum);
goto error; |
This indeed makes the tests pass again: # python3.7 -m test.regrtest test_multiprocessing_fork test_multiprocessing_forkserver test_multiprocessing_spawn Total duration: 3 min 40 sec |
Why not export and use the canonical way of sigemptyset/sigfillset/sigaddset/sigdelset/sigismember instead of pushing for more potential non-conformant code? For glibc sigfillset will correctly fill all the signal set structure while removing the internal used signals. This is par what POSIX specifies [1] where it states that: 'either sigemptyset() or sigfillset() must be called prior to any other use of the signal set' And more importantly: 'For example, blocking or ignoring an implementation-defined signal may have undesirable side-effects, whereas the default action for that signal is harmless. In such a case, it would be preferable for such a signal to be excluded from the signal set returned by sigfillset().' Also keep that since is an implementation detail, different libcs can use different internal signals. UCLIBC, for instance, uses the same 2 signals as GLIBC, however MUSL allocates signal 32, 33, and 34 for internal usage (and excludes in sigfillset and warns with EINVAL on sigaddset). |
Also keep in mind that POSIX [1] specifies that sigaddset *may* fail with EINVAL for not support signals, so a conforming implementation may not fail on sigaddset and still remove the internal signal in a sigprocmask (uclibc for instance). |
I agree this is the proper fix and that's what I plan to do in Python 3.8. For Python 3.7 and earlier, though, we cannot add new features anymore, which is why I'm leaning towards a variant of the patch I showed above (which also minimizes the risk for regressions by introducing and using a new API). |
One option would be to create a list of possible defined signals and check if the signal is on the list. For realtime signals, it just a matter to check if SIGRTMIN <= signal <= SIGRTMAX. The glibc defined signals can be checked at tst-signal.c [1] or from main signal(7). It should cover usual ISO C, POSIX, and some linux arch-specific signals, but you will still need to check if other OS defined extra signals uses elsewhere (another option would to add this check only for Linux/glibc). |
I don't think we want to be in the business of maintaining a list of cross-platform of supported signal values. Python may be compiled on bizarre non-glibc systems (or even systems with a proprietary libc). |
Yes I am aware, but I can't see a really portable way to provide the same functionality as 'sigfillset'. Ideally a libc implementation would return EINVAL on 'sigaddset' for invalid signals, but even for glibc this is not true (the very issue I fixed). |
Right, but it seems that even if sigaddset() allowed you to "set" signals 32 and 33, that would be ignored by pthread_sigmask(). This is what I get here (Ubuntu 16.04, glibc 2.23): >>> signal.pthread_sigmask(signal.SIG_BLOCK, range(1, 65))
set()
>>> signal.pthread_sigmask(signal.SIG_BLOCK, range(1, 65))
{<Signals.SIGHUP: 1>, <Signals.SIGINT: 2>, <Signals.SIGQUIT: 3>, <Signals.SIGILL: 4>, <Signals.SIGTRAP: 5>, <Signals.SIGABRT: 6>, <Signals.SIGBUS: 7>, <Signals.SIGFPE: 8>, <Signals.SIGUSR1: 10>, <Signals.SIGSEGV: 11>, <Signals.SIGUSR2: 12>, <Signals.SIGPIPE: 13>, <Signals.SIGALRM: 14>, <Signals.SIGTERM: 15>, 16, <Signals.SIGCHLD: 17>, <Signals.SIGCONT: 18>, <Signals.SIGTSTP: 20>, <Signals.SIGTTIN: 21>, <Signals.SIGTTOU: 22>, <Signals.SIGURG: 23>, <Signals.SIGXCPU: 24>, <Signals.SIGXFSZ: 25>, <Signals.SIGVTALRM: 26>, <Signals.SIGPROF: 27>, <Signals.SIGWINCH: 28>, <Signals.SIGIO: 29>, <Signals.SIGPWR: 30>, <Signals.SIGSYS: 31>, <Signals.SIGRTMIN: 34>, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, <Signals.SIGRTMAX: 64>} |
Yes, this is the issue I referred in previous comment [1]. Unfortunately it is only fixed on master (which will become 2.28). |
Not sure what you mean. In the example above, I try to block signals 32 and 33 using pthread_sigmask(), but the pthread_sigmask() return value shows they weren't blocked, which is ok. |
What I mean is for master glibc, which contains BZ#22391, 'sigaddset' will fail do add signal 32 and 33. But since you are now ignoring 'sigaddset' return code, it does not matter. |
Miro, this should be fixed now. Please reopen if it isn't. The sigfillset() functionality will be exposed to Python users in bpo-33332. |
Antoine, thank you very much for swift communication and fix. |
These issues also occur on Python 3.4 and 3.5. And I'm now upgraded to Ubuntu 18.10 which I guess has the new version of glibc. The regression test suite for both 3.4 and 3.5 blocks forever on three tests (multiprocessing_fork, multiprocessing_forkserver, and multiprocessing_spawn). This affects my ability as RM to run the regression test suite, not to mention affecting the fundamental usability and trustability of 3.4 and 3.5 releases. Can I please get backports of this patch to 3.4 and 3.5? Note that I consider this a release blocker for 3.4 and 3.5. And I'd expected to tag 3.4.10rc1 and 3.5.7rc1 in about twenty-four hours. So I sure would appreciate a quick turnaround on this if folks are available--otherwise I'll have to slip the schedule. Sorry for the late notice! |
I am not able to do PRs right now but here are the Fedora patches: https://src.fedoraproject.org/rpms/python34/blob/master/f/00302-fix-multiprocessing-regression-on-newer-glibcs.patch They seem quite identical to the applied 3.6 patch. |
Now fixed in 3.4 and 3.5. I can cut the RCs. Huzzah! |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: