OS signals delivered to wrong thread under load

[TFBW]

Module: threads

Description

Documented threads behaviour is that OS signals "are caught by the main thread (thread ID = 0) of a script." There seems to be a race condition such that rapid-fire signals can result in delivery to other threads.

Steps to Reproduce

use strict;
use warnings;
use threads;
my $SIG = 'USR2';
sub nap { select(undef, undef, undef, 0.01 + rand(0.01)) }
my $nsig = 0;
$SIG{$SIG} = sub { $nsig++ };
my @thr = map {
    nap;
    async sub {
        $SIG{$SIG} = sub { warn "Thread #@{[threads->tid]} got @_\n" };
        for (1..10) { nap; kill $SIG => $$ }
        return;
    }
} 1..10;
warn "Threads running.\n";
$_->join for @thr;
warn "Threads done; #sigs=$nsig\n";
exit;

Expected behavior

Per documented behaviour, the signal handler installed inside the thread should never be called; only the one in the main thread counting signals should be called. In actual practice, some varying number of signals will trigger an in-thread handler, producing its warning output.

Perl configuration

Summary of my perl5 (revision 5 version 38 subversion 2) configuration:
   
  Platform:
    osname=linux
    osvers=6.1.0
    archname=x86_64-linux-gnu-thread-multi
    uname='linux localhost 6.1.0 #1 smp debian 6.1.0 x86_64 gnulinux '
    config_args='-Dmksymlinks -Dusethreads -Duselargefiles -Dcc=x86_64-linux-gnu-gcc -Dcpp=x86_64-linux-gnu-cpp -Dld=x86_64-linux-gnu-gcc -Dccflags=-DDEBIAN -Wdate-time -D_FORTIFY_SOURCE=3 -g -O2 -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -ffile-prefix-map=/dummy/build/dir=. -flto=auto -ffat-lto-objects -fstack-protector-strong -fstack-clash-protection -Wformat -Werror=format-security -fcf-protection -fdebug-prefix-map=/dummy/build/dir=/usr/src/perl-5.38.2-3.2ubuntu0.2 -Dldflags= -Wl,-Bsymbolic-functions -flto=auto -ffat-lto-objects -Wl,-z,relro -Dlddlflags=-shared -Wl,-Bsymbolic-functions -flto=auto -ffat-lto-objects -Wl,-z,relro -Dcccdlflags=-fPIC -Darchname=x86_64-linux-gnu -Dprefix=/usr -Dprivlib=/usr/share/perl/5.38 -Darchlib=/usr/lib/x86_64-linux-gnu/perl/5.38 -Dvendorprefix=/usr -Dvendorlib=/usr/share/perl5 -Dvendorarch=/usr/lib/x86_64-linux-gnu/perl5/5.38 -Dsiteprefix=/usr/local -Dsitelib=/usr/local/share/perl/5.38.2 -Dsitearch=/usr/local/lib/x86_64-linux-gnu/perl/5.38.2 -Dman1dir=/usr/share/man/man1 -Dman3dir=/usr/share/man/man3 -Dsiteman1dir=/usr/local/man/man1 -Dsiteman3dir=/usr/local/man/man3 -Duse64bitint -Dman1ext=1 -Dman3ext=3perl -Dpager=/usr/bin/sensible-pager -Uafs -Ud_csh -Ud_ualarm -Uusesfio -Uusenm -Ui_libutil -Ui_xlocale -Uversiononly -Ud_strlcpy -Ud_strlcat -DDEBUGGING=-g -Doptimize=-O2 -dEs -Duseshrplib -Dlibperl=libperl.so.5.38.2'
    hint=recommended
    useposix=true
    d_sigaction=define
    useithreads=define
    usemultiplicity=define
    use64bitint=define
    use64bitall=define
    uselongdouble=undef
    usemymalloc=n
    default_inc_excludes_dot=define
  Compiler:
    cc='x86_64-linux-gnu-gcc'
    ccflags ='-D_REENTRANT -D_GNU_SOURCE -DDEBIAN -fwrapv -fno-strict-aliasing -pipe -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64'
    optimize='-O2 -g'
    cppflags='-D_REENTRANT -D_GNU_SOURCE -DDEBIAN -fwrapv -fno-strict-aliasing -pipe -I/usr/local/include'
    ccversion=''
    gccversion='13.3.0'
    gccosandvers=''
    intsize=4
    longsize=8
    ptrsize=8
    doublesize=8
    byteorder=12345678
    doublekind=3
    d_longlong=define
    longlongsize=8
    d_longdbl=define
    longdblsize=16
    longdblkind=3
    ivtype='long'
    ivsize=8
    nvtype='double'
    nvsize=8
    Off_t='off_t'
    lseeksize=8
    alignbytes=8
    prototype=define
  Linker and Libraries:
    ld='x86_64-linux-gnu-gcc'
    ldflags =' -fstack-protector-strong -L/usr/local/lib'
    libpth=/usr/local/lib /usr/lib/x86_64-linux-gnu /usr/lib /lib/x86_64-linux-gnu /lib
    libs=-lgdbm -lgdbm_compat -ldb -ldl -lm -lpthread -lc -lcrypt
    perllibs=-ldl -lm -lpthread -lc -lcrypt
    libc=/lib/x86_64-linux-gnu/libc.so.6
    so=so
    useshrplib=true
    libperl=libperl.so.5.38
    gnulibc_version='2.39'
  Dynamic Linking:
    dlsrc=dl_dlopen.xs
    dlext=so
    d_dlsymun=undef
    ccdlflags='-Wl,-E'
    cccdlflags='-fPIC'
    lddlflags='-shared -L/usr/local/lib -fstack-protector-strong'


Characteristics of this binary (from libperl): 
  Compile-time options:
    HAS_LONG_DOUBLE
    HAS_STRTOLD
    HAS_TIMES
    MULTIPLICITY
    PERLIO_LAYERS
    PERL_COPY_ON_WRITE
    PERL_DONT_CREATE_GVSV
    PERL_HASH_FUNC_SIPHASH13
    PERL_HASH_USE_SBOX32
    PERL_MALLOC_WRAP
    PERL_OP_PARENT
    PERL_PRESERVE_IVUV
    PERL_USE_SAFE_PUTENV
    USE_64_BIT_ALL
    USE_64_BIT_INT
    USE_ITHREADS
    USE_LARGE_FILES
    USE_LOCALE
    USE_LOCALE_COLLATE
    USE_LOCALE_CTYPE
    USE_LOCALE_NUMERIC
    USE_LOCALE_TIME
    USE_PERLIO
    USE_PERL_ATOF
    USE_REENTRANT_API
    USE_THREAD_SAFE_LOCALE
  Locally applied patches:
    DEBPKG:debian/cpan_definstalldirs - Provide a sensible INSTALLDIRS default for modules installed from CPAN.
    DEBPKG:debian/db_file_ver - https://bugs.debian.org/340047 Remove overly restrictive DB_File version check.
    DEBPKG:debian/doc_info - Replace generic man(1) instructions with Debian-specific information.
    DEBPKG:debian/enc2xs_inc - https://bugs.debian.org/290336 Tweak enc2xs to follow symlinks and ignore missing @INC directories.
    DEBPKG:debian/errno_ver - https://bugs.debian.org/343351 Remove Errno version check due to upgrade problems with long-running processes.
    DEBPKG:debian/libperl_embed_doc - https://bugs.debian.org/186778 Note that libperl-dev package is required for embedded linking
    DEBPKG:fixes/respect_umask - Respect umask during installation
    DEBPKG:debian/writable_site_dirs - Set umask approproately for site install directories
    DEBPKG:debian/extutils_set_libperl_path - EU:MM: set location of libperl.a under /usr/lib
    DEBPKG:debian/no_packlist_perllocal - Don't install .packlist or perllocal.pod for perl or vendor
    DEBPKG:debian/fakeroot - Postpone LD_LIBRARY_PATH evaluation to the binary targets.
    DEBPKG:debian/instmodsh_doc - Debian policy doesn't install .packlist files for core or vendor.
    DEBPKG:debian/ld_run_path - Remove standard libs from LD_RUN_PATH as per Debian policy.
    DEBPKG:debian/libnet_config_path - Set location of libnet.cfg to /etc/perl/Net as /usr may not be writable.
    DEBPKG:debian/perlivp - https://bugs.debian.org/510895 Make perlivp skip include directories in /usr/local
    DEBPKG:debian/squelch-locale-warnings - https://bugs.debian.org/508764 Squelch locale warnings in Debian package maintainer scripts
    DEBPKG:debian/patchlevel - https://bugs.debian.org/567489 List packaged patches for 5.38.2-3.2ubuntu0.2 in patchlevel.h
    DEBPKG:fixes/document_makemaker_ccflags - https://bugs.debian.org/628522 [rt.cpan.org #68613] Document that CCFLAGS should include $Config{ccflags}
    DEBPKG:debian/find_html2text - https://bugs.debian.org/640479 Configure CPAN::Distribution with correct name of html2text
    DEBPKG:debian/perl5db-x-terminal-emulator.patch - https://bugs.debian.org/668490 Invoke x-terminal-emulator rather than xterm in perl5db.pl
    DEBPKG:debian/cpan-missing-site-dirs - https://bugs.debian.org/688842 Fix CPAN::FirstTime defaults with nonexisting site dirs if a parent is writable
    DEBPKG:debian/makemaker-pasthru - https://bugs.debian.org/758471 Pass LD settings through to subdirectories
    DEBPKG:debian/makemaker-manext - https://bugs.debian.org/247370 Make EU::MakeMaker honour MANnEXT settings in generated manpage headers
    DEBPKG:debian/kfreebsd-softupdates - https://bugs.debian.org/796798 Work around Debian Bug#796798
    DEBPKG:debian/hurd-softupdates - https://bugs.debian.org/822735 Fix t/op/stat.t failures on hurd
    DEBPKG:fixes/math_complex_doc_great_circle - https://bugs.debian.org/697567 [rt.cpan.org #114104] Math::Trig: clarify definition of great_circle_midpoint
    DEBPKG:fixes/math_complex_doc_angle_units - https://bugs.debian.org/731505 [rt.cpan.org #114106] Math::Trig: document angle units
    DEBPKG:fixes/cpan_web_link - https://bugs.debian.org/367291 CPAN: Add link to main CPAN web site
    DEBPKG:debian/installman-utf8 - https://bugs.debian.org/840211 Generate man pages with UTF-8 characters
    DEBPKG:debian/perldoc-pager - https://bugs.debian.org/870340 [rt.cpan.org #120229] Fix perldoc terminal escapes when sensible-pager is less
    DEBPKG:debian/prune_libs - https://bugs.debian.org/128355 Prune the list of libraries wanted to what we actually need.
    DEBPKG:debian/mod_paths - Tweak @INC ordering for Debian
    DEBPKG:debian/deprecate-with-apt - https://bugs.debian.org/747628 Point users to Debian packages of deprecated core modules
    DEBPKG:debian/disable-stack-check - https://bugs.debian.org/902779 [GH #16607] Disable debugperl stack extension checks for binary compatibility with perl
    DEBPKG:debian/perlbug-editor - https://bugs.debian.org/922609 Use "editor" as the default perlbug editor, as per Debian policy
    DEBPKG:debian/eu-mm-perl-base - https://bugs.debian.org/962138 Suppress an ExtUtils::MakeMaker warning about our non-default @INC
    DEBPKG:fixes/io_socket_ip_ipv6 - Disable getaddrinfo(3) AI_ADDRCONFIG for localhost and IPv4 numeric addresses
    DEBPKG:debian/usrmerge-lib64 - https://bugs.debian.org/914128 Configure / libpth.U: Do not adjust glibpth when /usr/lib64 is present.
    DEBPKG:debian/usrmerge-realpath - https://bugs.debian.org/914128 Configure / libpth.U: use realpath --no-symlinks on Debian
    DEBPKG:fixes/x32-io-msg-skip - https://bugs.debian.org/922609 Skip io/msg.t on x32 due to broken System V message queues
    DEBPKG:debian/configure-regen - https://bugs.debian.org/762638 Regenerate Configure et al. after probe unit changes
    DEBPKG:fixes/bignum-recursion - https://bugs.debian.org/1012704 [rt.cpan.org #146411] Fix CPAN RT #146411
    DEBPKG:fixes/math-bigfloat-subclass - https://bugs.debian.org/1043234 Fix Math::BigFloat subclass numeric comparison
    DEBPKG:fixes/unbreak-locale-initialization - [GH #21366] https://bugs.debian.org/1060679 Revert "locale.c: Do uselocale() earlier in init process"
    DEBPKG:CVE-2024-56406.patch -
    DEBPKG:fixes/CVE-2025-40909-metaconfig - https://bugs.debian.org/1098226 [GH #23010] Check for fdopendir
    DEBPKG:fixes/CVE-2025-40909-1 - https://bugs.debian.org/1098226 [GH #23010] [84be063] CVE-2025-40909: Clone dirhandles without fchdir
    DEBPKG:fixes/CVE-2025-40909-2 - https://bugs.debian.org/1098226 [GH #23010] [b088e97] Minor corrections to 1f9097b342e0e37d619dfab6ea82ea99611b30bf
    DEBPKG:fixes/CVE-2025-40909-3 - https://bugs.debian.org/1098226 [GH #23010] [08dffa0] Use PerlLIO_dup_cloexec in Perl_dirp_dup to set O_CLOEXEC
    DEBPKG:fixes/CVE-2025-40909-metaconfig-reorder - Slightly reorder Configure and config_h.SH to match metaconfig output
    DEBPKG:fixes/CVE-2025-40909-generated - Update generated files and checksums
  Built under linux
  Compiled at Jul 25 2025 17:26:40
  %ENV:
    PERLBREW_HOME="/home/famous/.perlbrew"
    PERLBREW_ROOT="/home/famous/perl5/perlbrew"
    PERLBREW_SHELLRC_VERSION="0.98"
  @INC:
    /etc/perl
    /usr/local/lib/x86_64-linux-gnu/perl/5.38.2
    /usr/local/share/perl/5.38.2
    /usr/lib/x86_64-linux-gnu/perl5/5.38
    /usr/share/perl5
    /usr/lib/x86_64-linux-gnu/perl-base
    /usr/lib/x86_64-linux-gnu/perl/5.38
    /usr/share/perl/5.38
    /usr/local/lib/site_perl

[tonycoz]

I'm pretty sure that is a bug in the documentation (perldoc threads).

Except (and only recently) for the case where the perl signal handler receives a signal in a non-perl thread, such as a library thread, perl delivers the signal to the interpreter associated with the current thread.

It is the operating system that chooses which OS thread to deliver the signal to.

I think @Leont could point out my errors.

[Leont]

I'm pretty sure that is a bug in the documentation (perldoc threads).

It is.

It is the operating system that chooses which OS thread to deliver the signal to.

Signals can be either directed at a process (e.g. kill) or at a thread (e.g. a segfault or pthread_kill). Delivering them in the same thread is needed for some purposes, but there is a bit of a mismatch in the signal handler concepts because signal disposition is shared within a process, but %SIG is specific to each interpreter.

In general I would recommend setting handlers before starting threads and not touch them afterwards; and use signal masking if you don't want specific signals to be delivered in specific threads.

[TFBW]

My particular issue is that I need the signal to be processed in the main thread because that's where the callback is: CODE refs are not portable across threads. The pseudo-signals in threads are not ideal because they don't interrupt system calls.

The solution I've adopted is to use the Linux-specific "tgkill" syscall. This seems to work, and I fall back to pseudo-signals if I can't find appropriate defs for it.

[jkeenan]

I'm pretty sure that is a bug in the documentation (perldoc threads).

Except (and only recently) for the case where the perl signal handler receives a signal in a non-perl thread, such as a library thread, perl delivers the signal to the interpreter associated with the current thread.

It is the operating system that chooses which OS thread to deliver the signal to.

I think @Leont could point out my errors.

@tonycoz @Leont can you suggest a patch for the documentation weakness? Thanks.

[Leont]

My particular issue is that I need the signal to be processed in the main thread

You can use POSIX::sigprocmask to mask that signal in all threads that shouldn't process it (or use my Signal::Mask module for an easier interface to it).

because that's where the callback is: CODE refs are not portable across threads.

That is rather ambiguously phrased, but I guess you either mean that setting it in one thread won't set it in other existing threads, or that they won't share data that they need to share.

The pseudo-signals in threads are not ideal because they don't interrupt system calls.

Yeah they're not very useful IME.

The solution I've adopted is to use the Linux-specific "tgkill" syscall.

That sounds rather similar to the kill method in my threads::posix module.

[TFBW]

You can use POSIX::sigprocmask to mask that signal in all threads that shouldn't process it (or use my Signal::Mask module for an easier interface to it).

"The use of sigprocmask() is unspecified in a multithreaded process," or so says the man page, but I suppose it works in practice?

That is rather ambiguously phrased, but I guess you either mean that setting it in one thread won't set it in other existing threads, or that they won't share data that they need to share.

To be more explicit, there is a callback (CODE ref) waiting in the main thread for a result that some worker thread is computing. I can't execute the callback in the worker thread because CODE refs can't be shared. I have to share the result and then signal the main thread, calling the CODE ref with the result from within the signal handler.

Thanks for the threads::posix tip. I won't add a dependency on it, but I'll add support for it.

[Leont]

"The use of sigprocmask() is unspecified in a multithreaded process," or so says the man page, but I suppose it works in practice?

Yeah, I once wrote Thread::SigMask because of that; it's because of M:N threading. I've heard rumors of old OpenBSD having such issues but I've never encountered such things myself. Everyone does 1:1 threading nowadays; on any major OS sigprocmask will do the right thing.

[tonycoz]

Should POSIX::sigprocmask() use pthread_sigmask() on threaded builds?

I see the freebsd pthread_sigprocmask() specific disallows blocking SIGCANCEL:

https://github.com/freebsd/freebsd-src/blob/5f7d5709e0c242d5a5fa5516d6079433ee06c347/lib/libthr/thread/thr_sig.c#L658-L667

and glibc disallows blocking SIGCANCEL and SIGSETXID (an alias for some RT signal):

https://sourceware.org/git/?p=glibc.git;a=blob;f=nptl/pthread_sigmask.c;h=752fd179776531f97a66eafcaf004cdbf99ac10c;hb=HEAD#l30