3683a9ee5a breaks many tests on OpenBSD threaded builds

[jkeenan]

Commits within the last two days or so have caused massive new test failures on threaded builds on the OpenBSD operating system.

These failures are evident in these two smoke-test reports:

• OpenBSD-6.6: https://perl5.test-smoke.org/report/5010661 | v5.35.3-119-g75595dd289

• OpenBSD-6.9: https://perl5.test-smoke.org/report/5010672 | v5.35.3-123-g0ae3a5cd82

The tests fail in a variety of ways. t/run/switches.t reports a core dump.

t/run/switches ................................................... # Failed test 121 - threads while in-place editing at ./test.pl line 1077
#      got "Bizarre SvTYPE [85] at - line 6.\nSegmentation fault (core dumped) "
# expected "ok"
# PROG: 
# use threads;
# use strict;
# @ARGV = ("tmp19365G/foo");
# $^I = "";
# while (<>) {
#   threads->create(sub { })->join;
#   print "yy\n";
# }
# print "ok\n";
# STATUS: 35584
# Failed test 123 - file successfully saved after chdir at run/switches.t line 527
#      got "xx\n"
# expected "yy\n"
# Failed test 129 - no extra files at run/switches.t line 613
#      got "1"
# expected "0"
# Found EJhBZH6a, expected none
FAILED at test 121

Also observed:

t/io/msg ......................................................... SysV.c: loadable library and perl binaries are mismatched (got handshake key 0xeae458d19b8, needed 0xeaea47840c0)
FAILED--no leader found
...
t/io/sem ......................................................... SysV.c: loadable library and perl binaries are mismatched (got handshake key 0xd18ea8c59e8, needed 0xd19956ae0c0)
FAILED--no leader found
...
t/io/shm ......................................................... SysV.c: loadable library and perl binaries are mismatched (got handshake key 0x14e851d9928, needed 0x14ddced40c0)
FAILED--no leader found

Unthreaded builds appear to be okay so far.

Bisecting with the following invocation ...

perl Porting/bisect.pl -Duseithreads --target=t/io/sem.t \
    --start=756f0910f9 --end=75595dd289

... points to the following commit:

commit 3683a9ee5ad61f8cb5eac30e61b4936bd0445bdb (refs/bisect/bad)
Author:     Nicholas Clark <nick@ccl4.org>
AuthorDate: Sun Jun 20 17:24:15 2021 +0000
Commit:     Tony Cook <tony@develop-help.com>
CommitDate: Tue Sep 7 15:46:16 2021 +1000

    If we have thread local storage, use it instead of posix_getspecific().
    
    Declare and use a variable PL_current_context to store a pointer to the
    thread's interpreter struct. Use this to implement implicit context undef
    ITHREADS, instead of a call to posix_getspecific().
    
    For normal threaded code, this will eliminate function calls. For threaded
    code in shared objects (the typical configuration for Linux distribution
    builds) there still needs to be a function call to get the pointer to
    thread local variables, but there will only be one per function, not one
    per read of the variable.

@nwc10, @tonycoz, can you take a look?

Thank you very much.
Jim Keenan

[jkeenan]

I can confirm that at the commit (65f7068) before the one identified by bisection, all tests passed on a threaded build on OpenBSD.

[nwc10]

@tonycoz - are you in a better position than me to carry this on...

I can replicate this on the OpenBSD machine on the gcc compiler farm. Minimal test case I can find is:

gcc220$ ./perl -Ilib -MIPC::SysV -e0
SysV.c: loadable library and perl binaries are mismatched (got handshake key 0x6ff82e6e418, needed 0x6ff6e0fd0c0)

(I don't know why I get the handshake warning, rather than just a good old fashioned SEGV)

Problem case appears to be modules that don't define PERL_NO_GET_CONTEXT

So, on that machine, I get this far with a breakpoint on boot_IPC__SysV

...

(gdb)
Single stepping until exit from function _libc_pthread_getspecific,
which has no line number information.
_libc_pthread_self () at /usr/src/lib/libc/thread/rthread.c:114
114     /usr/src/lib/libc/thread/rthread.c: No such file or directory.
        in /usr/src/lib/libc/thread/rthread.c
(gdb) where
#0  _libc_pthread_self () at /usr/src/lib/libc/thread/rthread.c:114
#1  0x000009022ec19629 in _libc_pthread_getspecific (key=1)
    at /usr/src/lib/libc/thread/rthread_tls.c:112
#2  0x000008ff84f3ad0f in __emutls_get_address (control=0x8ff84f54180)
    at /usr/src/gnu/lib/libcompiler_rt/../../llvm/compiler-rt/lib/builtins/emutls.c:97
#3  0x000008ff84da2d3e in Perl_xs_handshake (key=255853799,
    v_my_perl=0x9018849b448, file=0x9020495e33a "SysV.c") at util.c:5592
#4  0x000009020495ff6d in boot_IPC__SysV (my_perl=0x9025649e0c0,
    cv=0x9021403c130) at SysV.c:837
#5  0x000008ff84deb2c9 in Perl_pp_entersub (my_perl=0x9025649e0c0)
    at pp_hot.c:5272
#6  0x000008ff84dd5186 in Perl_runops_standard (my_perl=0x9025649e0c0)
    at run.c:41
#7  0x000008ff84c907b6 in Perl_call_sv (my_perl=0x9025649e0c0,
    sv=0x9019a4fb340, flags=13) at perl.c:3079
#8  0x000008ff84c8bf0b in Perl_call_list (my_perl=0x9025649e0c0, oldscope=2,
    paramList=0x9019a4fb3b8) at perl.c:5140
#9  0x000008ff84c6c009 in S_process_special_blocks (my_perl=0x9025649e0c0,
    floor=40, fullname=0x9026656d1a0 "BEGIN", gv=0x9019a4fb3d0,
    cv=0x9019a4fb340) at op.c:11857
#10 0x000008ff84c5f9bc in Perl_newATTRSUB_x (my_perl=0x9025649e0c0, floor=40,
    o=0x90246daa910, proto=0x0, attrs=0x0, block=0x9026824aaf8, o_is_gv=false)
    at op.c:11783
#11 0x000008ff84c60822 in Perl_utilize (my_perl=0x9025649e0c0, aver=1,
    floor=40, version=0x0, idop=0x901db81d9f8, arg=0x0) at op.c:8842
#12 0x000008ff84cf2aeb in Perl_yyparse (my_perl=0x9025649e0c0, gramtype=258)
    at perly.y:365
#13 0x000008ff84c8f23f in S_parse_body (my_perl=0x9025649e0c0, env=0x0,
    xsinit=0x8ff84c45ec0 <xs_init>) at perl.c:2558
#14 0x000008ff84c8d07b in perl_parse (my_perl=0x9025649e0c0,
    xsinit=0x8ff84c45ec0 <xs_init>, argc=4, argv=0x7f7fffff9468, env=0x0)
    at perl.c:1853
#15 0x000008ff84c45e06 in main (argc=4, argv=0x7f7fffff9468,
    env=0x7f7fffff9490) at perlmain.c:109
(gdb) s
43              __asm__ ("movq %%fs:(%1),%0" : "=r" (val) : "r" (offset));
(gdb)
0x000009022ec1962c in _libc_pthread_getspecific (key=1)
   from /usr/lib/libc.so.96.0
(gdb)
Single stepping until exit from function _libc_pthread_getspecific,
which has no line number information.
__emutls_get_address (control=0x8ff84f54180)
    at /usr/src/gnu/lib/libcompiler_rt/../../llvm/compiler-rt/lib/builtins/emutls.c:358
358     /usr/src/gnu/lib/libcompiler_rt/../../llvm/compiler-rt/lib/builtins/emutls.c: No such file or directory.
        in /usr/src/gnu/lib/libcompiler_rt/../../llvm/compiler-rt/lib/builtins/emutls.c
(gdb)
0x000008ff84f3ad14 in __emutls_get_address (control=0x8ff84f54180)
   from /home/nick/Perl/perl/perl
(gdb)
Single stepping until exit from function __emutls_get_address,
which has no line number information.
Perl_xs_handshake (key=255853799, v_my_perl=0x9018849b448,
    file=0x9020495e33a "SysV.c") at util.c:5599
5599        va_start(args, file);
Current language:  auto; currently c
(gdb)
5601        got = INT2PTR(void*, (UV)(key & HSm_KEY_MATCH));
(gdb) p my_perl
Segmentation fault (core dumped)
gcc220$

@tonycoz - do you have access to an OpenBSD machine where gdb doesn't SEGV printing a pointer?

I have suspicion that we're just going to need to disable it in the hints for OpenBSD, much like OS X and (IIRC) cygwin. Seems that this is another platform where the toolchain writes cheques that it cannot cash.

[nwc10]

Aha, there is a second (not-a-router) OpenBSD machine. To cut to the chase:

diff --git a/util.c b/util.c
index f187145202..633961f633 100644
--- a/util.c
+++ b/util.c
@@ -5588,6 +5588,7 @@ Perl_xs_handshake(const U32 key, void * v_my_perl, const char * file, ...)
     U32 items, ax;
     void * got;
     void * need;
+    const char *stage = "first";
 #ifdef MULTIPLICITY
     dTHX;
     tTHX xs_interp;
@@ -5611,6 +5612,8 @@ Perl_xs_handshake(const U32 key, void * v_my_perl, const char * file, ...)
     xs_interp = (tTHX)v_my_perl;
     got = xs_interp;
     need = my_perl;
+    Perl_warn_nocontext("In here PL_current_context is %p",
+                        PL_current_context);
 #else
 /* try to catch where an unthreaded perl interp DLL (for ex. perl522.dll) is
    loaded into a process by a XS DLL built by an unthreaded perl522.dll perl,
@@ -5624,12 +5627,13 @@ Perl_xs_handshake(const U32 key, void * v_my_perl, const char * file, ...)
     got = xs_spp;
     need = &PL_stack_sp;
 #endif
+    stage = "second";
     if(UNLIKELY(got != need)) {
         bad_handshake:/* recycle branch and string from above */
         if(got != (void *)HSf_NOCHK)
             noperl_die("%s: loadable library and perl binaries are mismatched"
-                       " (got handshake key %p, needed %p)\n",
-                file, got, need);
+                       " (got %s handshake key %p, needed %p)\n",
+                       file, stage, got, need);
     }

     if(key & HSf_SETXSUBFN) {     /* this might be called from a module bootstrap */

and with

obsd$ diff -u SysV.c~ SysV.c
--- SysV.c~     Wed Sep  8 13:26:51 2021
+++ SysV.c      Wed Sep  8 14:05:47 2021
@@ -831,6 +831,8 @@
 XS_EXTERNAL(boot_IPC__SysV); /* prototype to pass -Wmissing-prototypes */
 XS_EXTERNAL(boot_IPC__SysV)
 {
+    Perl_warn_nocontext("Out here PL_current_context is %p",
+                        PL_current_context);
 #if PERL_VERSION_LE(5, 21, 5)
     dVAR; dXSARGS;
 #else

we see this:

obsd$ ./perl  -Ilib -MIPC::SysV -e0
Out here PL_current_context is a7e9eb3b3e8 at lib/XSLoader.pm line 112.
In here PL_current_context is a7e8e3e50c0 at lib/XSLoader.pm line 112.
SysV.c: loadable library and perl binaries are mismatched (got second handshake key 0xa7e9eb3b3e8, needed 0xa7e8e3e50c0)

So clearly thread local storage and shared libraries don't work together on OpenBSD - compiles and links as if it's all going to work, and then feeds us garbage at runtime. :-(

Yes, I understand that getting this stuff to work with dynamically loaded shared objects is really really hard, but it would be nice if it failed with an error rather than pretending and then crashing in flames. (But hey, Apple have far more money and they goof this too.)

Need to disable this in the hints - I'm too tired to write a correct patch this evening.

[xenu]

TLS is emulated on OpenBSD, so not using it is probably for the best. I suspect our original context storage method is faster there.

[nwc10]

If it worked, it might still be faster, at least for naïve extensions, because the default is to make one pthread_getspecific call for each Perl API call in the XS function (to hide MULTIPLICTIY from code that has no idea about it without needing any code changes to pre-declare a variable). Whereas if OpenBSD is "faking it", that should mean only one pthread_getspecific call per XS function.

But "not working" is what we have. #19110 will disable it.

[jkeenan]

#19110 eliminated the test failures and has been merged to blead. I hope that we can get some comment on Nick's diagnoses from @afresh1 or other OpenBSD-knowledgeable people, and I am self-assigning the ticket with the expectation that I'll close it in about 7 days.

[jkeenan]

#19110 eliminated the test failures and has been merged to blead. I hope that we can get some comment on Nick's diagnoses from @afresh1 or other OpenBSD-knowledgeable people, and I am self-assigning the ticket with the expectation that I'll close it in about 7 days.

Closing as per schedule.