ivykis cause segment fault on arm64

[sandy-lcq]

From v0.39(including), pthr_atfork function is called during loading, but for arm platform, glibc seems don't support this symbol in libpthread, so make segment fault when call it as it is NULL.

detail info　please refer this link:
syslog-ng/syslog-ng#2263

[Kokan]

I think this issue might be related: #6
It has this backtrace:

Program received signal SIGSEGV, Segmentation fault.
._dl_runtime_resolve () at ../sysdeps/powerpc/powerpc64/dl-trampoline.S:76
76 ld r0,0(r3)
(gdb) bt
#0 ._dl_runtime_resolve () at ../sysdeps/powerpc/powerpc64/dl-trampoline.S:76
#1 0x00000fffb7f81100 in pthr_atfork (prepare=@0xfffb7f986e0: 0xfffb7f817f0 <iv_signal_prepare>, parent=@0xfffb7f986f8: 0xfffb7f81860 <iv_signal_parent>,
child=@0xfffb7f98710: 0xfffb7f818d0 <iv_signal_child>) at pthr.h:55
#2 0x00000fffb7f81990 in iv_signal_init () at iv_signal.c:92

But it claims to be fixed in 0.42.3.

[sandy-lcq]

Yes, it is the same problem.

I think problem should be my environment don't have libpthread_nonshared.a, so it still go into the if part, and week pthread_atfork. Previous fix seems only can cover case with libpthread_nonshared.a.

#ifndef HAVE_LIBPTHREAD_NONSHARED
#pragma weak pthread_atfork
#endif

Result of checking libivykis.a.
nm lib/ivykis/src/.libs/libivykis.a | grep pthread_atfork
w pthread_atfork

I think we should also fix case without libpthread_nonshared.a, since it should not be always there.

[buytenh]

Hello! Thank you for the report. Linux/arm64 is one of the platforms I run the testsuite on with every release. Is this on Linux, and if yes, which distribution are you running, so that I may try to reproduce this?

[sandy-lcq]

Hi,

Here is the detail reproduce steps:

1. mkdir yocto ; cd yocto
2. git clone git://git.yoctoproject.org/poky
3. git clone git://git.openembedded.org/openembedded-core
4. git clone git://git.openembedded.org/meta-openembedded
5. mkdir build; cd poky
6. . oe-init-build-env ../build/
7. add path "XXX/yocto/openembedded-core/meta" and "XXX/yocto/meta-openembedded/meta-oe" to BBLAYERS in conf/bblayers.conf (Please pay attention to the change to your path), for mine, like this:
   BBLAYERS ?= "
   /cli/yocto/poky/meta
   /cli/yocto/poky/meta-poky
   /cli/yocto/poky/meta-yocto-bsp
   /cli/yocto/openembedded-core/meta
   /cli/yocto/meta-openembedded/meta-oe
   "
8. add below part into conf/local.conf (under build dir)
   IMAGE_INSTALL_append += " syslog-ng"
   VIRTUAL-RUNTIME_init_manager = "systemd"
   DISTRO_FEATURES_append = " systemd"
   DISTRO_FEATURES_BACKFILL_CONSIDERED += "sysvinit"
   KERNEL_FEATURES_append = " cfg/systemd.scc"
9. comments "MACHINE ??= "qemux86" in conf/local.conf
   and uncomments "MACHINE ?= "qemuarm64"
10. bitbake core-image-sato
11. after build success, run command
    runqemu qemuarm64 core-image-sato nographic qemuparams="-m 2048" bootparams="selinux=0 enforing=0"

After qemu is bootup, we can see syslog-ng service start failed. Actually it is run below command failed.
/usr/sbin/syslog-ng -F --enable-core --cfgfile /etc/syslog-ng/syslog-ng.conf --control /var/lib/syslog-ng/syslog-ng.ctl --persist-file /var/lib/syslog-ng/syslog-ng.persist --pidfile /var/run/syslog-ng.pid

if you want to open debug and add gdb into this image, add below lines into conf/local.conf
DEBUG_BUILD = "1"
EXTRA_IMAGE_FEATURES_append += " dbg-pkgs"
MAGE_INSTALL += "gdb"
IMAGE_INSTALL += "gdbserver"

Besides, refer solution of systemd, I try to fix by replace pthread_atfork with __register_atfork.
I test on arm64 and x86-64, both works, syslog-ng can startup successfully. Please see
the attached patch to check if this solution is proper here, thanks.

http://refspecs.linuxbase.org/LSB_3.0.0/LSB-PDA/LSB-PDA/baselib--register-atfork.html

fix.patch.zip

[Bonstra]

I'm having the same issue (pthread_atfork is NULL) on ARMv5, with glibc 2.28, running ArchLinuxARM.

In glibc 2.28, pthread_atfork was moved from pthread_nonshared.a to libc_nonshared.a, and pthread_nonshared.a was removed.
The commit doing that change: https://sourceware.org/git/?p=glibc.git;a=commit;h=bd60ce86520b781ca24b99b2555e2ad389bbfeaa
x86_64 is not affected because the pthread_atfork symbol is defined in libpthread.so.

[furiel]

Thanks @Bonstra, your explanation helped me. I changed

-AC_CHECK_LIB([pthread_nonshared], [pthread_atfork])
+AC_CHECK_LIB([c_nonshared], [pthread_atfork]

and

-#ifndef HAVE_LIBPTHREAD_NONSHARED
+#ifndef HAVE_LIBC_NONSHARE

and the crash is gone.

[buytenh]

Sorry that I haven't addressed this yet, but I still can't reproduce this.

I installed a few virtual machines with Linux distributions that use glibc 2.28, and they indeed have pthread_atfork() now defined in libc_nonshared.a instead of in libpthread_nonshared.a, but I can't seem to reproduce a crash due to pthread_atfork being NULL, as libc_nonshared.a seems to be getting pulled in automatically, because of:

$ cat /lib64/libc.so
/* GNU ld script
   Use the shared library, but some functions are only in
   the static library, so try that secondarily.  */
OUTPUT_FORMAT(elf64-littleaarch64)
GROUP ( /lib64/libc.so.6 /usr/lib64/libc_nonshared.a  AS_NEEDED ( /lib/ld-linux-aarch64.so.1 ) )
$

Does the distribution you are seeing this on not have this? (What distribution are you seeing this bug on?)

I also don't see what would be ARM-specific about this issue, as I see the exact same behavior on x86_64 and on aarch64.

[buytenh]

This issue reports this happening on Fedora 29 for aarch64: syslog-ng/syslog-ng#2377

I installed a fresh Fedora 29 for aarch64 (by using qemu-system-aarch64), and the stock syslog-ng package indeed segfaults there:

[root@aarch64 ~]# syslog-ng --help
Segmentation fault (core dumped)
[root@aarch64 ~]# 

I then downloaded the source RPM for ivykis as shipped with Fedora 29, and rebuilt it:

dnf download --source ivykis
rpm -i ivykis-0.42.3-2.fc29.src.rpm
cd rpmbuild/SPECS/
vi ivykis.spec           # modified the Release: field to distinguish my custom build
dnf install rpm-build
rpmbuild -ba ivykis.spec

And then I installed the just-built ivykis packages over the stock ones:

[root@aarch64 aarch64]# rpm -Uvh ivykis-0.42.3-2.fc29lb.aarch64.rpm ivykis-devel-0.42.3-2.fc29lb.aarch64.rpm 
Verifying...                          ################################# [100%]
Preparing...                          ################################# [100%]
Updating / installing...
   1:ivykis-0.42.3-2.fc29lb           ################################# [ 25%]
   2:ivykis-devel-0.42.3-2.fc29lb     ################################# [ 50%]
Cleaning up / removing...
   3:ivykis-devel-0.42.3-2.fc29       ################################# [ 75%]
   4:ivykis-0.42.3-2.fc29             ################################# [100%]
[root@aarch64 aarch64]#

And now I get:

[root@aarch64 ~]# syslog-ng --help
Usage:
  syslog-ng [OPTION?] syslog-ng

Help Options:
[...]

This suggests to me that the Fedora 29 ivykis packages for aarch64 have been misbuilt somehow.

cc @czanik

[buytenh]

On Fedora 29, it seems that for both aarch64 and x86_64, the
libivykis.so as shipped in the distribution's ivykis package
references pthread_atfork, but a manually built ivykis package
yields a libivykis.so that references __register_atfork instead.

However, while the x86_64 libpthread.so provides pthread_atfork:

[root@test ~]$ rpm -q glibc
glibc-2.28-23.fc29.x86_64
glibc-2.28-23.fc29.i686
[root@test ~]$ nm /usr/lib64/libpthread.so | grep atfork
0000000000010220 t .annobin_old_pthread_atfork.c
0000000000010230 t .annobin_old_pthread_atfork.c_end
0000000000010220 t __dyn_pthread_atfork
0000000000010220 T pthread_atfork@GLIBC_2.2.5
                 U __register_atfork@@GLIBC_2.3.2
[root@test ~]$ 

the aarch64 libpthread.so does not:

[root@aarch64 ~]# rpm -q glibc
glibc-2.28-23.fc29.aarch64
[root@aarch64 ~]# nm /usr/lib64/libpthread.so | grep atfork
000000000000f384 t .annobin_old_pthread_atfork.c
000000000000f384 t .annobin_old_pthread_atfork.c_end
[root@aarch64 ~]#

I'm guessing that the pthread_atfork references in libivykis.so are
there because these ivykis packages were built against an older glibc
version, and that the pthread_atfork symbol is only provided for
backward compatibility now, but I don't see why this symbol is provided
on x86_64 but not on aarch64. (Note that the glibc versions are the same.)

I'm also guessing that the c_nonshared patch is a red herring, as it
seems that rebuilding the Fedora ivykis package even without any changes
makes things work again.

I still don't understand why this also appears to be an issue when
everything is (presumably) built from scratch, like in the Yocto case.

[Bonstra]

/lib/libc.so (glibc 2.28) on my ARMv5 ArchLinux looks like:

/* GNU ld script
   Use the shared library, but some functions are only in
   the static library, so try that secondarily.  */
OUTPUT_FORMAT(elf32-littlearm)
GROUP ( /usr/lib/libc.so.6 /usr/lib/libc_nonshared.a  AS_NEEDED ( /usr/lib/ld-linux.so.3 )

I don't know if this is relevant, but on my system, I have no ivykis shared library (ArchLinux doesn't even provide a libivykis package). The syslog-ng 3.18.1 tarball embeds ivykis 0.42.3, and compiles it as a static library, which is then used by the linker when creating libsyslog-ng-3.18.so.0.

I rebuilt the syslog-ng packages, and the built libivykis.a contains a weak undefined symbol "pthread_atfork" (referenced from iv_signal.o), which is expected. However, despite the libc_nonshared.a in /lib/libc.so, the libsyslog-ng-3.18.so.0 still has the weak undefined "pthread_atfork" symbol.

In that specific case, that could be a syslog-ng issue at link time.

[buytenh]

The stock Fedora 29 libivykis.so does not seem to have been linked
against libc_nonshared.a, as it has this code sequence in it:

    3644:       910ec042        add     x2, x2, #0x3b0
    3648:       910ac021        add     x1, x1, #0x2b0
    364c:       91064000        add     x0, x0, #0x190
    3650:       97ffff44        bl      3360 <pthread_atfork@plt>

While a rebuilt libivykis.so has this code sequence instead:

    3644:       910ec042        add     x2, x2, #0x3b0
    3648:       910ac021        add     x1, x1, #0x2b0
    364c:       91064000        add     x0, x0, #0x190
    3650:       94001d0e        bl      aa88 <iv_inotify_watch_unregister@@IVYKIS_0.29+0x38>

where the 'iv_inotify_watch_unregister' symbol is a red herring, as
the code pointed to is this:

000000000000aa50 <iv_inotify_watch_unregister@@IVYKIS_0.29>:
[...]
    aa7c:       a8c27bfd        ldp     x29, x30, [sp], #32
    aa80:       17ffe28c        b       34b0 <iv_avl_tree_delete@plt>
    aa84:       d503201f        nop
    aa88:       b00000a3        adrp    x3, 1f000 <iv_inotify_watch_unregister@@IVYKIS_0.29+0x145b0>
    aa8c:       f9441463        ldr     x3, [x3, #2088]
    aa90:       17ffe168        b       3030 <__register_atfork@plt>

In the stock libivykis.so, the iv_inotify_watch_unregister function
ends like this (where it jumps to iv_avl_tree_delete because of tail
call optimisation):

000000000000aa50 <iv_inotify_watch_unregister@@IVYKIS_0.29>:
[...]
    aa7c:       a8c27bfd        ldp     x29, x30, [sp], #32
    aa80:       17ffe28c        b       34b0 <iv_avl_tree_delete@plt>

In other words, the rebuilt libivykis.so has some bytes tacked onto
the end of iv_inotify_watch_unregister (which is the last function in
the shared object), and these extra bytes look suspiciously like the
definition of __pthread_atfork in libc_nonshared.a:

0000000000000000 <__pthread_atfork>:
   0:   90000003        adrp    x3, 0 <__dso_handle>
   4:   f9400063        ldr     x3, [x3]
   8:   14000000        b       0 <__register_atfork>

So for Fedora, I would say that the problem is that their ivykis package
was built against a glibc that does not have libc_nonshared.a, and
that glibc was then updated with a version that moves pthread_atfork
from libpthread_nonshared.a to libc_nonshared.a and changes the symbol
name in the process, and that ivykis was not rebuilt after that change
was made, and that the aarch64 version of glibc does not provide the
appropriate compatibility symbol. I would recommend rebuilding the
ivykis package as the way to fix this.

[buytenh]

For ArchLinux I am guessing that the problem is because of how syslog-ng
builds its bundled copy of ivykis. To verify this, @Bonstra, could you
send me a copy of your libsyslog-ng-3.18.so.0 file?

(BTW, there is this: https://aur.archlinux.org/packages/libivykis-git/ )

[buytenh]

I noticed just now that there is a new ivykis package available for
Fedora 29 now, ivykis-0.42.3-3.fc29, and which seems to fix the issue
on aarch64:

[root@aarch64 ~]# rpm -Uvh ivykis-0.42.3-3.fc29.aarch64.rpm
Verifying...                          ################################# [100%]
Preparing...                          ################################# [100%]
Updating / installing...
   1:ivykis-0.42.3-3.fc29             ################################# [ 50%]
Cleaning up / removing...
   2:ivykis-0.42.3-2.fc29             ################################# [100%]
[root@aarch64 ~]# syslog-ng -V
syslog-ng 3 (3.17.2)
[...]

The changelog for this package says:

[root@aarch64 ~]# rpm -q --changelog ivykis
* Wed Dec 19 2018 My Karlsson <mk@acc.umu.se> - 0.42.3-3
- Rebuilt

[...]

So it seems that that is all that was needed.

[ghost]

@buytenh Thanks for confirming that it resolved the issue for you. ~My

[buytenh]

@mykarlsson Thanks for the rebuild!

[Bonstra]

@Bonstra, could you > send me a copy of your libsyslog-ng-3.18.so.0 file?

Here it is (gzipped, because Github doesn't like .so files):
libsyslog-ng-3.18.so.0.0.0.gz

Alternatively, you can download the package here:
https://archlinuxarm.org/packages/arm/syslog-ng

[buytenh]

@Bonstra Thank you for the shared library!

The libsyslog-ng-3.18.so.0.0.0 you sent me does not seem to have
been linked against libc_nonshared.a, as it contains:

0001fc30 <pthread_atfork@plt>:
   1fc30:       e28fc600        add     ip, pc, #0, 12
   1fc34:       e28cca91        add     ip, ip, #593920 ; 0x91000
   1fc38:       e5bcf6dc        ldr     pc, [ip, #1756]!        ; 0x6dc

[...]

   206c8:       e59f20d0        ldr     r2, [pc, #208]  ; 207a0 <g_mutex_free@plt+0x150>
   206cc:       e59f10d0        ldr     r1, [pc, #208]  ; 207a4 <g_mutex_free@plt+0x154>
   206d0:       e59f00d0        ldr     r0, [pc, #208]  ; 207a8 <g_mutex_free@plt+0x158>
   206d4:       e08f2002        add     r2, pc, r2
   206d8:       e08f1001        add     r1, pc, r1
   206dc:       e08f0000        add     r0, pc, r0
   206e0:       ebfffd52        bl      1fc30 <pthread_atfork@plt>

Either this libsyslog-ng-3.18.so.0.0.0 was built against an older
glibc that did not have libc_nonshared.a, or it failed to be linked
against libc_nonshared.a for some other reason even though the glibc
it was built against did provide libc_nonshared.a.

If I understood correctly, the syslog-ng crash on startup problem
persists on ARM on Arch Linux after rebuilding the syslog-ng package
without any changes (unlike what was the case for Fedora), right?
This would suggest the latter explanation.

I always build ivykis as a shared library, and I don't test with it
built as a static library, and I'm not entirely sure what the linking
behavior here this should be, but it seems to be that:

• If I build ivykis as a shared library, the auto* machinery
  produces a libivykis.so that will have been linked against
  libc_nonshared.a.

• If I build ivykis as a static library (using
  ./configure --disable-shared), then I get a libivykis.a which
  will not have been linked against libc_nonshared.a.

• On a --disable-shared build, libc_nonshared.a gets linked into
  ivykis programs during the final linking of the binary.

From this, it would seem that if syslog-ng builds ivykis as a static
library and then links that into its shared library[*], then syslog-ng
should make sure that its shared library is linked against
libc_nonshared.a when it is produced, and this doesn't appear to be
the case. (And linking libc_nonshared.a into libivykis.a would
actually be the wrong thing to do.)

Is there any chance you could add -verbose to the linker invocation
for the linking of libsyslog-ng-3.18.so.0.0.0 and share the output?

[*] Which I'm also not sure is an entirely sane thing to do, as it links
non-PIC code into a PIC library? (But I am way out of my depth on
this topic and I don't really know what I am talking about. :-) )

[Bonstra]

If I understood correctly, the syslog-ng crash on startup problem
persists on ARM on Arch Linux after rebuilding the syslog-ng package
without any changes (unlike what was the case for Fedora), right?

That is correct.

However, symlinking /usr/lib/libpthread_nonshared.a to /usr/lib/libc_nonshared.a and rebuilding produces a working libsyslog-ng.so.

And I think I found out why.

In src/pthr.h, the pthread_atfork symbol is marked as weak only if HAVE_LIBPTHREAD_NONSHARED is not defined.
Creating the symlink caused autoconf to define that macro in config.h, which in turn, caused the compiler to ignore the #pragma weak pthread_atfork directive.
The effect is that object files referencing pthread_atfork (iv_signal.o) will have an undefined global symbol in their symbol table instead of a weak symbol.

And this changes everything! Quoting the System V ABI specification (emphasis by me):

When the link editor searches archive libraries [see ``Archive File'' in Chapter 7], it extracts archive members that contain definitions of undefined global symbols. [...] The link editor does not extract archive members to resolve undefined weak symbols. Unresolved weak symbols have a zero value.

So it makes sense why the resulting libsyslog-ng.so has the undefined pthread_atfork weak symbol present even with the libc.so linker script referencing libc_nonshared.a: because the linker does not even bother looking for this specific symbol in the libc_nonshared.a archive!

Is there any chance you could add -verbose to the linker invocation
for the linking of libsyslog-ng-3.18.so.0.0.0 and share the output?

Here is the full build log, with V=1 and -verbose added to the linker flags:
build-verbose.log

The log is pretty long, but it contains all the command lines used to run the compiler and linker. The linker is invoked at line 13284 to produce libsyslog-ng.so.

As you can see, at line 13990, the linker opens /usr/lib/libc_nonshared.a, but does not extract any archive member.
However, when pthread_atfork is a global symbol (when building with the symlink present), the line (/usr/lib/libc_nonshared.a)pthread_atfork.oS appears imediately after.

So, to solve this issue, I think DerDakon's pull request #16 would do the job.
Or alternatively, it could be a good opportunity to look why the pthread_* symbols were declared weak, and remove some #pragmas if they no longer serve a purpose.

[buytenh]

@Bonstra Thank you very much for your investigation, and sorry once
more for the delay.

I was a bit stumped trying to reproduce this issue on my aarch64 VM,
but as it turns out, Fedora for aarch64 has:

[root@aarch64 ~]# ls -al /lib64/*nonshared*
-rw-r--r-- 1 root root 35900 Apr  4 08:41 /lib64/libc_nonshared.a
-rw-r--r-- 1 root root     8 Apr  4 08:41 /lib64/libpthread_nonshared.a
[root@aarch64 ~]#

Which causes this library check to succeed:

checking for pthread_atfork in -lpthread_nonshared... yes

even when my pthread_nonshared.a doesn't provide a symbol for
pthread_atfork() -- and this was causing pthread_atfork to not be
marked as weak (and therefore, everything to work fine) even though
my system provides it in libc_nonshared.a.

If I remove my /lib64/libpthread_nonshared.a, I can reproduce this
problem, and it indeed seems to be because of what you quoted, that is,
pthread_atfork being NULL if it is provided by an archive library
if it was marked as weak. This very reason is why the #pragma weak
in src/pthr.h was conditional in the first place.

I'll apply something along the lines of the patch in pull request #16.

Or alternatively, it could be a good opportunity to look why the
pthread_* symbols were declared weak, and remove some #pragmas if
they no longer serve a purpose.

We do this because we want ivykis to operate correctly if we're running
inside a multi-threaded application, which requires some use of the
various pthread_*() functions if they are available (that is, if
libpthread.so has been linked into the application we're running in),
but we don't want to unconditionally pull in libpthread.so, because
we don't want to force single-threaded users of ivykis to pay the
performance penalty of pulling in libpthread.so.

[buytenh]

Does https://github.com/buytenh/ivykis/commits/libc-nonshared work for you?

[hbakken]

Sorry about the late response, but the answer it yes, it does.

[Bonstra]

Works for me™
I could build a working syslog-ng using ivykis libc-nonshared branch. The libyvykis.so.0 built as part of the syslog-ng build process has a local pthread_atfork symbol.

   [...]
   500: 00004ff0   544 FUNC    LOCAL  DEFAULT   12 iv_fd_poll_and_run
   501: 0000997c    20 FUNC    LOCAL  DEFAULT   12 pthread_atfork
   502: 00003dd0    16 FUNC    LOCAL  DEFAULT   12 iv_tls_total_state_size
   [...]

[buytenh]

I've merged this fix as commit a5e9cad ("Avoid marking pthread_atfork()
as weak if provided by libc_nonshared.a."), and I have released ivykis
v0.42.4 with this fix included.

Thanks again to all involved!