5.0.0: Error in dlsym(RTLD_NEXT, "pthread_create")

[Safari77]

jemalloc 4.5.0 worked, but not 5.0.0.

$ LD_DEBUG=files ldd mpv 2>&1|grep libjemalloc
      3778:	file=libjemalloc.so.2 [0];  needed by /lib64/libavcodec.so.57 [0]
      3778:	file=libjemalloc.so.2 [0];  generating link map
	libjemalloc.so.2 => /lib64/libjemalloc.so.2 (0x00007f5664908000)

$ ./mpv
<jemalloc>: Error in dlsym(RTLD_NEXT, "pthread_create")
Aborted

Fedora 25 x86_64, glibc-2.24-6.fc25.

[Safari77]

Hello World program linked with -ljemalloc works OK, though.

[Shooooooooo]

I am looking at a similar issue. Static link my program with jemalloc seems to be broken for me. It works fine without -static.
/med/code3/shchen/jemalloc/src/background_thread.c:786: undefined reference to `dlsym'
Was using redhat6.

[davidtgoldblatt]

Assigning to @interwq on the assumption that this is related to the background thread stuff.

[Earnestly]

Perhaps related: https://bugs.archlinux.org/task/54447

[davidtgoldblatt]

People hitting this bug: could you clarify if you're linking with -pthread?

[Safari77]

-pthread is used at least when linking mpv, and

     29877:     file=libpthread.so.0 [0];  needed by ./mpv [0]

[Earnestly]

The commit a4d6fe7 has solved the issue with bulding neovim and running it as discussed here: neovim/neovim#6895

[Safari77]

a4d6fe7 and 52fc887 crash with vlc in malloc_mutex_trylock_final

Program received signal SIGSEGV, Segmentation fault.
0x0000000000000000 in ?? ()
(gdb) bt
#0  0x0000000000000000 in ?? ()
#1  0x00007fffe92fae44 in malloc_mutex_trylock_final (mutex=0x7fffe95a3040 <init_lock>) at include/jemalloc/internal/mutex.h:146
#2  0x00007fffe92faee9 in malloc_mutex_lock (tsdn=0x0, mutex=0x7fffe95a3040 <init_lock>) at include/jemalloc/internal/mutex.h:204
#3  0x00007fffe93045a1 in malloc_init_hard () at src/jemalloc.c:1452
#4  0x00007fffe9300cab in malloc_init () at src/jemalloc.c:216
#5  0x00007fffe9308428 in jemalloc_constructor () at src/jemalloc.c:3013
#6  0x00007ffff7de7a3a in call_init (l=<optimized out>, argc=argc@entry=1, argv=argv@entry=0x7fffffffc248, env=env@entry=0x7fffffffc258)
    at dl-init.c:72
#7  0x00007ffff7de7b4b in call_init (env=0x7fffffffc258, argv=0x7fffffffc248, argc=1, l=<optimized out>) at dl-init.c:30
#8  _dl_init (main_map=main_map@entry=0x5555557f9420, argc=1, argv=0x7fffffffc248, env=0x7fffffffc258) at dl-init.c:120
#9  0x00007ffff7decab6 in dl_open_worker (a=a@entry=0x7fffffffb930) at dl-open.c:564
#10 0x00007ffff7de78e4 in _dl_catch_error (objname=objname@entry=0x7fffffffb920, errstring=errstring@entry=0x7fffffffb928, 
    mallocedp=mallocedp@entry=0x7fffffffb91f, operate=operate@entry=0x7ffff7dec750 <dl_open_worker>, args=args@entry=0x7fffffffb930)
    at dl-error.c:187
#11 0x00007ffff7dec089 in _dl_open (file=0x5555557fa850 "/usr/lib64/vlc/plugins/video_chroma/libswscale_plugin.so", mode=-2147483647, 
    caller_dlopen=0x7ffff795e430 <module_Load+32>, nsid=-2, argc=<optimized out>, argv=<optimized out>, env=0x7fffffffc258) at dl-open.c:649
#12 0x00007ffff6af7f09 in dlopen_doit (a=a@entry=0x7fffffffbb60) at dlopen.c:66
#13 0x00007ffff7de78e4 in _dl_catch_error (objname=0x555555758050, errstring=0x555555758058, mallocedp=0x555555758048, 
    operate=0x7ffff6af7eb0 <dlopen_doit>, args=0x7fffffffbb60) at dl-error.c:187
#14 0x00007ffff6af8591 in _dlerror_run (operate=operate@entry=0x7ffff6af7eb0 <dlopen_doit>, args=args@entry=0x7fffffffbb60) at dlerror.c:163
#15 0x00007ffff6af7fa2 in __dlopen (file=<optimized out>, mode=<optimized out>) at dlopen.c:87
#16 0x00007ffff795e430 in module_Load () from /lib64/libvlccore.so.8
#17 0x00007ffff78daba0 in AllocatePluginDir () from /lib64/libvlccore.so.8
#18 0x00007ffff78daaf3 in AllocatePluginDir () from /lib64/libvlccore.so.8
#19 0x00007ffff78db056 in AllocatePluginPath () from /lib64/libvlccore.so.8
#20 0x00007ffff78db71a in module_LoadPlugins () from /lib64/libvlccore.so.8
#21 0x00007ffff78c1f24 in libvlc_InternalInit () from /lib64/libvlccore.so.8
#22 0x00007ffff7bb9777 in libvlc_new () from /lib64/libvlc.so.5
#23 0x000055555555532b in main ()
(gdb) 

$ ldd /usr/lib64/vlc/plugins/video_chroma/libswscale_plugin.so
	linux-vdso.so.1 (0x00007ffc557d1000)
	libvlccore.so.8 => /lib64/libvlccore.so.8 (0x00007f3ecef9e000)
	libidn.so.11 => /lib64/libidn.so.11 (0x00007f3eced6a000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f3eceb4c000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00007f3ece948000)
	libanl.so.1 => /lib64/libanl.so.1 (0x00007f3ece742000)
	libdbus-1.so.3 => /lib64/libdbus-1.so.3 (0x00007f3ece4f2000)
	libswscale.so.4 => /lib64/libswscale.so.4 (0x00007f3ece261000)
	libm.so.6 => /lib64/libm.so.6 (0x00007f3ecdf58000)
	libc.so.6 => /lib64/libc.so.6 (0x00007f3ecdb92000)
	/lib64/ld-linux-x86-64.so.2 (0x000055b69098b000)
	libsystemd.so.0 => /lib64/libsystemd.so.0 (0x00007f3ecdb07000)
	libavutil.so.55 => /lib64/libavutil.so.55 (0x00007f3ecd87d000)
	libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f3ecd662000)
	libselinux.so.1 => /lib64/libselinux.so.1 (0x00007f3ecd43b000)
	libcap.so.2 => /lib64/libcap.so.2 (0x00007f3ecd236000)
	librt.so.1 => /lib64/librt.so.1 (0x00007f3ecd02e000)
	liblzma.so.5 => /lib64/liblzma.so.5 (0x00007f3ecce08000)
	liblz4.so.1 => /lib64/liblz4.so.1 (0x00007f3eccbf2000)
	libgcrypt.so.20 => /lib64/libgcrypt.so.20 (0x00007f3ecc8e5000)
	libgpg-error.so.0 => /lib64/libgpg-error.so.0 (0x00007f3ecc6d1000)
	libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f3ecc4ba000)
	libX11.so.6 => /lib64/libX11.so.6 (0x00007f3ecc17b000)
	libvdpau.so.1 => /lib64/libvdpau.so.1 (0x00007f3ecbf77000)
	libva.so.1 => /lib64/libva.so.1 (0x00007f3ecbd55000)
	libva-x11.so.1 => /lib64/libva-x11.so.1 (0x00007f3ecbb4f000)
	libva-drm.so.1 => /lib64/libva-drm.so.1 (0x00007f3ecb94c000)
	libjemalloc.so.2 => /lib64/libjemalloc.so.2 (0x00007f3ecb489000)
	libpcre.so.1 => /lib64/libpcre.so.1 (0x00007f3ecb216000)
	libxcb.so.1 => /lib64/libxcb.so.1 (0x00007f3ecafee000)
	libXext.so.6 => /lib64/libXext.so.6 (0x00007f3ecadda000)
	libXfixes.so.3 => /lib64/libXfixes.so.3 (0x00007f3ecabd4000)
	libdrm.so.2 => /lib64/libdrm.so.2 (0x00007f3eca9c3000)
	libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f3eca63b000)
	libXau.so.6 => /lib64/libXau.so.6 (0x00007f3eca437000)

[jasone]

I built and ran nvim on an Ubuntu 16.04.2 LTS system, with jemalloc 5.0.0 built in. nvim ran for me without incident, and so far none of the primary jemalloc developers have managed to reproduce the issue. I'm not satisfied that we have a general solution to this issue yet, but a4d6fe7 mitigates the issue sufficiently for the 5.0.1 release.

[DavidPfander-UniStuttgart]

I'm another affected user: I was using jemalloc master on a Intel Knight Landing hardware. On that platform, I ran into the exact error described in the first post. Things work fine with 4.5. BTW great fan of jemalloc!

[interwq]

@DavidPfander-UniStuttgart: thanks for letting us know! Can you please share more information about the application / platform, and the exact jemalloc commit you are using? We expect 5.0.1 release to workaround this, and we are still looking for a repro so that we can investigate further.

[DavidPfander-UniStuttgart]

Unfortunately, I'm unable to reproduce this issue with current master on my desktop machine. Please note that I originally encountered the problem on a Intel Knights Landing machine with a very different software environment.
Originally, I was using the then current jemalloc master. We're working on this project: https://github.com/STEllAR-GROUP/octotiger. Unfortunately, the application is not trivial to build.

[Whissi]

Seeing this with jemalloc-5.0.1 and rsyslog-8.28.0:

<jemalloc>: Error in dlsym(RTLD_NEXT, "pthread_create")

Program received signal SIGABRT, Aborted.
__GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
51      ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
#1  0x00007f563960dbea in __GI_abort () at abort.c:89
#2  0x00007f5639fb7f49 in je_background_thread_boot0 ()
    at /var/tmp/portage/dev-libs/jemalloc-5.0.1/work/jemalloc-5.0.1/src/background_thread.c:829
#3  0x00007f5639fa9aad in malloc_init_hard_recursible ()
    at /var/tmp/portage/dev-libs/jemalloc-5.0.1/work/jemalloc-5.0.1/src/jemalloc.c:1311
#4  malloc_init_hard () at /var/tmp/portage/dev-libs/jemalloc-5.0.1/work/jemalloc-5.0.1/src/jemalloc.c:1473
#5  0x00007f5639fab075 in malloc_init ()
    at /var/tmp/portage/dev-libs/jemalloc-5.0.1/work/jemalloc-5.0.1/src/jemalloc.c:216
#6  imalloc (dopts=<synthetic pointer>, sopts=<synthetic pointer>)
    at /var/tmp/portage/dev-libs/jemalloc-5.0.1/work/jemalloc-5.0.1/src/jemalloc.c:1927
#7  malloc (size=72704) at /var/tmp/portage/dev-libs/jemalloc-5.0.1/work/jemalloc-5.0.1/src/jemalloc.c:1977
#8  0x00007f56392686f0 in ?? () from /usr/lib/gcc/x86_64-pc-linux-gnu/6.3.0/libstdc++.so.6
#9  0x00007f563ac4644a in call_init (l=<optimized out>, argc=argc@entry=1, argv=argv@entry=0x7fffffffdeb8,
    env=env@entry=0x7fffffffdec8) at dl-init.c:72
#10 0x00007f563ac4655b in call_init (env=0x7fffffffdec8, argv=0x7fffffffdeb8, argc=1, l=<optimized out>)
    at dl-init.c:30
#11 _dl_init (main_map=0x7f563ae5b110, argc=1, argv=0x7fffffffdeb8, env=0x7fffffffdec8) at dl-init.c:120
#12 0x00007f563ac37b8a in _dl_start_user () from /lib64/ld-linux-x86-64.so.2
#13 0x0000000000000001 in ?? ()
#14 0x00007fffffffe1a6 in ?? ()
#15 0x0000000000000000 in ?? ()

jemalloc-5.0.1 build.log: https://www.dropbox.com/s/r7h8ijtm2aworxe/i907_jemallic_build.log?dl=1

rsyslog-8.28.0 build.log: https://www.dropbox.com/s/mm742vr7r40irxt/i907_rsyslog_build.log?dl=1

[TheMarex]

I ran into the same issue on 5.0.1 with Linux version 4.4.87-25-default SUSE Linux using GCC 7.

However this only was the case when I dynamically linked to libjemalloc. Using the LD_PRELOAD method it worked just fine.

EDIT: Found the culprit for me: Linking order is the important part. -lpthread needs to be linked after -ljemalloc.

[toddlipcon]

I'm also running into this trying to link against jemalloc in a shared library build. Similar to @TheMarex I found that it's important that libjemalloc shows up in the 'ldd' output prior to libpthread, or else it won't find any result for RTLD_NEXT (since it only looks at later entries in the library path).

I am able to workaround it on my linker invocation by using -Wl,--no-as-needed /path/to/jemalloc.so -Wl,--as-needed, but that's pretty messy and hard to get working in my cmake project.

Another solution might be to use RTLD_DEFAULT instead of RTLD_NEXT in the case that LAZY_LOCK is not enabled (in that case we don't have our own pthread_create wrapper and so the default symbol resolution ought to be fine?)

[fti-akumari]

I'm facing an issue with the dig and nslookup commands when using jemalloc version 5.3.0 on my Linux machine. The error message Error in dlsym(RTLD_NEXT, "pthread_create") is encountered, which leads to the termination of the commands.

<jemalloc>: Error in dlsym(RTLD_NEXT, "pthread_create")
Aborted

System Configuration:

• OS: Debian

• Version: 12 (Bookworm)

• Kernel: 6.5.0-0.deb12.4-amd64#1SMP PREEMPT_DYNAMIC Debian 6.5.10-1~bpo12+1 (2023-11-23) x86_64 GNU/Linux

Observations:

• jemalloc Version: This issue arises specifically with jemalloc version 5.3.0.

• Error Context: The error seems related to dynamic linking (dlsym) of pthread_create.

• Lazy-lock: Enabling lazy-lock exacerbates the issue, causing the mentioned commands to fail consistently.

• Workaround: Disabling lazy-lock resolves the problem, but this isn't an optimal solution as it impacts performance.