Thread local storage/pthread issues on jemalloc upgrade from 4.5 to 5.0.0

[martinetd]

I'm getting crashes (segfault) with qemu since upgrading jemalloc.
qemu has not been recompiled/relinked, so I'm thinking there is some subtle linking confusion between qemu, jemalloc and pthread - rebuilding qemu solves the issue locally (EDIT: I have no idea if it does, didn't use jemalloc on rebuild; will check today as well) but there are other applications that seem to crash similarily (like mariadb, probably any multithread program using jemalloc) and it's always better to understand what's happening :)

I've got two backtraces for now, they happen when qemu creates a new thread or does something with TLS memory management:

#1 0x00007f869c80b860 in () at /usr/lib/libjemalloc.so.2
#2 0x00007f869c80b9d4 in () at /usr/lib/libjemalloc.so.2
#3 0x00007f869c80c9a1 in () at /usr/lib/libjemalloc.so.2
#4 0x00007f869c7be29a in calloc () at /usr/lib/libjemalloc.so.2
#5 0x00007f86a2a16852 in allocate_dtv () at /lib64/ld-linux-x86-64.so.2
#6 0x00007f86a2a171ce in _dl_allocate_tls () at /lib64/ld-linux-x86-64.so.2
#7 0x00007f869bc66dca in pthread_create@@GLIBC_2.2.5 () at /usr/lib/libpthread.so.0
#8 0x0000559e3e77e475 in qemu_thread_create ()
#9 0x0000559e3e7792a7 in ()
#10 0x0000559e3e779308 in ()
#11 0x00007f869bc66297 in start_thread () at /usr/lib/libpthread.so.0
#12 0x00007f869b9a725f in clone () at /usr/lib/libc.so.6

#1 0x00007f7dfd2955cd in () at /usr/lib/libjemalloc.so.2
#2 0x00007f7dfd2977d8 in () at /usr/lib/libjemalloc.so.2
#3 0x00007f7dfc6f03c8 in __nptl_deallocate_tsd.part.4 () at /usr/lib/libpthread.so.0
#4 0x00007f7dfc6f135d in start_thread () at /usr/lib/libpthread.so.0
#5 0x00007f7dfc43225f in clone () at /usr/lib/libc.so.6

I don't have symbols for jemalloc itself, will rebuild jemalloc without stripping and provide new backtraces today if that helps.
FWIW, we have an arch bug open for this: https://bugs.archlinux.org/task/54483

Does that ring a bell to anyone? I could find recent issues about pthread but they all look slightly different.
Thanks!

[martinetd]

Here's a backtrace with jemalloc rebuilt with debug symbols:

#0  0x00007f202bb621bc in je_tcache_arena_associate (tsdn=0x7f201ec0aba8, tcache=0x7f201ec0ad60, arena=0x7f201e20c8c0) at src/tcache.c:295
#1  0x00007f202bb63860 in arena_choose_impl (arena=0x0, internal=false, tsd=0x7f201ec0aba8) at include/jemalloc/internal/jemalloc_internal_inlines_b.h:35
#2  0x00007f202bb63860 in arena_choose (arena=0x0, tsd=0x7f201ec0aba8) at include/jemalloc/internal/jemalloc_internal_inlines_b.h:63
#3  0x00007f202bb63860 in je_tsd_tcache_data_init (tsd=tsd@entry=0x7f201ec0aba8) at src/tcache.c:410
#4  0x00007f202bb639d4 in je_tsd_tcache_enabled_data_init (tsd=tsd@entry=0x7f201ec0aba8) at src/tcache.c:339
#5  0x00007f202bb649a1 in tsd_data_init (tsd=0x7f201ec0aba8) at src/tsd.c:74
#6  0x00007f202bb649a1 in je_tsd_fetch_slow (tsd=tsd@entry=0x7f201ec0aba8, internal=internal@entry=false) at src/tsd.c:126
#7  0x00007f202bb1629a in tsd_fetch_impl (internal=false, init=true) at include/jemalloc/internal/tsd.h:260
#8  0x00007f202bb1629a in tsd_fetch () at include/jemalloc/internal/tsd.h:275
#9  0x00007f202bb1629a in imalloc (dopts=<synthetic pointer>, sopts=<synthetic pointer>) at src/jemalloc.c:1940
#10 0x00007f202bb1629a in calloc (num=29, size=16) at src/jemalloc.c:2060
#11 0x00007f2031d6d852 in allocate_dtv () at /lib64/ld-linux-x86-64.so.2
#12 0x00007f2031d6e1ce in _dl_allocate_tls () at /lib64/ld-linux-x86-64.so.2
#13 0x00007f202afbedca in pthread_create@@GLIBC_2.2.5 () at /usr/lib/libpthread.so.0
#14 0x000055ef4c4a0475 in qemu_thread_create ()
#15 0x000055ef4c49b2a7 in  ()
#16 0x000055ef4c49b308 in  ()
#17 0x00007f202afbe297 in start_thread () at /usr/lib/libpthread.so.0
#18 0x00007f202acff25f in clone () at /usr/lib/libc.so.6

[interwq]

The trace looks alright to me. If it's crashing there, it implies there was some corruption happened to TLS of other threads. Did rebuilding qemu change anything? Also, if it's related to fork(), you may want to try the most recent dev branch.

[martinetd]

Took me a while but I've just reproduced on a fresh build, so that does not help. I need a better reproducer, this is too sloppy...

I don't think qemu ever forks, so we should be fine with current jemalloc build - I need a more controlled way to make qemu create and kill threads to reproduce more reliably but I have kept the core dumps so far, if you need anything from there I can print whatever you ask for.

[interwq]

Which commit is this based on? Might be #896 which caused similar crashes.

[interwq]

Rebase past #911 if not yet. Let us know if it still happens.

[martinetd]

The arch package I used has exactly 5.0.0 (5018fe3) + a4d6fe7 (only abort on dlsym when necessary)
I've cherry-picked 9b1befa on top of this instead of rebasing due to the way I think packages are handled (I reckon the fork fix would also probably be useful there at some point)

So far it seems to hold out (VM been up for 2 hours), will post another comment tomorrow as I'm not 100% confident crash always happens this fast, but it looks good for now.

[martinetd]

Still looks good, I'd say that's it ; thank you for pointing to the right commit (and the original fix!) :)

"Fixed by #911"