Parallel rpmfc thread-safety - libmagic may fork helpers

[pmatilai]

When using MAGIC_COMPRESS (as rpmfc does), libmagic may fork helper executables to deal with decompressing files. Since commit 41f0e21 we are running the classifier in threads, and mixing fork() and threads tends to mean trouble.

Filing this ticket as a reminder to look into and deal with it one way or the other: use of MAGIC_COMPRESS dates back to 2009 (commit c3770c9), predating the current customizable classifier and availability of path based rules. Looking into compressed files was necessary for at least font provide extraction but we could trivially use paths for that now instead.

[pmatilai]

For the record, there have been some reports of builds hanging during classification on systems with large number of CPUs using file 5.36 which where fixed by updating to 5.37. Another workaround is to set %_smp_build_nthreads to a smaller number, ultimately 1 to make the whole classification run in serial as it always used to.

[kanavin]

Sadly, I am seeing this when rpm is linked against file 5.37 as well. Here's a couple tracebacks obtained by attaching gdb to running (hanging) rpm processes:

0x00007fc5a9ee22a1 in ?? () from /usr/lib/x86_64-linux-gnu/libgomp.so.1
(gdb) bt
#0 0x00007fc5a9ee22a1 in ?? () from /usr/lib/x86_64-linux-gnu/libgomp.so.1
#1 0x00007fc5ab0579be in ?? () from /home/alexander/development/poky/build-x86-64/tmp/work/core2-64-poky-linux/elfutils/0.177-r0/recipe-sysroot-native/usr/bin/../lib/librpmbuild.so.9
#2 0x00007fc5a9ed6ecf in GOMP_parallel () from /usr/lib/x86_64-linux-gnu/libgomp.so.1
#3 0x00007fc5ab058b3c in rpmfcClassify () from /home/alexander/development/poky/build-x86-64/tmp/work/core2-64-poky-linux/elfutils/0.177-r0/recipe-sysroot-native/usr/bin/../lib/librpmbuild.so.9
#4 0x00007fc5ab0598e6 in ?? () from /home/alexander/development/poky/build-x86-64/tmp/work/core2-64-poky-linux/elfutils/0.177-r0/recipe-sysroot-native/usr/bin/../lib/librpmbuild.so.9
#5 0x00007fc5ab04aa3b in ?? () from /home/alexander/development/poky/build-x86-64/tmp/work/core2-64-poky-linux/elfutils/0.177-r0/recipe-sysroot-native/usr/bin/../lib/librpmbuild.so.9
#6 0x00007fc5ab044f74 in ?? () from /home/alexander/development/poky/build-x86-64/tmp/work/core2-64-poky-linux/elfutils/0.177-r0/recipe-sysroot-native/usr/bin/../lib/librpmbuild.so.9
#7 0x0000562e400c1ad1 in ?? ()
#8 0x0000562e400c1c33 in ?? ()
#9 0x0000562e400c0da3 in ?? ()
#10 0x00007fc5aa7d3ddb in __libc_start_main () from /home/alexander/development/poky/build-x86-64/tmp/sysroots-uninative/x86_64-linux/lib/libc.so.6
#11 0x0000562e400c107a in ?? ()

0x00007f2220f000ee in waitpid () from /home/alexander/development/poky/build-x86-64/tmp/sysroots-uninative/x86_64-linux/lib/libc.so.6
(gdb) bt
#0 0x00007f2220f000ee in waitpid () from /home/alexander/development/poky/build-x86-64/tmp/sysroots-uninative/x86_64-linux/lib/libc.so.6
#1 0x00007f221ff53f46 in ?? () from /home/alexander/development/poky/build-x86-64/tmp/work/core2-64-poky-linux/util-linux/2.34-r0/recipe-sysroot-native/usr/lib/rpm/.././libmagic.so.1
#2 0x00007f221ff54478 in ?? () from /home/alexander/development/poky/build-x86-64/tmp/work/core2-64-poky-linux/util-linux/2.34-r0/recipe-sysroot-native/usr/lib/rpm/.././libmagic.so.1
#3 0x00007f221ff57b16 in ?? () from /home/alexander/development/poky/build-x86-64/tmp/work/core2-64-poky-linux/util-linux/2.34-r0/recipe-sysroot-native/usr/lib/rpm/.././libmagic.so.1
#4 0x00007f221ff48e54 in ?? () from /home/alexander/development/poky/build-x86-64/tmp/work/core2-64-poky-linux/util-linux/2.34-r0/recipe-sysroot-native/usr/lib/rpm/.././libmagic.so.1
#5 0x00007f2221223bb9 in ?? () from /home/alexander/development/poky/build-x86-64/tmp/work/core2-64-poky-linux/util-linux/2.34-r0/recipe-sysroot-native/usr/lib/rpm/../librpmbuild.so.9
#6 0x00007f221f894ecf in GOMP_parallel () from /usr/lib/x86_64-linux-gnu/libgomp.so.1
#7 0x00007f2221224b3c in rpmfcClassify () from /home/alexander/development/poky/build-x86-64/tmp/work/core2-64-poky-linux/util-linux/2.34-r0/recipe-sysroot-native/usr/lib/rpm/../librpmbuild.so.9
#8 0x0000556ed5245571 in ?? ()
#9 0x00007f2220e5eddb in __libc_start_main () from /home/alexander/development/poky/build-x86-64/tmp/sysroots-uninative/x86_64-linux/lib/libc.so.6
#10 0x0000556ed524579a in ?? ()

Process tree shows:

rpmbuild.real─┬─2*[bzip2]
└─31*[{rpmbuild.real}]

rpmdeps.real─┬─29*[xz]
└─31*[{rpmdeps.real}]

[kanavin]

TBH, this seems like a serious enough issue in 4.15.0 release, that parallel file classification should be simply disabled for the time being, with a 4.15.0.1 issued to include that. I don't want to set %_smp_build_nthreads to 1, as Yocto project does benefit greatly from parallel binary package creation.

[pmatilai]

TBH, the thought of reverting the whole damn threading business has crossed my mind more than once by now, it is (quite expectedly) proving painful. I've resisted the temptation so far...

The thing is that this hasn't been an issue on Fedora at all, AFAIK. If we just disable without understanding where the difference comes from, it'll still be there the next time we try enabling it.
Instead it's the parallel binary package creation that has been the source of problems there, and I've been considering disabling that on 32bit architectures entirely.

[pmatilai]

Anyway, there will be a bugfix release to 4.15.x soonish, there's a handful of fairly critical things that needs addressing. I mean, it is a .0 release...

[kanavin]

Let's work together and sort the problem. If you would like me to run additional diagnostics, I'd be happy to help.

[kanavin]

I dug a bit deeper: the issue happens when libmagic calls out to external executables, which isn't implemented in a thread-safe way, and locks up. Enabling internal compression support (done via linking to libraries) makes the issue go away.

[pmatilai]

Yup, at least one such issue has just been addressed here: file/file@81f15c2b

[pmatilai]

Based on internal testing, the above seems to have cured the remaining cases, so considering closed.