segfault on createing metadata with --workers=2 (3,4,5,etc too)

[fedya]

[New LWP 30387]

Program received signal SIGSEGV, Segmentation fault.
[Switching to LWP 30387]
0x00007ffff7f2677b in g_string_chunk_insert_len () from /lib64/libglib-2.0.so.0
(gdb) bt
#0 0x00007ffff7f2677b in g_string_chunk_insert_len () from /lib64/libglib-2.0.so.0
#1 0x00007ffff7fd4277 in cr_package_from_header () from /usr/lib64/libcreaterepo_c.so.0
#2 0x00007ffff7fd4884 in cr_package_from_rpm_base () from /usr/lib64/libcreaterepo_c.so.0
rpm-software-management#3 0x00007ffff7fcc016 in cr_dumper_thread () from /usr/lib64/libcreaterepo_c.so.0
rpm-software-management#4 0x00007ffff7f2a7fc in ?? () from /lib64/libglib-2.0.so.0
rpm-software-management#5 0x00007ffff7f2a296 in ?? () from /lib64/libglib-2.0.so.0
rpm-software-management#6 0x00007ffff6f4402d in start_thread () from /lib64/libpthread.so.0
rpm-software-management#7 0x00007ffff7b3f10f in clone () from /lib64/libc.so.6

[fedya]

==30457== LEAK SUMMARY:
==30457== definitely lost: 20,688 bytes in 431 blocks
==30457== indirectly lost: 600 bytes in 5 blocks
==30457== possibly lost: 3,200 bytes in 10 blocks
==30457== still reachable: 298,496,237 bytes in 117,582 blocks
==30457== suppressed: 0 bytes in 0 blocks
==30457==
==30457== ERROR SUMMARY: 4 errors from 4 contexts (suppressed: 0 from 0)
==30457==
==30457== 1 errors in context 1 of 4:
==30457== Thread 7 pool:
==30457== Invalid read of size 1
==30457== at 0x57FE1EE: urlPath (in /usr/lib64/librpmio-5.4.so)
==30457== by 0x56EFA92: ??? (in /usr/lib64/librpmdb-5.4.so)
==30457== by 0x56EFBE8: ??? (in /usr/lib64/librpmdb-5.4.so)
==30457== by 0x56E0D49: headerGet (in /usr/lib64/librpmdb-5.4.so)
==30457== by 0x40376CC: cr_package_from_header (in /usr/lib64/libcreaterepo_c.so.0.10.0)
==30457== by 0x4038883: cr_package_from_rpm_base (in /usr/lib64/libcreaterepo_c.so.0.10.0)
==30457== by 0x4030015: cr_dumper_thread (in /usr/lib64/libcreaterepo_c.so.0.10.0)
==30457== by 0x40AC7FB: ??? (in /lib64/libglib-2.0.so.0.5000.2)
==30457== by 0x40AC295: ??? (in /lib64/libglib-2.0.so.0.5000.2)
==30457== by 0x5ABC02C: start_thread (in /lib64/libpthread-2.24.so)
==30457== by 0x4EE810E: clone (in /lib64/libc-2.24.so)
==30457== Address 0x653030373728202d is not stack'd, malloc'd or (recently) free'd
==30457==
==30457== ERROR SUMMARY: 4 errors from 4 contexts (suppressed: 0 from 0)

[fedya]

Program received signal SIGSEGV, Segmentation fault.
[Switching to LWP 421]
0x00007ffff6f45d74 in pthread_mutex_lock () from /lib64/libpthread.so.0
(gdb) bt
#0 0x00007ffff6f45d74 in pthread_mutex_lock () from /lib64/libpthread.so.0
#1 0x00007ffff73ddad6 in yarnPossess () from /usr/lib64/librpmio-5.4.so
#2 0x00007ffff73a420e in rpmioLinkPoolItem () from /usr/lib64/librpmio-5.4.so
rpm-software-management#3 0x00007ffff73a6761 in pgpDigNew () from /usr/lib64/librpmio-5.4.so
rpm-software-management#4 0x00007ffff74f06a7 in rpmtsDig () from /usr/lib64/librpmdb-5.4.so
rpm-software-management#5 0x00007ffff74f38cc in rpmReadPackageFile () from /usr/lib64/librpmdb-5.4.so
rpm-software-management#6 0x00007ffff7fd479e in read_header (filename=, hdr=, err=) at /usr/src/debug/createrepo_c-0.10.0/src/parsepkg.c:127
rpm-software-management#7 cr_package_from_rpm_base (filename=0x7e7a10 "/share/platforms/cooker/repository/x86_64/main/release/TiMidity++-2.14.2-0.20140127.2-omv2015.0.x86_64.rpm", changelog_limit=10, flags=CR_HDRR_NONE,
err=0x7fffe47df518) at /usr/src/debug/createrepo_c-0.10.0/src/parsepkg.c:164
rpm-software-management#8 0x00007ffff7fcc016 in cr_dumper_thread (data=, user_data=) at /usr/src/debug/createrepo_c-0.10.0/src/dumper_thread.c:242
rpm-software-management#9 0x00007ffff7f2a7fc in ?? () from /lib64/libglib-2.0.so.0
rpm-software-management#10 0x00007ffff7f2a296 in ?? () from /lib64/libglib-2.0.so.0
rpm-software-management#11 0x00007ffff6f4402d in start_thread () from /lib64/libpthread.so.0
rpm-software-management#12 0x00007ffff7b3f10f in clone () from /lib64/libc.so.6

[n3npq]

Yes. I've added a mutex to serialize header I/O, which protects ts->dig (returned by pgpDigNew()) for now.

Serialization seems to limit the number of worker threads reading packages to ~3, which is likely most of the benefit of multiple threads. Threads formatting XML/SQLITE are unaffected (and that's likely most of the work).

A better fix will have to wait for a new version of rpm to add a refcount on ts->dig (but its a very tricky code path to test, including automatically fetching public keys with HKP and more).