Librdkafka fails to create thread on RHEL 8

[harishankar1998]

Description

Back Trace

#0 0x00007ffff7833340 in thrd_create () from /usr/lib64/libpthread.so.0
#1 0x00007ffff2193afe in rd_kafka_new (type=RD_KAFKA_CONSUMER, app_conf=0x894840, errstr=0x88c438 "",
errstr_size=512) at rdkafka.c:2179

[New Thread 0x7ffff0ca3700 (LWP 31923)]
rd_kafka_new (type=RD_KAFKA_CONSUMER, app_conf=0x894840, errstr=0x88c438 "", errstr_size=512) at rdkafka.c:2181
2181 rk->rk_init_wait_cnt--;
(gdb) n
2182 ret_err = RD_KAFKA_RESP_ERR__CRIT_SYS_RESOURCE;
(gdb) n
2183 ret_errno = errno;
(gdb) n
2184 if (errstr)
(gdb) n
2185 rd_snprintf(errstr, errstr_size,
(gdb) n
2188 rd_kafka_wrunlock(rk);
(gdb) p errstr
$1 = 0x88c438 "Failed to create thread: No such file or directory (2)"
(gdb)

Librdkafka is built with the following options
./configure --enable-gssapi --enable-ssl --disable-optimization
.........................
.........................
checking for libpthread (by pkg-config)... failed
checking for libpthread (by compile)... ok (cached)
checking for c11threads (by pkg-config)... failed
checking for c11threads (by compile)... failed (disable)
..................................
..................................

Checklist

IMPORTANT: We will close issues where the checklist has not been completed.

Please provide the following information:

• librdkafka version (release number or git tag): <latest>
• Operating system: <RHEL 8>
• Critical issue

[edenhill]

I believe the error code is irrelevant ("No such file or directory") since the glibc thrd_create() does not set errno, looking at the manual page it seems the only failure it can return is thrd_nomem, which indicates that the current process has a too strict thread, memory or stack limit.
Also check cat /proc/sys/kernel/threads-max

[edenhill]

Todo: fix error string

[harishankar1998]

cat /proc/sys/kernel/threads-max

30015

[Lukas955]

I noticed the same error message on Fedora 31 with librdkafka (0.11.6) from Fedora RPM repository (probably built without c11 thread support).

In my case, an application is loading a plugin, which depends on librdkafka, using dlopen(). I discovered that the problem is caused by linker and multiple definitions of thrd_create() - one provided by libpthread and another by librdkafka. Since my application doesn't depend on librdkafka, the linker binds thrd_create to implementation provided by libpthread before startup of the app. So far, so good.

However, when the plugin is later loaded (by dlopen) and thrd_create symbol required by librdkafka must be resolved again, the linker decides to use early used binding i.e. implementation from libpthread. Here is the problem: exact value of thrd_success is undefined by C11, however, for libpthread and GCC 9.2.1 thrd_success value is 0, but tinycthread.h defined thrd_success as 2 and thrd_error as 0. As the librdkafka was built against values from tinycthread, it was expecting value 2 on success, but implementation from libpthread returned 0 (on success).

https://github.com/edenhill/librdkafka/blob/1f3f46b808efe712c6b894c7ad82c81f7406c522/src/tinycthread.h#L184

I resolved this issue by RTLD_DEEPBIND flag of dlopen(). Nevertheless, I think the library should not redefine these common symbols.

[edenhill]

Thanks for the root cause of this issue, @Lukas955
Will see what we can do about it.