3.1.x dies in rlm_cache.c #1714

Open
alanbuxey opened this Issue Sep 5, 2016 · 7 comments

Projects

None yet

2 participants

@alanbuxey
Member

Issue type

  • Questions about the server or its usage should be posted to the users mailing list.
  • Remote security exploits MUST be sent to security@freeradius.org.
  • Defect - Crash or memory corruption.
  • Defect - Non compliance with a standards document, or incorrect API usage.
  • Defect - Unexpected behaviour (obvious or verified by project member).
  • Feature request.

See here for debugging instructions.

Defect/Feature description

How to reproduce issue

enable cache module - using the redis driver

Output of [radiusd|freeradius] -X showing issue occurring

Mon Sep  5 15:38:50 2016 : Info  : Ready to process requests
Mon Sep  5 15:38:52 2016 : Info  : (7,1)          cache_tls_session - Initiating cluster remap
Mon Sep  5 15:38:52 2016 : Error : ASSERT FAILED src/modules/rlm_cache/rlm_cache.c[632]: handle
CAUGHT SIGNAL: Aborted
Backtrace of last 35 frames:
/usr/lib/libfreeradius-radius.so(fr_fault+0x114)[0x7f655ac551b9]
/usr/lib/libfreeradius-server.so(rad_assert_fail+0x46)[0x7f655aec6c66]
/usr/lib/rlm_cache.so(+0x323c)[0x7f65567c923c]
/usr/sbin/radiusd[0x419301]
/usr/sbin/radiusd[0x41a7a7]
/usr/sbin/radiusd[0x41ae72]
/usr/sbin/radiusd(unlang_interpret+0xba)[0x41b230]
/usr/sbin/radiusd[0x42ebb8]
/usr/sbin/radiusd(process_authorize+0x22)[0x42ec00]
/usr/sbin/radiusd(tls_cache_process+0x161)[0x452fe8]
/usr/sbin/radiusd[0x45340f]
/lib64/libssl.so.10(ssl_update_cache+0x90)[0x7f655a425a90]
/lib64/libssl.so.10(ssl3_accept+0x728)[0x7f655a404298]
/lib64/libssl.so.10(ssl3_read_bytes+0xb20)[0x7f655a40ebf0]
/lib64/libssl.so.10(+0x299b4)[0x7f655a40b9b4]
/usr/sbin/radiusd(tls_session_handshake+0x1af)[0x45bab6]
/usr/lib/libfreeradius-eap.so(+0x73ae)[0x7f655aa3e3ae]
/usr/lib/libfreeradius-eap.so(eap_tls_process+0x2f2)[0x7f655aa3e763]
/usr/lib/rlm_eap_peap.so(+0x2147)[0x7f6555389147]
/usr/lib/rlm_eap.so(+0x36ef)[0x7f655599f6ef]
/usr/lib/rlm_eap.so(eap_method_select+0x423)[0x7f65559a007a]
/usr/lib/rlm_eap.so(+0x2450)[0x7f655599e450]
/usr/sbin/radiusd[0x419301]
/usr/sbin/radiusd[0x41a7a7]
/usr/sbin/radiusd[0x41ae72]
/usr/sbin/radiusd(unlang_interpret+0xba)[0x41b230]
/usr/sbin/radiusd[0x42ebb8]
/usr/sbin/radiusd(process_authenticate+0x22)[0x42ec24]
/usr/sbin/radiusd[0x410edb]
/usr/sbin/radiusd(rad_authenticate+0x56b)[0x4115eb]
/usr/sbin/radiusd[0x446e7b]
/usr/sbin/radiusd[0x447092]
/usr/sbin/radiusd[0x442fc7]
/lib64/libpthread.so.0(+0x7dc5)[0x7f655951cdc5]
/lib64/libc.so.6(clone+0x6d)[0x7f6559012ced]
Calling: gdb -silent -x /etc/raddb/panic.gdb /usr/sbin/radiusd 75576 2>&1 | tee /var/log/radius/gdb-radiusd-75576.log
Temporarily setting PR_DUMPABLE to 1
Resetting PR_DUMPABLE to 0
Panic action exited with 0
_EXIT(0) CALLED src/lib/debug.c[752]

Full backtrace from LLDB or GDB

# cat /var/log/radius/gdb-radiusd-75576.log
Reading symbols from /usr/sbin/radiusd...done.
Attaching to program: /usr/sbin/radiusd, process 75576
Reading symbols from /usr/lib/libfreeradius-server.so...done.
Loaded symbols for /usr/lib/libfreeradius-server.so
Reading symbols from /usr/lib/libfreeradius-radius.so...done.
Loaded symbols for /usr/lib/libfreeradius-radius.so
Reading symbols from /usr/lib/libfreeradius-eap.so...done.
Loaded symbols for /usr/lib/libfreeradius-eap.so
Reading symbols from /lib64/libcrypto.so.10...Reading symbols from /lib64/libcrypto.so.10...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libcrypto.so.10
Reading symbols from /lib64/libssl.so.10...Reading symbols from /lib64/libssl.so.10...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libssl.so.10
Reading symbols from /lib64/libtalloc.so.2...Reading symbols from /lib64/libtalloc.so.2...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libtalloc.so.2
Reading symbols from /lib64/libpcre.so.1...Reading symbols from /lib64/libpcre.so.1...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libpcre.so.1
Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/librt.so.1
Reading symbols from /lib64/libnsl.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libnsl.so.1
Reading symbols from /lib64/libresolv.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libresolv.so.2
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.
[New LWP 75705]
[New LWP 75704]
[New LWP 75703]
[New LWP 75702]
[New LWP 75701]
[New LWP 75700]
[New LWP 75699]
[New LWP 75698]
[New LWP 75697]
[New LWP 75696]
[New LWP 75695]
[New LWP 75694]
[New LWP 75693]
[New LWP 75692]
[New LWP 75691]
[New LWP 75690]
[New LWP 75689]
[New LWP 75688]
[New LWP 75687]
[New LWP 75686]
[New LWP 75685]
[New LWP 75684]
[New LWP 75683]
[New LWP 75682]
[New LWP 75681]
[New LWP 75680]
[New LWP 75679]
[New LWP 75678]
[New LWP 75677]
[New LWP 75676]
[New LWP 75675]
[New LWP 75674]
[New LWP 75673]
[New LWP 75672]
[New LWP 75671]
[New LWP 75670]
[New LWP 75669]
[New LWP 75668]
[New LWP 75667]
[New LWP 75666]
[New LWP 75665]
[New LWP 75664]
[New LWP 75663]
[New LWP 75662]
[New LWP 75661]
[New LWP 75660]
[New LWP 75659]
[New LWP 75658]
[New LWP 75657]
[New LWP 75656]
[New LWP 75655]
[New LWP 75654]
[New LWP 75653]
[New LWP 75652]
[New LWP 75651]
[New LWP 75650]
[New LWP 75649]
[New LWP 75648]
[New LWP 75647]
[New LWP 75646]
[New LWP 75645]
[New LWP 75644]
[New LWP 75643]
[New LWP 75642]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libcrypt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libcrypt.so.1
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/libpcap.so.1...Reading symbols from /lib64/libpcap.so.1...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libpcap.so.1
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib64/libz.so.1...Reading symbols from /lib64/libz.so.1...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libz.so.1
Reading symbols from /lib64/libgssapi_krb5.so.2...Reading symbols from /lib64/libgssapi_krb5.so.2...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libgssapi_krb5.so.2
Reading symbols from /lib64/libkrb5.so.3...Reading symbols from /lib64/libkrb5.so.3...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libkrb5.so.3
Reading symbols from /lib64/libcom_err.so.2...Reading symbols from /lib64/libcom_err.so.2...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libcom_err.so.2
Reading symbols from /lib64/libk5crypto.so.3...Reading symbols from /lib64/libk5crypto.so.3...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libk5crypto.so.3
Reading symbols from /lib64/libfreebl3.so...Reading symbols from /lib64/libfreebl3.so...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libfreebl3.so
Reading symbols from /lib64/libkrb5support.so.0...Reading symbols from /lib64/libkrb5support.so.0...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libkrb5support.so.0
Reading symbols from /lib64/libkeyutils.so.1...Reading symbols from /lib64/libkeyutils.so.1...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libkeyutils.so.1
Reading symbols from /lib64/libselinux.so.1...Reading symbols from /lib64/libselinux.so.1...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libselinux.so.1
Reading symbols from /lib64/liblzma.so.5...Reading symbols from /lib64/liblzma.so.5...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/liblzma.so.5
Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libgcc_s.so.1
Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libnss_files.so.2
Reading symbols from /usr/lib/rlm_always.so...done.
Loaded symbols for /usr/lib/rlm_always.so
Reading symbols from /usr/lib/rlm_attr_filter.so...done.
Loaded symbols for /usr/lib/rlm_attr_filter.so
Reading symbols from /usr/lib/rlm_cache.so...done.
Loaded symbols for /usr/lib/rlm_cache.so
Reading symbols from /usr/lib/rlm_chap.so...done.
Loaded symbols for /usr/lib/rlm_chap.so
Reading symbols from /usr/lib/rlm_detail.so...done.
Loaded symbols for /usr/lib/rlm_detail.so
Reading symbols from /usr/lib/rlm_digest.so...done.
Loaded symbols for /usr/lib/rlm_digest.so
Reading symbols from /usr/lib/rlm_dhcp.so...done.
Loaded symbols for /usr/lib/rlm_dhcp.so
Reading symbols from /usr/lib/libfreeradius-dhcp.so...done.
Loaded symbols for /usr/lib/libfreeradius-dhcp.so
Reading symbols from /usr/lib/rlm_dynamic_clients.so...done.
Loaded symbols for /usr/lib/rlm_dynamic_clients.so
Reading symbols from /usr/lib/rlm_eap.so...done.
Loaded symbols for /usr/lib/rlm_eap.so
Reading symbols from /usr/lib/rlm_eap_tls.so...done.
Loaded symbols for /usr/lib/rlm_eap_tls.so
Reading symbols from /usr/lib/rlm_eap_ttls.so...done.
Loaded symbols for /usr/lib/rlm_eap_ttls.so
Reading symbols from /usr/lib/rlm_eap_peap.so...done.
Loaded symbols for /usr/lib/rlm_eap_peap.so
Reading symbols from /usr/lib/rlm_eap_mschapv2.so...done.
Loaded symbols for /usr/lib/rlm_eap_mschapv2.so
Reading symbols from /usr/lib/rlm_exec.so...done.
Loaded symbols for /usr/lib/rlm_exec.so
Reading symbols from /usr/lib/rlm_expiration.so...done.
Loaded symbols for /usr/lib/rlm_expiration.so
Reading symbols from /usr/lib/rlm_expr.so...done.
Loaded symbols for /usr/lib/rlm_expr.so
Reading symbols from /usr/lib/rlm_files.so...done.
Loaded symbols for /usr/lib/rlm_files.so
Reading symbols from /usr/lib/rlm_linelog.so...done.
Loaded symbols for /usr/lib/rlm_linelog.so
Reading symbols from /usr/lib/rlm_logintime.so...done.
Loaded symbols for /usr/lib/rlm_logintime.so
Reading symbols from /usr/lib/rlm_mschap.so...done.
Loaded symbols for /usr/lib/rlm_mschap.so
Reading symbols from /usr/local/samba/lib/libwbclient.so.0...(no debugging symbols found)...done.
Loaded symbols for /usr/local/samba/lib/libwbclient.so.0
Reading symbols from /usr/local/samba/lib/private/libwinbind-client-samba4.so...(no debugging symbols found)...done.
Loaded symbols for /usr/local/samba/lib/private/libwinbind-client-samba4.so
Reading symbols from /usr/local/samba/lib/private/libreplace-samba4.so...(no debugging symbols found)...done.
Loaded symbols for /usr/local/samba/lib/private/libreplace-samba4.so
Reading symbols from /usr/lib/rlm_pap.so...done.
Loaded symbols for /usr/lib/rlm_pap.so
Reading symbols from /usr/lib/rlm_passwd.so...done.
Loaded symbols for /usr/lib/rlm_passwd.so
Reading symbols from /usr/lib/rlm_preprocess.so...done.
Loaded symbols for /usr/lib/rlm_preprocess.so
Reading symbols from /usr/lib/rlm_realm.so...done.
Loaded symbols for /usr/lib/rlm_realm.so
Reading symbols from /usr/lib/rlm_replicate.so...done.
Loaded symbols for /usr/lib/rlm_replicate.so
Reading symbols from /usr/lib/rlm_soh.so...done.
Loaded symbols for /usr/lib/rlm_soh.so
Reading symbols from /usr/lib/rlm_radutmp.so...done.
Loaded symbols for /usr/lib/rlm_radutmp.so
Reading symbols from /usr/lib/rlm_unpack.so...done.
Loaded symbols for /usr/lib/rlm_unpack.so
Reading symbols from /usr/lib/rlm_utf8.so...done.
Loaded symbols for /usr/lib/rlm_utf8.so
Reading symbols from /usr/lib/rlm_ldap.so...done.
Loaded symbols for /usr/lib/rlm_ldap.so
Reading symbols from /usr/local/lib/libldap-2.4.so.2...done.
Loaded symbols for /usr/local/lib/libldap-2.4.so.2
Reading symbols from /usr/local/lib/liblber-2.4.so.2...done.
Loaded symbols for /usr/local/lib/liblber-2.4.so.2
Reading symbols from /lib64/libsasl2.so.3...Reading symbols from /lib64/libsasl2.so.3...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libsasl2.so.3
Reading symbols from /usr/lib/rlm_sql.so...done.
Loaded symbols for /usr/lib/rlm_sql.so
Reading symbols from /usr/lib/rlm_sql_postgresql.so...done.
Loaded symbols for /usr/lib/rlm_sql_postgresql.so
Reading symbols from /lib64/libpq.so.5...Reading symbols from /lib64/libpq.so.5...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libpq.so.5
Reading symbols from /usr/local/lib/libldap_r-2.4.so.2...done.
Loaded symbols for /usr/local/lib/libldap_r-2.4.so.2
Reading symbols from /usr/lib/rlm_krb5.so...done.
Loaded symbols for /usr/lib/rlm_krb5.so
Reading symbols from /usr/lib/rlm_winbind.so...done.
Loaded symbols for /usr/lib/rlm_winbind.so
Reading symbols from /lib64/libnss_dns.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libnss_dns.so.2
Reading symbols from /usr/lib/rlm_cache_rbtree.so...done.
Loaded symbols for /usr/lib/rlm_cache_rbtree.so
Reading symbols from /usr/lib/rlm_cache_redis.so...done.
Loaded symbols for /usr/lib/rlm_cache_redis.so
Reading symbols from /lib64/libhiredis.so.0.12...Reading symbols from /lib64/libhiredis.so.0.12...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libhiredis.so.0.12
Reading symbols from /usr/lib/libfreeradius-redis.so...done.
Loaded symbols for /usr/lib/libfreeradius-redis.so
0x00007f655900a413 in select () from /lib64/libc.so.6
No symbol table info available.
No symbol table info available.

Thread 65 (Thread 0x7f65503a7700 (LWP 75642)):
#0  0x00007f6559522790 in sem_wait () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x0000000000442db3 in request_handler_thread (arg=0x124ef030)
    at src/main/threads.c:625
        now = 0
        request = 0x0
        thread = 0x124ef030
#2  0x00007f655951cdc5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#3  0x00007f6559012ced in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 64 (Thread 0x7f654fba6700 (LWP 75643)):
#0  0x00007f6559522790 in sem_wait () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x0000000000442db3 in request_handler_thread (arg=0x124ef370)
    at src/main/threads.c:625
        now = 0
        request = 0x0
        thread = 0x124ef370
#2  0x00007f655951cdc5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#3  0x00007f6559012ced in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 63 (Thread 0x7f654f3a5700 (LWP 75644)):
#0  0x00007f6559522790 in sem_wait () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x0000000000442db3 in request_handler_thread (arg=0x124ef6b0)
    at src/main/threads.c:625
        now = 0
        request = 0x0
        thread = 0x124ef6b0
#2  0x00007f655951cdc5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#3  0x00007f6559012ced in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 62 (Thread 0x7f654eba4700 (LWP 75645)):
#0  0x00007f6559522790 in sem_wait () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x0000000000442db3 in request_handler_thread (arg=0x124ef9f0)
    at src/main/threads.c:625
        now = 0
        request = 0x0
        thread = 0x124ef9f0
#2  0x00007f655951cdc5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#3  0x00007f6559012ced in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 61 (Thread 0x7f654e3a3700 (LWP 75646)):
#0  0x00007f6559522790 in sem_wait () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x0000000000442db3 in request_handler_thread (arg=0x124efd30)
    at src/main/threads.c:625
        now = 0
        request = 0x0
        thread = 0x124efd30
#2  0x00007f655951cdc5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#3  0x00007f6559012ced in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 60 (Thread 0x7f654dba2700 (LWP 75647)):
#0  0x00007f6559522790 in sem_wait () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x0000000000442db3 in request_handler_thread (arg=0x124f0070)
    at src/main/threads.c:625
        now = 0
        request = 0x0
        thread = 0x124f0070
#2  0x00007f655951cdc5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#3  0x00007f6559012ced in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 59 (Thread 0x7f654d3a1700 (LWP 75648)):
#0  0x00007f6559522790 in sem_wait () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x0000000000442db3 in request_handler_thread (arg=0x124f03b0)
    at src/main/threads.c:625
        now = 0
        request = 0x0
        thread = 0x124f03b0
#2  0x00007f655951cdc5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#3  0x00007f6559012ced in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 58 (Thread 0x7f654cba0700 (LWP 75649)):
#0  0x00007f6559522790 in sem_wait () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x0000000000442db3 in request_handler_thread (arg=0x124f06f0)
    at src/main/threads.c:625
        now = 0
        request = 0x0
        thread = 0x124f06f0
#2  0x00007f655951cdc5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#3  0x00007f6559012ced in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 57 (Thread 0x7f654c39f700 (LWP 75650)):
#0  0x00007f6559522790 in sem_wait () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x0000000000442db3 in request_handler_thread (arg=0x124f0a30)
    at src/main/threads.c:625
        now = 0
        request = 0x0
        thread = 0x124f0a30
#2  0x00007f655951cdc5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#3  0x00007f6559012ced in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 56 (Thread 0x7f654bb9e700 (LWP 75651)):
#0  0x00007f6559522790 in sem_wait () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x0000000000442db3 in request_handler_thread (arg=0x124f0d70)
    at src/main/threads.c:625
        now = 0
        request = 0x0
        thread = 0x124f0d70
#2  0x00007f655951cdc5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#3  0x00007f6559012ced in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 55 (Thread 0x7f654b39d700 (LWP 75652)):
#0  0x00007f6559522790 in sem_wait () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x0000000000442db3 in request_handler_thread (arg=0x124f10b0)
    at src/main/threads.c:625
        now = 0
        request = 0x0
        thread = 0x124f10b0
#2  0x00007f655951cdc5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#3  0x00007f6559012ced in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 54 (Thread 0x7f654ab9c700 (LWP 75653)):
#0  0x00007f6559522790 in sem_wait () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x0000000000442db3 in request_handler_thread (arg=0x124f13f0)
    at src/main/threads.c:625
        now = 0
        request = 0x0
        thread = 0x124f13f0
#2  0x00007f655951cdc5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#3  0x00007f6559012ced in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 53 (Thread 0x7f654a39b700 (LWP 75654)):
#0  0x00007f6559522790 in sem_wait () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x0000000000442db3 in request_handler_thread (arg=0x124f1730)
    at src/main/threads.c:625
        now = 0
        request = 0x0
        thread = 0x124f1730
#2  0x00007f655951cdc5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#3  0x00007f6559012ced in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 52 (Thread 0x7f6549b9a700 (LWP 75655)):
#0  0x00007f6559522790 in sem_wait () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x0000000000442db3 in request_handler_thread (arg=0x124f1a70)
    at src/main/threads.c:625
        now = 0
        request = 0x0
        thread = 0x124f1a70
#2  0x00007f655951cdc5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#3  0x00007f6559012ced in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 51 (Thread 0x7f6549399700 (LWP 75656)):
#0  0x00007f6559522790 in sem_wait () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x0000000000442db3 in request_handler_thread (arg=0x124f1db0)
    at src/main/threads.c:625
        now = 0
        request = 0x0
        thread = 0x124f1db0
#2  0x00007f655951cdc5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#3  0x00007f6559012ced in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 50 (Thread 0x7f6548b98700 (LWP 75657)):
#0  0x00007f6559522790 in sem_wait () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x0000000000442db3 in request_handler_thread (arg=0x124f20f0)
    at src/main/threads.c:625
        now = 0
        request = 0x0
        thread = 0x124f20f0
#2  0x00007f655951cdc5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#3  0x00007f6559012ced in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 49 (Thread 0x7f6548397700 (LWP 75658)):
#0  0x00007f6559522790 in sem_wait () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x0000000000442db3 in request_handler_thread (arg=0x124f2430)
    at src/main/threads.c:625
        now = 0
        request = 0x0
        thread = 0x124f2430
#2  0x00007f655951cdc5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#3  0x00007f6559012ced in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 48 (Thread 0x7f6547b96700 (LWP 75659)):
#0  0x00007f6559522790 in sem_wait () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x0000000000442db3 in request_handler_thread (arg=0x124f2770)
    at src/main/threads.c:625
        now = 0
        request = 0x0
        thread = 0x124f2770
#2  0x00007f655951cdc5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#3  0x00007f6559012ced in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 47 (Thread 0x7f6547395700 (LWP 75660)):
#0  0x00007f6559522790 in sem_wait () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x0000000000442db3 in request_handler_thread (arg=0x124f2ab0)
    at src/main/threads.c:625
        now = 0
        request = 0x0
        thread = 0x124f2ab0
#2  0x00007f655951cdc5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#3  0x00007f6559012ced in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 46 (Thread 0x7f6546b94700 (LWP 75661)):
#0  0x00007f6559522790 in sem_wait () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x0000000000442db3 in request_handler_thread (arg=0x124f2df0)
    at src/main/threads.c:625
        now = 0
        request = 0x0
        thread = 0x124f2df0
#2  0x00007f655951cdc5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#3  0x00007f6559012ced in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 45 (Thread 0x7f6546393700 (LWP 75662)):
#0  0x00007f6559522790 in sem_wait () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x0000000000442db3 in request_handler_thread (arg=0x124f3130)
    at src/main/threads.c:625
        now = 0
        request = 0x0
        thread = 0x124f3130
#2  0x00007f655951cdc5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#3  0x00007f6559012ced in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 44 (Thread 0x7f6545b92700 (LWP 75663)):
#0  0x00007f6559522790 in sem_wait () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x0000000000442db3 in request_handler_thread (arg=0x124f3470)
    at src/main/threads.c:625
        now = 0
        request = 0x0
        thread = 0x124f3470
#2  0x00007f655951cdc5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#3  0x00007f6559012ced in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 43 (Thread 0x7f6545391700 (LWP 75664)):
#0  0x00007f6559522790 in sem_wait () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x0000000000442db3 in request_handler_thread (arg=0x124f37b0)
    at src/main/threads.c:625
        now = 0
        request = 0x0
        thread = 0x124f37b0
#2  0x00007f655951cdc5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#3  0x00007f6559012ced in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 42 (Thread 0x7f6544b90700 (LWP 75665)):
#0  0x00007f6559522790 in sem_wait () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x0000000000442db3 in request_handler_thread (arg=0x124f3af0)
    at src/main/threads.c:625
        now = 0
        request = 0x0
        thread = 0x124f3af0
#2  0x00007f655951cdc5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#3  0x00007f6559012ced in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 41 (Thread 0x7f654438f700 (LWP 75666)):
#0  0x00007f6559522790 in sem_wait () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x0000000000442db3 in request_handler_thread (arg=0x124f3e30)
    at src/main/threads.c:625
        now = 0
        request = 0x0
        thread = 0x124f3e30
#2  0x00007f655951cdc5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#3  0x00007f6559012ced in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 40 (Thread 0x7f6543b8e700 (LWP 75667)):
#0  0x00007f6559522790 in sem_wait () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x0000000000442db3 in request_handler_thread (arg=0x124f4170)
    at src/main/threads.c:625
        now = 0
        request = 0x0
        thread = 0x124f4170
#2  0x00007f655951cdc5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#3  0x00007f6559012ced in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 39 (Thread 0x7f654338d700 (LWP 75668)):
#0  0x00007f6559522790 in sem_wait () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x0000000000442db3 in request_handler_thread (arg=0x124f44b0)
    at src/main/threads.c:625
        now = 0
        request = 0x0
        thread = 0x124f44b0
#2  0x00007f655951cdc5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#3  0x00007f6559012ced in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 38 (Thread 0x7f6542b8c700 (LWP 75669)):
#0  0x00007f6559522790 in sem_wait () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x0000000000442db3 in request_handler_thread (arg=0x124f47f0)
    at src/main/threads.c:625
        now = 0
        request = 0x0
        thread = 0x124f47f0
#2  0x00007f655951cdc5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#3  0x00007f6559012ced in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 37 (Thread 0x7f654238b700 (LWP 75670)):
#0  0x00007f6559522790 in sem_wait () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x0000000000442db3 in request_handler_thread (arg=0x124f4b30)
    at src/main/threads.c:625
        now = 0
        request = 0x0
        thread = 0x124f4b30
#2  0x00007f655951cdc5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#3  0x00007f6559012ced in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 36 (Thread 0x7f6541b8a700 (LWP 75671)):
#0  0x00007f6559522790 in sem_wait () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x0000000000442db3 in request_handler_thread (arg=0x124f4e70)
    at src/main/threads.c:625
        now = 0
        request = 0x0
        thread = 0x124f4e70
#2  0x00007f655951cdc5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#3  0x00007f6559012ced in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 35 (Thread 0x7f6541389700 (LWP 75672)):
#0  0x00007f6559522790 in sem_wait () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x0000000000442db3 in request_handler_thread (arg=0x124f51b0)
    at src/main/threads.c:625
        now = 0
        request = 0x0
        thread = 0x124f51b0
#2  0x00007f655951cdc5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#3  0x00007f6559012ced in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 34 (Thread 0x7f6540b88700 (LWP 75673)):
#0  0x00007f6559522790 in sem_wait () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x0000000000442db3 in request_handler_thread (arg=0x124f54f0)
    at src/main/threads.c:625
        now = 0
        request = 0x0
        thread = 0x124f54f0
#2  0x00007f655951cdc5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#3  0x00007f6559012ced in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 33 (Thread 0x7f6540387700 (LWP 75674)):
#0  0x00007f6559522790 in sem_wait () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x0000000000442db3 in request_handler_thread (arg=0x124f5830)
    at src/main/threads.c:625
        now = 0
        request = 0x0
        thread = 0x124f5830
#2  0x00007f655951cdc5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#3  0x00007f6559012ced in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 32 (Thread 0x7f653fb86700 (LWP 75675)):
#0  0x00007f6559522790 in sem_wait () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x0000000000442db3 in request_handler_thread (arg=0x124f5b70)
    at src/main/threads.c:625
        now = 0
        request = 0x0
        thread = 0x124f5b70
#2  0x00007f655951cdc5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#3  0x00007f6559012ced in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 31 (Thread 0x7f653f385700 (LWP 75676)):
#0  0x00007f6559522790 in sem_wait () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x0000000000442db3 in request_handler_thread (arg=0x124f5eb0)
    at src/main/threads.c:625
        now = 0
        request = 0x0
        thread = 0x124f5eb0
#2  0x00007f655951cdc5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#3  0x00007f6559012ced in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 30 (Thread 0x7f653eb84700 (LWP 75677)):
#0  0x00007f6559522790 in sem_wait () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x0000000000442db3 in request_handler_thread (arg=0x124f61f0)
    at src/main/threads.c:625
        now = 0
        request = 0x0
        thread = 0x124f61f0
#2  0x00007f655951cdc5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#3  0x00007f6559012ced in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 29 (Thread 0x7f653e383700 (LWP 75678)):
#0  0x00007f6559522790 in sem_wait () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x0000000000442db3 in request_handler_thread (arg=0x124f6530)
    at src/main/threads.c:625
        now = 0
        request = 0x0
        thread = 0x124f6530
#2  0x00007f655951cdc5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#3  0x00007f6559012ced in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 28 (Thread 0x7f653db82700 (LWP 75679)):
#0  0x00007f6559522790 in sem_wait () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x0000000000442db3 in request_handler_thread (arg=0x124f6870)
    at src/main/threads.c:625
        now = 0
        request = 0x0
        thread = 0x124f6870
#2  0x00007f655951cdc5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#3  0x00007f6559012ced in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 27 (Thread 0x7f653d381700 (LWP 75680)):
#0  0x00007f6559522790 in sem_wait () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x0000000000442db3 in request_handler_thread (arg=0x124f6bb0)
    at src/main/threads.c:625
        now = 0
        request = 0x0
        thread = 0x124f6bb0
#2  0x00007f655951cdc5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#3  0x00007f6559012ced in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 26 (Thread 0x7f653cb80700 (LWP 75681)):
#0  0x00007f6559522790 in sem_wait () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x0000000000442db3 in request_handler_thread (arg=0x124f6ef0)
    at src/main/threads.c:625
        now = 0
        request = 0x0
        thread = 0x124f6ef0
#2  0x00007f655951cdc5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#3  0x00007f6559012ced in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 25 (Thread 0x7f653c37f700 (LWP 75682)):
#0  0x00007f6559522790 in sem_wait () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x0000000000442db3 in request_handler_thread (arg=0x124f7230)
    at src/main/threads.c:625
        now = 0
        request = 0x0
        thread = 0x124f7230
#2  0x00007f655951cdc5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#3  0x00007f6559012ced in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 24 (Thread 0x7f653bb7e700 (LWP 75683)):
#0  0x00007f6559522790 in sem_wait () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x0000000000442db3 in request_handler_thread (arg=0x124f7570)
    at src/main/threads.c:625
        now = 0
        request = 0x0
        thread = 0x124f7570
#2  0x00007f655951cdc5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#3  0x00007f6559012ced in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 23 (Thread 0x7f653b37d700 (LWP 75684)):
#0  0x00007f6559522790 in sem_wait () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x0000000000442db3 in request_handler_thread (arg=0x124f78b0)
    at src/main/threads.c:625
        now = 0
        request = 0x0
        thread = 0x124f78b0
#2  0x00007f655951cdc5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#3  0x00007f6559012ced in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 22 (Thread 0x7f653ab7c700 (LWP 75685)):
#0  0x00007f6559522790 in sem_wait () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x0000000000442db3 in request_handler_thread (arg=0x124f7bf0)
    at src/main/threads.c:625
        now = 0
        request = 0x0
        thread = 0x124f7bf0
#2  0x00007f655951cdc5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#3  0x00007f6559012ced in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 21 (Thread 0x7f653a37b700 (LWP 75686)):
#0  0x00007f6559522790 in sem_wait () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x0000000000442db3 in request_handler_thread (arg=0x124f7f30)
    at src/main/threads.c:625
        now = 0
        request = 0x0
        thread = 0x124f7f30
#2  0x00007f655951cdc5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#3  0x00007f6559012ced in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 20 (Thread 0x7f6539b7a700 (LWP 75687)):
#0  0x00007f6559522790 in sem_wait () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x0000000000442db3 in request_handler_thread (arg=0x124f8270)
    at src/main/threads.c:625
        now = 0
        request = 0x0
        thread = 0x124f8270
#2  0x00007f655951cdc5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#3  0x00007f6559012ced in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 19 (Thread 0x7f6539379700 (LWP 75688)):
#0  0x00007f6559522790 in sem_wait () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x0000000000442db3 in request_handler_thread (arg=0x124f85b0)
    at src/main/threads.c:625
        now = 0
        request = 0x0
        thread = 0x124f85b0
#2  0x00007f655951cdc5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#3  0x00007f6559012ced in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 18 (Thread 0x7f6538b78700 (LWP 75689)):
#0  0x00007f6559522790 in sem_wait () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x0000000000442db3 in request_handler_thread (arg=0x124f88f0)
    at src/main/threads.c:625
        now = 0
        request = 0x0
        thread = 0x124f88f0
#2  0x00007f655951cdc5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#3  0x00007f6559012ced in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 17 (Thread 0x7f6538377700 (LWP 75690)):
#0  0x00007f6559522790 in sem_wait () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x0000000000442db3 in request_handler_thread (arg=0x124f8c30)
    at src/main/threads.c:625
        now = 0
        request = 0x0
        thread = 0x124f8c30
#2  0x00007f655951cdc5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#3  0x00007f6559012ced in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 16 (Thread 0x7f6537b76700 (LWP 75691)):
#0  0x00007f6559522790 in sem_wait () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x0000000000442db3 in request_handler_thread (arg=0x124f8f70)
    at src/main/threads.c:625
        now = 0
        request = 0x0
        thread = 0x124f8f70
#2  0x00007f655951cdc5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#3  0x00007f6559012ced in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 15 (Thread 0x7f6537375700 (LWP 75692)):
#0  0x00007f6559522790 in sem_wait () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x0000000000442db3 in request_handler_thread (arg=0x124f92b0)
    at src/main/threads.c:625
        now = 0
        request = 0x0
        thread = 0x124f92b0
#2  0x00007f655951cdc5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#3  0x00007f6559012ced in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 14 (Thread 0x7f6536b74700 (LWP 75693)):
#0  0x00007f6559522790 in sem_wait () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x0000000000442db3 in request_handler_thread (arg=0x124f95f0)
    at src/main/threads.c:625
        now = 0
        request = 0x0
        thread = 0x124f95f0
#2  0x00007f655951cdc5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#3  0x00007f6559012ced in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 13 (Thread 0x7f6536373700 (LWP 75694)):
#0  0x00007f6559522790 in sem_wait () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x0000000000442db3 in request_handler_thread (arg=0x124f9930)
    at src/main/threads.c:625
        now = 0
        request = 0x0
        thread = 0x124f9930
#2  0x00007f655951cdc5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#3  0x00007f6559012ced in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 12 (Thread 0x7f6535b72700 (LWP 75695)):
#0  0x00007f6559522790 in sem_wait () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x0000000000442db3 in request_handler_thread (arg=0x124f9c70)
    at src/main/threads.c:625
        now = 0
        request = 0x0
        thread = 0x124f9c70
#2  0x00007f655951cdc5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#3  0x00007f6559012ced in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 11 (Thread 0x7f6535371700 (LWP 75696)):
#0  0x00007f6559522790 in sem_wait () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x0000000000442db3 in request_handler_thread (arg=0x124f9fb0)
    at src/main/threads.c:625
        now = 0
        request = 0x0
        thread = 0x124f9fb0
#2  0x00007f655951cdc5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#3  0x00007f6559012ced in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 10 (Thread 0x7f6534b70700 (LWP 75697)):
#0  0x00007f6559522790 in sem_wait () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x0000000000442db3 in request_handler_thread (arg=0x124fa2f0)
    at src/main/threads.c:625
        now = 0
        request = 0x0
        thread = 0x124fa2f0
#2  0x00007f655951cdc5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#3  0x00007f6559012ced in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 9 (Thread 0x7f653436f700 (LWP 75698)):
#0  0x00007f6559522790 in sem_wait () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x0000000000442db3 in request_handler_thread (arg=0x124fa630)
    at src/main/threads.c:625
        now = 0
        request = 0x0
        thread = 0x124fa630
#2  0x00007f655951cdc5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#3  0x00007f6559012ced in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 8 (Thread 0x7f6533b6e700 (LWP 75699)):
#0  0x00007f6559522790 in sem_wait () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x0000000000442db3 in request_handler_thread (arg=0x124fa970)
    at src/main/threads.c:625
        now = 0
        request = 0x0
        thread = 0x124fa970
#2  0x00007f655951cdc5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#3  0x00007f6559012ced in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 7 (Thread 0x7f653336d700 (LWP 75700)):
#0  0x00007f6559522790 in sem_wait () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x0000000000442db3 in request_handler_thread (arg=0x124facb0)
    at src/main/threads.c:625
        now = 0
        request = 0x0
        thread = 0x124facb0
#2  0x00007f655951cdc5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#3  0x00007f6559012ced in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 6 (Thread 0x7f6532b6c700 (LWP 75701)):
#0  0x00007f6559522790 in sem_wait () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x0000000000442db3 in request_handler_thread (arg=0x124faff0)
    at src/main/threads.c:625
        now = 0
        request = 0x0
        thread = 0x124faff0
#2  0x00007f655951cdc5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#3  0x00007f6559012ced in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 5 (Thread 0x7f653236b700 (LWP 75702)):
#0  0x00007f6559522790 in sem_wait () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x0000000000442db3 in request_handler_thread (arg=0x124fb330)
    at src/main/threads.c:625
        now = 0
        request = 0x0
        thread = 0x124fb330
#2  0x00007f655951cdc5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#3  0x00007f6559012ced in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 4 (Thread 0x7f6531b6a700 (LWP 75703)):
#0  0x00007f6559522790 in sem_wait () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x0000000000442db3 in request_handler_thread (arg=0x124fb670)
    at src/main/threads.c:625
        now = 0
        request = 0x0
        thread = 0x124fb670
#2  0x00007f655951cdc5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#3  0x00007f6559012ced in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 3 (Thread 0x7f6531369700 (LWP 75704)):
#0  0x00007f6558fd9b59 in waitpid () from /lib64/libc.so.6
No symbol table info available.
#1  0x00007f6558f5e092 in do_system () from /lib64/libc.so.6
No symbol table info available.
#2  0x00007f6558f5e441 in system () from /lib64/libc.so.6
No symbol table info available.
#3  0x00007f655ac55375 in fr_fault (sig=6) at src/lib/debug.c:734
        disable = true
        cmd = "gdb -silent -x /etc/raddb/panic.gdb /usr/sbin/radiusd 75576 2>&1 | tee /var/log/radius/gdb-radiusd-75576.log", '\000' <repeats 423 times>
        out = 0x7f6531365758 ".log"
        left = 428
        ret = 45
        p = 0x7f655ae9a5c2 <panic_action+98> ".log"
        q = 0x0
        code = 0
#4  0x00007f655aec6c66 in rad_assert_fail (
    file=0x7f65567ca7d0 "src/modules/rlm_cache/rlm_cache.c", line=632, 
    expr=0x7f65567cabaa "handle") at src/main/util.c:468
No locals.
#5  0x00007f65567c923c in mod_cache_it (instance=0x5770120, 
    request=0x12512770) at src/modules/rlm_cache/rlm_cache.c:632
        c = 0x0
        inst = 0x5770120
        handle = 0x0
        cursor = {first = 0x7f652c02dc80, found = 0x7f652c02dc80, last = 0x0, 
          current = 0x12512770, next = 0x0}
        vp = 0x7f652c02dc80
        merge = true
        insert = true
        expire = true
        set_ttl = false
        exists = 0
        buffer = "p\206IWe\177\000\000\220\363\006\002\000\000\000\000\240Z61e\177\000\000\374\300\305Ze\177\000\000\250\206IWe\177\000\000`\235\332\001\000\000\000\000\200Z61e\177\000\000Auth-Type\000\006\002\000\000\000\000Cache-TTL\000Action\000\363\006\002\000\000\000\000\060Z61e\177\000\000TLS-Cache-Action\000\023\332\001\000\000\000\000\320\016\000,e\177\000\000pZ61e\177\000\000\320\016\000,e\177\000\000\200Z61e\177\000\000\002\003\000\000\000\000\000\000[\307\305Ze\177\000\000\320\016\000,e\177\000\000\240Z61e\177\000\000\002\003\000\000\000\000\000\000"...
        key = 0x7f652c030910 "\305f\264P (\347\372\003૫\215L\344?\256o=\256\366&\226V\031\327.73\310\361\377\220"
        key_len = 32
        rcode = RLM_MODULE_NOTFOUND
        ttl = 3600
#6  0x0000000000419301 in unlang_module (component=MOD_AUTHORIZE, 
    sp=0x123d4d80, request=0x12512770) at src/main/interpreter.c:97
        blocked = 0
#7  0x000000000041a7a7 in unlang_single (request=0x12512770, 
    stack=0x7f6531365f10, presult=0x7f6531365ec4, priority=0x7f6531365ec8)
    at src/main/interpreter.c:679
        sp = 0x123d4d80
        entry = 0x7f6531365fc8
        c = 0x123d4d80
#8  0x000000000041ae72 in unlang_run (request=0x12512770, 
    stack=0x7f6531365f10, presult=0x7f6531366a18, ppriority=0x7f6531366a1c)
    at src/main/interpreter.c:885
        c = 0x123d4d80
        priority = 2
        result = RLM_MODULE_NOOP
        entry = 0x7f6531365fc8
        action = UNLANG_CALCULATE_RESULT
#9  0x000000000041b230 in unlang_interpret (request=0x12512770, c=0x123d4650, 
    component=MOD_AUTHORIZE) at src/main/interpreter.c:1036
        priority = 0
        result = RLM_MODULE_NOTFOUND
        stack = {depth = 2, entry = {{result = RLM_MODULE_REJECT, 
              priority = 0, unwind = MOD_NULL, do_next_sibling = false, 
              was_if = false, if_taken = false, resume = false, c = 0x0, {
                foreach = {cursor = {first = 0x0, found = 0x0, last = 0x0, 
                    current = 0x0, next = 0x0}, vps = 0x0, variable = 0x0, 
                  depth = 0}, redundant = {child = 0x0, found = 0x0}}}, {
              result = RLM_MODULE_NOTFOUND, priority = 0, unwind = MOD_NULL, 
              do_next_sibling = true, was_if = false, if_taken = false, 
              resume = false, c = 0x123d4650, {foreach = {cursor = {
                    first = 0x0, found = 0x0, last = 0x0, current = 0x0, 
                    next = 0x0}, vps = 0x0, variable = 0x0, depth = 0}, 
                redundant = {child = 0x0, found = 0x0}}}, {
              result = RLM_MODULE_NOOP, priority = 2, unwind = MOD_NULL, 
              do_next_sibling = true, was_if = false, if_taken = false, 
              resume = false, c = 0x123d4d80, {foreach = {cursor = {
                    first = 0x0, found = 0x0, last = 0x0, current = 0x0, 
                    next = 0x0}, vps = 0x0, variable = 0x0, depth = 0}, 
                redundant = {child = 0x0, found = 0x0}}}, {
              result = RLM_MODULE_REJECT, priority = 0, unwind = MOD_NULL, 
              do_next_sibling = false, was_if = false, if_taken = false, 
              resume = false, c = 0x0, {foreach = {cursor = {first = 0x0, 
                    found = 0x0, last = 0x0, current = 0x0, next = 0x0}, 
                  vps = 0x0, variable = 0x0, depth = 0}, redundant = {
                  child = 0x0, found = 0x0}}} <repeats 29 times>}}
#10 0x000000000042ebb8 in indexed_modcall (comp=MOD_AUTHORIZE, idx=1002, 
    request=0x12512770) at src/main/modules.c:2067
        module = 0x0
        component = 0x46a290 "authenticate"
        rcode = 32613
        list = 0x123d4650
        server = 0x123d38e0
        cs = 0x2268740
#11 0x000000000042ec00 in process_authorize (autz_type=1002, 
    request=0x12512770) at src/main/modules.c:2082
No locals.
#12 0x0000000000452fe8 in tls_cache_process (request=0x12512770, 
    virtual_server=0x235a330 "tls-cache", autz_type=2)
    at src/main/tls/cache.c:136
        rcode = 32613
        vp = 0x7f652c02dbc0
        server = 0x22f5070 "lboro"
        module = 0x7f655538c8b6 "eap_peap"
        component = 0x46a290 "authenticate"
#13 0x000000000045340f in tls_cache_write (ssl=0x7f652c000ed0, 
    sess=0x7f652c0255a0) at src/main/tls/cache.c:250
        conf = 0x2358510
        request = 0x12512770
        len = 133
        rcode = 133
        slen = 32
        p = 0x7f652c026915 "BB", <incomplete sequence \305>
        data = 0x0
        vp = 0x7f652c026980
        buffer = "\305f\264P (\347\372\003૫\215L\344?\256o=\256\366&\226V\031\327.73\310\361\377()\000,e\177\000\000\061\177AZe\177\000\000\000\001\000\000\000\000\000\000\020\000\000\000 ", '\000' <repeats 11 times>, "\a\223\371Xe\177\000\000\000\000\000\000\000\000\000\000\001\000\000\000\020\000\000\000\200.\000,e\177\000\000\a\223\371Xe\177\000\000\271Ҡ1\326,\"\337\001\000\000\000e\177\000\000\026\000\000\000\000\000\000\000\320\361\331\001", '\000' <repeats 12 times>, "\240\062\002,e\177\000\000\240\062\002,e\177\000\000\240\062\002,e\177\000\000\247\276G\000\000\000\000\000\300\061\002,e\177\000\000\300\061\002,e\177\000\000\300\061\002"...
#14 0x00007f655a425a90 in ssl_update_cache () from /lib64/libssl.so.10
No symbol table info available.
#15 0x00007f655a404298 in ssl3_accept () from /lib64/libssl.so.10
No symbol table info available.
#16 0x00007f655a40ebf0 in ssl3_read_bytes () from /lib64/libssl.so.10
No symbol table info available.
#17 0x00007f655a40b9b4 in ssl3_read_internal () from /lib64/libssl.so.10
No symbol table info available.
#18 0x000000000045bab6 in tls_session_handshake (request=0x12512770, 
    session=0x7f652c002e80) at src/main/tls/session.c:1259
        ret = 134
#19 0x00007f655aa3e3ae in eap_tls_handshake (eap_session=0x7f652c000a60)
    at src/modules/rlm_eap/libeap/eap_tls.c:781
        request = 0x12512770
        eap_tls_session = 0x7f652c000e20
        tls_session = 0x7f652c002e80
#20 0x00007f655aa3e763 in eap_tls_process (eap_session=0x7f652c000a60)
    at src/modules/rlm_eap/libeap/eap_tls.c:958
        eap_tls_session = 0x7f652c000e20
        tls_session = 0x7f652c002e80
        this_round = 0x7f652c028aa0
        status = EAP_TLS_RECORD_RECV_COMPLETE
        request = 0x12512770
        eap_tls_data = 0x7f652c02a1c5
        data = 0x7f652c02a1ca "\026\003\001"
        data_len = 134
#21 0x00007f6555389147 in mod_process (arg=0x3d4d420, 
    eap_session=0x7f652c000a60)
    at src/modules/rlm_eap/types/rlm_eap_peap/rlm_eap_peap.c:240
        rcode = 825651712
        status = 32613
        inst = 0x3d4d420
        eap_tls_session = 0x7f652c000e20
        tls_session = 0x7f652c002e80
        peap = 0x7f652c014ea0
        request = 0x12512770
#22 0x00007f655599f6ef in eap_module_call (module=0x3d4a060, 
    eap_session=0x7f652c000a60) at src/modules/rlm_eap/eap.c:176
        rcode = 1
        request = 0x12512770
        caller = 0x21206f0 "eap"
#23 0x00007f65559a007a in eap_method_select (inst=0x2356ef0, 
    eap_session=0x7f652c000a60) at src/modules/rlm_eap/eap.c:407
        type = 0x7f652c014e10
        request = 0x12512770
        next = PW_EAP_TLS
        vp = 0x907f6531367228
#24 0x00007f655599e450 in mod_authenticate (instance=0x2356ef0, 
    request=0x12512770) at src/modules/rlm_eap/rlm_eap.c:202
        inst = 0x2356ef0
        eap_session = 0x7f652c000a60
        eap_packet = 0x0
        status = EAP_NOTFOUND
        rcode = RLM_MODULE_REJECT
#25 0x0000000000419301 in unlang_module (component=MOD_AUTHENTICATE, 
    sp=0x12439540, request=0x12512770) at src/main/interpreter.c:97
        blocked = 0
#26 0x000000000041a7a7 in unlang_single (request=0x12512770, 
    stack=0x7f65313673e0, presult=0x7f6531367394, priority=0x7f6531367398)
    at src/main/interpreter.c:679
        sp = 0x12439540
        entry = 0x7f6531367498
        c = 0x12439540
#27 0x000000000041ae72 in unlang_run (request=0x12512770, 
    stack=0x7f65313673e0, presult=0x7f6531367ee8, ppriority=0x7f6531367eec)
    at src/main/interpreter.c:885
        c = 0x12439540
        priority = -1
        result = RLM_MODULE_UNKNOWN
        entry = 0x7f6531367498
        action = UNLANG_PUSHED_CHILD
#28 0x000000000041b230 in unlang_interpret (request=0x12512770, c=0x12439440, 
    component=MOD_AUTHENTICATE) at src/main/interpreter.c:1036
        priority = 0
        result = RLM_MODULE_REJECT
        stack = {depth = 2, entry = {{result = RLM_MODULE_REJECT, 
              priority = 0, unwind = MOD_NULL, do_next_sibling = false, 
              was_if = false, if_taken = false, resume = false, c = 0x0, {
                foreach = {cursor = {first = 0x0, found = 0x0, last = 0x0, 
                    current = 0x0, next = 0x0}, vps = 0x0, variable = 0x0, 
                  depth = 0}, redundant = {child = 0x0, found = 0x0}}}, {
              result = RLM_MODULE_REJECT, priority = 0, unwind = MOD_NULL, 
              do_next_sibling = true, was_if = false, if_taken = false, 
              resume = false, c = 0x12439440, {foreach = {cursor = {
                    first = 0x0, found = 0x0, last = 0x0, current = 0x0, 
                    next = 0x0}, vps = 0x0, variable = 0x0, depth = 0}, 
                redundant = {child = 0x0, found = 0x0}}}, {
              result = RLM_MODULE_REJECT, priority = 0, unwind = MOD_NULL, 
              do_next_sibling = true, was_if = false, if_taken = false, 
              resume = false, c = 0x12439540, {foreach = {cursor = {
                    first = 0x0, found = 0x0, last = 0x0, current = 0x0, 
                    next = 0x0}, vps = 0x0, variable = 0x0, depth = 0}, 
                redundant = {child = 0x0, found = 0x0}}}, {
              result = RLM_MODULE_REJECT, priority = 0, unwind = MOD_NULL, 
              do_next_sibling = false, was_if = false, if_taken = false, 
              resume = false, c = 0x0, {foreach = {cursor = {first = 0x0, 
                    found = 0x0, last = 0x0, current = 0x0, next = 0x0}, 
                  vps = 0x0, variable = 0x0, depth = 0}, redundant = {
                  child = 0x0, found = 0x0}}} <repeats 29 times>}}
#29 0x000000000042ebb8 in indexed_modcall (comp=MOD_AUTHENTICATE, 
    idx=6299943, request=0x12512770) at src/main/modules.c:2067
        module = 0x0
        component = 0x4750f4 "<core>"
        rcode = RLM_MODULE_REJECT
        list = 0x12439440
        server = 0x12438ca0
        cs = 0x2292ac0
#30 0x000000000042ec24 in process_authenticate (auth_type=6299943, 
    request=0x12512770) at src/main/modules.c:2090
No locals.
#31 0x0000000000410edb in rad_check_password (request=0x12512770)
    at src/main/auth.c:251
        cursor = {first = 0x125127d0, found = 0x7f652c02a100, last = 0x0, 
          current = 0x0, next = 0x0}
        auth_type_pair = 0x0
        auth_type = 6299943
        result = 307272512
        auth_type_count = 1
#32 0x00000000004115eb in rad_authenticate (request=0x12512770)
    at src/main/auth.c:516
        check_item = 0x0
        module_msg = 0x446464 <request_pre_handler+644>
        tmp = 0x0
        result = 2
        autz_retry = 0 '\000'
        autz_type = 0
#33 0x0000000000446e7b in request_running (request=0x12512770, 
    action=FR_ACTION_RUN) at src/main/process.c:1455
        ret = 0
        __FUNCTION__ = "request_running"
#34 0x0000000000447092 in request_queued (request=0x12512770, 
    action=FR_ACTION_RUN) at src/main/process.c:1533
        __FUNCTION__ = "request_queued"
#35 0x0000000000442fc7 in request_handler_thread (arg=0x124fb9b0)
    at src/main/threads.c:731
        now = 1473089932
        request = 0x12512770
        thread = 0x124fb9b0
#36 0x00007f655951cdc5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#37 0x00007f6559012ced in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 2 (Thread 0x7f6530b68700 (LWP 75705)):
#0  0x00007f6558fd9efd in nanosleep () from /lib64/libc.so.6
No symbol table info available.
#1  0x00007f655900ab34 in usleep () from /lib64/libc.so.6
No symbol table info available.
#2  0x0000000000417f51 in detail_handler_thread (arg=0x233a830)
    at src/main/detail.c:969
        packet = 0x0
        c = 0 '\000'
        this = 0x233a830
        data = 0x233b100
#3  0x00007f655951cdc5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#4  0x00007f6559012ced in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 1 (Thread 0x7f655b2f0040 (LWP 75576)):
#0  0x00007f655900a413 in select () from /lib64/libc.so.6
No symbol table info available.
#1  0x00007f655ac81104 in fr_event_loop (el=0x124e4e40) at src/lib/event.c:595
        i = 10
        rcode = 1
        when = {tv_sec = 0, tv_usec = 204329}
        wake = 0x7ffe1862c190
        maxfd = 295
        read_fds = {fds_bits = {0, 0, 0, 0, 1092330979328, 
            0 <repeats 11 times>}}
        master_fds = {fds_bits = {0, 0, 0, 0, 1092330979328, 
            0 <repeats 11 times>}}
#2  0x0000000000450d99 in radius_event_process () at src/main/process.c:5452
No locals.
#3  0x00000000004352b7 in main (argc=1, argv=0x7ffe1862c358)
    at src/main/radiusd.c:586
        rcode = 0
        status = 0
        argval = -1
        display_version = false
        from_child = {4, 5}
        p = 0x7ffe1862cf95 "/radiusd"
        autofree = 0x1d35090
A debugging session is active.

    Inferior 1 [process 75576] will be detached.

Quit anyway? (y or n) [answered Y; input not from terminal]
@alanbuxey
Member

redis directly implicated as no issue with using local rbtree cache

cluster code suspected

rlm_cache (cache_ocsp): Bootstrap server "127.0.0.1" returned invalid data: Cluster map 0 node 0 has incorrect number of elements, expected 2 got 3

127.0.0.1:6379> cluster info
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:3
cluster_size:3
cluster_current_epoch:3
cluster_my_epoch:1
cluster_stats_messages_sent:1517387
cluster_stats_messages_received:1517387
127.0.0.1:6379> cluster nodes
dcc92749541b070e47454c6d089a7c5a831052f5 172.16.54.3:6379 myself,master - 0 0 1 connected 0-5460
f0c928cf75d963ec816df669dcd4919286646d04 172.16.55.6:6379 master - 0 1473090341304 2 connected 5461-10922
98209ce77062dfedff2e04ea2c63eab17bdd8216 172.16.56.9:6379 master - 0 1473090342306 3 connected 10923-16383
@arr2036
Member
arr2036 commented Sep 5, 2016 edited

Sounds like the output of the redis cluster commands has changed. Could you run a tcpdump and see exactly what command it's issuing then attach the result of running the same command to the github issue.

@arr2036
Member
arr2036 commented Sep 6, 2016

rlm_cache (cache_ocsp): Bootstrap server "127.0.0.1" returned invalid data: Cluster map 0 node 0 has incorrect number of elements, expected 2 got 3

That's seen during initialisation right?

@arr2036
Member
arr2036 commented Sep 6, 2016

I looked over the code, and it's not obvious why you'd get a NULL handle with no error at that point. I'm guessing this happens on every query?

@alanbuxey
Member

it probably does.... i'll try to grab the debug/tcpdump etc this
week....can turn off encryption etc if that helps ... I'll write a small
unlang to do a redis: query too to see if
that does the job too....

On 6 September 2016 at 14:43, Arran Cudbard-Bell notifications@github.com
wrote:

I looked over the code, and it's not obvious why you'd get a NULL handle
with no error at that point. I'm guessing this happens on every query?


You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
#1714 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/ACE-VVlHUapyrGDFT6zpkeBYRMoXUjQ7ks5qnW4VgaJpZM4J1LPh
.

@alanbuxey
Member

That's seen during initialisation right?

thats correct, its as the server is starting up

@alanbuxey
Member

looks like this actually also happens with the current "pre big massive change" 4.0.x too - when run in service mode... it doesn't die when run in -X or -fxx -l stdout modes! :/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment