dnsdist: segfault at f ip

[discostur]

Hi,

running dnsdist 1.7 in front of auth-4.6 with proxy_protocol enabled. It seems that if dnsdist gets more AXFR / tcp requests it segfaults:

[ 8230.575255] dnsdist/tcpClie[205749]: segfault at f ip 00005588be79f118 sp 00007fd7ccff7a60 error 4 in dnsdist[5588be549000+86f000]
[ 8230.575400] Code: ff ff ff 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 31 c0 c3 66 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 41 54 55 53 <48> 8b 6f 10 48 89 fb 48 85 ed 74 3c 0f 1f 40 00 48 8b 7d 28 48 8d
[ 8238.787218] dnsdist/tcpClie[206702]: segfault at f ip 000055e925bfe118 sp 00007fdb5a7faa60 error 4 in dnsdist[55e9259a8000+86f000]
[ 8238.787364] Code: ff ff ff 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 31 c0 c3 66 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 41 54 55 53 <48> 8b 6f 10 48 89 fb 48 85 ed 74 3c 0f 1f 40 00 48 8b 7d 28 48 8d

Running multiple instances with the same config but only one seems to crash. Not sure if its related to the number of simultaneous TCP requests or if there are boguous requests ...

Regards,
Kilian

[setharnold]

Were corefiles left behind? Are debug packages available that would help get more useful backtraces out of the cores? (From within gdb, I think something like thread apply all bt does the trick.)

Thanks

[rgacogne]

Thank you for reporting this issue. Would you by any chance be able to tell us a bit more about your setup? The operating system the crash occurred on, how was dnsdist installed/compiled, and dnsdist's configuration would be good start, otherwise without a core dump or a gdb trace there is not much we can do to understand the issue.

[rgacogne]

Note that both crashes seem to occur at the same place in the code (00005588be79f118-5588be549000 = 256118, 000055e925bfe118-55e9259a8000 = 256118). You might be able to locate the issue by using objdump --demangle -d </path/to/dnsdist/binary> and looking for the instructions around offset 256118. I can do that if you provide a dump of that command. Of course running the whole thing under gdb and getting a thread apply all bt full would be much easier to understand if you can easily reproduce the issue :)

[discostur]

@rgacogne @setharnold sorry for the missing details - it was already a a litte bit too late yesterday ;)

Ok i re-configured my systemd-coredump settings and was able to gather a fresh coredump. I will attach the output of thread apply all bt full.

Regarding my environment:

• Linux 4.18.0-348.12.2.el8_5.x86_64
• AlmaLinux release 8.5 (Arctic Sphynx)
• RPM installed from the official dnsdist repository

I'm running more setups like this but the coredump only occures on the ones which are running whit "primary" pdns_auth backends. On the "secondary" nodes (same dnsdist config in front of auth) no problems so far.

dnsdist <-> auth are running with proxy_protocol enabled.

Two more infos about our setup:

• we are running with multiple threads because of multiple frontend IP adresses wich are configured via addLocal()
• since 1.6 we added setMaxTCPClientThreads(10) to our config because of https://dnsdist.org/reference/tuning.html?highlight=setmaxtcp#setMaxTCPClientThreads

kernel stacktrace:

Feb 16 17:06:06 kernel: dnsdist/tcpClie[1956394]: segfault at f ip 000055ce9e327118 sp 00007fe513ffda60 error 4 in dnsdist[55ce9e0d1000+86f000]
Feb 16 17:06:06 kernel: Code: ff ff ff 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 31 c0 c3 66 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 41 54 55 53 <48> 8b 6f 10 48 89 fb 48 85 ed 74 3c 0f 1f 40 00 48 8b 7d 28 48 8d
Feb 16 17:06:06 systemd[1]: Started Process Core Dump (PID 1963187/UID 0).
Feb 16 17:06:15 systemd-coredump[1963188]: Process 1956388 (dnsdist) of user 990 dumped core.#012#012Stack trace of thread 1956394:#012#0  0x000055ce9e327118 _ZNSt10_HashtableINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESt4pairIKS5_S5_ESaIS8_ENSt8__detail10_Select1stESt8equal_toIS5_ESt4hashIS5_ENSA_18_Mod_range_hashingENSA_20_Default_ranged_hashENSA_20_Prime_rehash_policyENSA_17_Hashtable_traitsILb1ELb0ELb1EEEED1Ev (dnsdist)#012#1  0x000055ce9e5fafde _ZN7IDStateC2EOS_ (dnsdist)#012#2  0x000055ce9e62052b _ZNSt5dequeIN22TCPConnectionToBackend14PendingRequestESaIS1_EE12emplace_backIJS1_EEERS1_DpOT_ (dnsdist)#012#3  0x000055ce9e61e0bc _ZN22TCPConnectionToBackend8handleIOERSt10shared_ptrIS_ERK7timeval (dnsdist)#012#4  0x000055ce9e61e914 _ZN22TCPConnectionToBackend16handleIOCallbackEiRN5boost3anyE (dnsdist)#012#5  0x000055ce9e6d418c _ZN18EpollFDMultiplexer3runEP7timevali (dnsdist)#012#6  0x000055ce9e627ca2 _ZL15tcpClientThreadiiii (dnsdist)#012#7  0x00007fe53e2a2ba3 execute_native_thread_routine (libstdc++.so.6)#012#8  0x00007fe53e57d17a start_thread (libpthread.so.0)#012#9  0x00007fe53d97ddc3 __clone (libc.so.6)#012#012Stack trace of thread 1956391:#012#0  0x00007fe53d97e0f7 epoll_wait (libc.so.6)#012#1  0x000055ce9e6d406f _ZN18EpollFDMultiplexer3runEP7timevali (dnsdist)#012#2  0x000055ce9e627ca2 _ZL15tcpClientThreadiiii (dnsdist)#012#3  0x00007fe53e2a2ba3 execute_native_thread_routine (libstdc++.so.6)#012#4  0x00007fe53e57d17a start_thread (libpthread.so.0)#012#5  0x00007fe53d97ddc3 __clone (libc.so.6)#012#012Stack trace of thread 1956393:#012#0  0x00007fe53d97e0f7 epoll_wait (libc.so.6)#012#1  0x000055ce9e6d406f _ZN18EpollFDMultiplexer3runEP7timevali (dnsdist)#012#2  0x000055ce9e627ca2 _ZL15tcpClientThreadiiii (dnsdist)#012#3  0x00007fe53e2a2ba3 execute_native_thread_routine (libstdc++.so.6)#012#4  0x00007fe53e57d17a start_thread (libpthread.so.0)#012#5  0x00007fe53d97ddc3 __clone (libc.so.6)#012#012Stack trace of thread 1956395:#012#0  0x00007fe53d97e0f7 epoll_wait (libc.so.6)#012#1  0x000055ce9e6d406f _ZN18EpollFDMultiplexer3runEP7timevali (dnsdist)#012#2  0x000055ce9e627ca2 _ZL15tcpClientThreadiiii (dnsdist)#012#3  0x00007fe53e2a2ba3 execute_native_thread_routine (libstdc++.so.6)#012#4  0x00007fe53e57d17a start_thread (libpthread.so.0)#012#5  0x00007fe53d97ddc3 __clone (libc.so.6)#012#012Stack trace of thread 1956397:#012#0  0x00007fe53d97e0f7 epoll_wait (libc.so.6)#012#1  0x000055ce9e6d406f _ZN18EpollFDMultiplexer3runEP7timevali (dnsdist)#012#2  0x000055ce9e627ca2 _ZL15tcpClientThreadiiii (dnsdist)#012#3  0x00007fe53e2a2ba3 execute_native_thread_routine (libstdc++.so.6)#012#4  0x00007fe53e57d17a start_thread (libpthread.so.0)#012#5  0x00007fe53d97ddc3 __clone (libc.so.6)#012#012Stack trace of thread 1956396:#012#0  0x00007fe53d97e0f7 epoll_wait (libc.so.6)#012#1  0x000055ce9e6d406f _ZN18EpollFDMultiplexer3runEP7timevali (dnsdist)#012#2  0x000055ce9e627ca2 _ZL15tcpClientThreadiiii (dnsdist)#012#3  0x00007fe53e2a2ba3 execute_native_thread_routine (libstdc++.so.6)#012#4  0x00007fe53e57d17a start_thread (libpthread.so.0)#012#5  0x00007fe53d97ddc3 __clone (libc.so.6)#012#012Stack trace of thread 1956404:#012#0  0x00007fe53d97e0f7 epoll_wait (libc.so.6)#012#1  0x000055ce9e6d3ea0 _ZN18EpollFDMultiplexer15getAvailableFDsERSt6vectorIiSaIiEEi (dnsdist)#012#2  0x000055ce9e655f6b _Z15responderThreadSt10shared_ptrI15DownstreamStateE (dnsdist)#012#3  0x000055ce9e361ba4 _ZNSt6thread11_State_implINS_8_InvokerISt5tupleIJPFvSt10shared_ptrI15DownstreamStateEES5_EEEEE6_M_runEv (dnsdist)#012#4  0x00007fe53e2a2ba3 execute_native_thread_routine (libstdc++.so.6)#012#5  0x00007fe53e57d17a start_thread (libpthread.so.0)#012#6  0x00007fe53d97ddc3 __clone (libc.so.6)#012#012Stack trace of thread 1956392:#012#0  0x00007fe53d97e0f7 epoll_wait (libc.so.6)#012#1  0x000055ce9e6d406f _ZN18EpollFDMultiplexer3runEP7timevali (dnsdist)#012#2  0x000055ce9e627ca2 _ZL15tcpClientThreadiiii (dnsdist)#012#3  0x00007fe53e2a2ba3 execute_native_thread_routine (libstdc++.so.6)#012#4  0x00007fe53e57d17a start_thread (libpthread.so.0)#012#5  0x00007fe53d97ddc3 __clone (libc.so.6)#012#012Stack trace of thread 1956399:#012#0  0x00007fe53d97e0f7 epoll_wait (libc.so.6)#012#1  0x000055ce9e6d406f _ZN18EpollFDMultiplexer3runEP7timevali (dnsdist)#012#2  0x000055ce9e627ca2 _ZL15tcpClientThreadiiii (dnsdist)#012#3  0x00007fe53e2a2ba3 execute_native_thread_routine (libstdc++.so.6)#012#4  0x00007fe53e57d17a start_thread (libpthread.so.0)#012#5  0x00007fe53d97ddc3 __clone (libc.so.6)#012#012Stack trace of thread 1956389:#012#0  0x00007fe53e5837e8 pthread_cond_timedwait@@GLIBC_2.3.2 (libpthread.so.0)#012#1  0x00007fe540bfa354 fstrm__iothr_thr (libfstrm.so.0)#012#2  0x00007fe53e57d17a start_thread (libpthread.so.0)#012#3  0x00007fe53d97ddc3 __clone (libc.so.6)#012#012Stack trace of thread 1956416:#012#0  0x00007fe53e5878b7 recvmsg (libpthread.so.0)#012#1  0x000055ce9e65d050 _ZL15udpClientThreadP11ClientState (dnsdist)#012#2  0x00007fe53e2a2ba3 execute_native_thread_routine (libstdc++.so.6)#012#3  0x00007fe53e57d17a start_thread (libpthread.so.0)#012#4  0x00007fe53d97ddc3 __clone (libc.so.6)#012#012Stack trace of thread 1956413:#012#0  0x00007fe53d97e0f7 epoll_wait (libc.so.6)#012#1  0x000055ce9e6d3ea0 _ZN18EpollFDMultiplexer15getAvailableFDsERSt6vectorIiSaIiEEi (dnsdist)#012#2  0x000055ce9e655f6b _Z15responderThreadSt10shared_ptrI15DownstreamStateE (dnsdist)#012#3  0x000055ce9e361ba4 _ZNSt6thread11_State_implINS_8_InvokerISt5tupleIJPFvSt10shared_ptrI15DownstreamStateEES5_EEEEE6_M_runEv (dnsdist)#012#4  0x00007fe53e2a2ba3 execute_native_thread_routine (libstdc++.so.6)#012#5  0x00007fe53e57d17a start_thread (libpthread.so.0)#012#6  0x00007fe53d97ddc3 __clone (libc.so.6)#012#012Stack trace of thread 1956412:#012#0  0x00007fe53d97e0f7 epoll_wait (libc.so.6)#012#1  0x000055ce9e6d3ea0 _ZN18EpollFDMultiplexer15getAvailableFDsERSt6vectorIiSaIiEEi (dnsdist)#012#2  0x000055ce9e655f6b _Z15responderThreadSt10shared_ptrI15DownstreamStateE (dnsdist)#012#3  0x000055ce9e361ba4 _ZNSt6thread11_State_implINS_8_InvokerISt5tupleIJPFvSt10shared_ptrI15DownstreamStateEES5_EEEEE6_M_runEv (dnsdist)#012#4  0x00007fe53e2a2ba3 execute_native_thread_routine (libstdc++.so.6)#012#5  0x00007fe53e57d17a start_thread (libpthread.so.0)#012#6  0x00007fe53d97ddc3 __clone (libc.so.6)#012#012Stack trace of thread 1956401:#012#0  0x00007fe53d97e0f7 epoll_wait (libc.so.6)#012#1  0x000055ce9e6d406f _ZN18EpollFDMultiplexer3runEP7timevali (dnsdist)#012#2  0x000055ce9e5f7148 _ZL15dohClientThreadi (dnsdist)#012#3  0x00007fe53e2a2ba3 execute_native_thread_routine (libstdc++.so.6)#012#4  0x00007fe53e57d17a start_thread (libpthread.so.0)#012#5  0x00007fe53d97ddc3 __clone (libc.so.6)#012#012Stack trace of thread 1956406:#012#0  0x00007fe53d97e0f7 epoll_wait (libc.so.6)#012#1  0x000055ce9e6d3ea0 _ZN18EpollFDMultiplexer15getAvailableFDsERSt6vectorIiSaIiEEi (dnsdist)#012#2  0x000055ce9e655f6b _Z15responderThreadSt10shared_ptrI15DownstreamStateE (dnsdist)#012#3  0x000055ce9e361ba4 _ZNSt6thread11_State_implINS_8_InvokerISt5tupleIJPFvSt10shared_ptrI15DownstreamStateEES5_EEEEE6_M_runEv (dnsdist)#012#4  0x00007fe53e2a2ba3 execute_native_thread_routine (libstdc++.so.6)#012#5  0x00007fe53e57d17a start_thread (libpthread.so.0)#012#6  0x00007fe53d97ddc3 __clone (libc.so.6)#012#012Stack trace of thread 1956420:#012#0  0x00007fe53e5878b7 recvmsg (libpthread.so.0)#012#1  0x000055ce9e65d050 _ZL15udpClientThreadP11ClientState (dnsdist)#012#2  0x00007fe53e2a2ba3 execute_native_thread_routine (libstdc++.so.6)#012#3  0x00007fe53e57d17a start_thread (libpthread.so.0)#012#4  0x00007fe53d97ddc3 __clone (libc.so.6)#012#012Stack trace of thread 1956388:#012#0  0x00007fe53e57e66d __GI___pthread_timedjoin_ex (libpthread.so.0)#012#1  0x00007fe53e2a2e27 _ZNSt6thread4joinEv (libstdc++.so.6)#012#2  0x000055ce9e2f7248 main (dnsdist)#012#3  0x00007fe53d8a4493 __libc_start_main (libc.so.6)#012#4  0x000055ce9e2f9a2e _start (dnsdist)#012#012Stack trace of thread 1956398:#012#0  0x00007fe53d97e0f7 epoll_wait (libc.so.6)#012#1  0x000055ce9e6d406f _ZN18EpollFDMultiplexer3runEP7timevali (dnsdist)#012#2  0x000055ce9e627ca2 _ZL

                Stack trace of thread 1956394:
                #0  0x000055ce9e327118 _ZNSt10_HashtableINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESt4pairIKS5_S5_ESaIS8_ENSt8__detail10_Select1stESt8equal_toIS5_ESt4hashIS5_ENSA_18_Mod_range_hashingENSA_20_Default_ranged_hashENSA_20_Prime_rehash_policyENSA_17_Hashtable_traitsILb1ELb0ELb1EEEED1Ev (dnsdist)
                #1  0x000055ce9e5fafde _ZN7IDStateC2EOS_ (dnsdist)
                #2  0x000055ce9e62052b _ZNSt5dequeIN22TCPConnectionToBackend14PendingRequestESaIS1_EE12emplace_backIJS1_EEERS1_DpOT_ (dnsdist)
                #3  0x000055ce9e61e0bc _ZN22TCPConnectionToBackend8handleIOERSt10shared_ptrIS_ERK7timeval (dnsdist)
                #4  0x000055ce9e61e914 _ZN22TCPConnectionToBackend16handleIOCallbackEiRN5boost3anyE (dnsdist)
                #5  0x000055ce9e6d418c _ZN18EpollFDMultiplexer3runEP7timevali (dnsdist)
                #6  0x000055ce9e627ca2 _ZL15tcpClientThreadiiii (dnsdist)
                #7  0x00007fe53e2a2ba3 execute_native_thread_routine (libstdc++.so.6)
                #8  0x00007fe53e57d17a start_thread (libpthread.so.0)
                #9  0x00007fe53d97ddc3 __clone (libc.so.6)
                
                Stack trace of thread 1956391:
                #0  0x00007fe53d97e0f7 epoll_wait (libc.so.6)
                #1  0x000055ce9e6d406f _ZN18EpollFDMultiplexer3runEP7timevali (dnsdist)
                #2  0x000055ce9e627ca2 _ZL15tcpClientThreadiiii (dnsdist)
                #3  0x00007fe53e2a2ba3 execute_native_thread_routine (libstdc++.so.6)
                #4  0x00007fe53e57d17a start_thread (libpthread.so.0)
                #5  0x00007fe53d97ddc3 __clone (libc.so.6)
                
                Stack trace of thread 1956393:
                #0  0x00007fe53d97e0f7 epoll_wait (libc.so.6)
                #1  0x000055ce9e6d406f _ZN18EpollFDMultiplexer3runEP7timevali (dnsdist)
                #2  0x000055ce9e627ca2 _ZL15tcpClientThreadiiii (dnsdist)
                #3  0x00007fe53e2a2ba3 execute_native_thread_routine (libstdc++.so.6)
                #4  0x00007fe53e57d17a start_thread (libpthread.so.0)
                #5  0x00007fe53d97ddc3 __clone (libc.so.6)
                
                Stack trace of thread 1956395:
                #0  0x00007fe53d97e0f7 epoll_wait (libc.so.6)
                #1  0x000055ce9e6d406f _ZN18EpollFDMultiplexer3runEP7timevali (dnsdist)
                #2  0x000055ce9e627ca2 _ZL15tcpClientThreadiiii (dnsdist)
                #3  0x00007fe53e2a2ba3 execute_native_thread_routine (libstdc++.so.6)
                #4  0x00007fe53e57d17a start_thread (libpthread.so.0)
                #5  0x00007fe53d97ddc3 __clone (libc.so.6)
                
                Stack trace of thread 1956397:
                #0  0x00007fe53d97e0f7 epoll_wait (libc.so.6)
                #1  0x000055ce9e6d406f _ZN18EpollFDMultiplexer3runEP7timevali (dnsdist)
                #2  0x000055ce9e627ca2 _ZL15tcpClientThreadiiii (dnsdist)
                #3  0x00007fe53e2a2ba3 execute_native_thread_routine (libstdc++.so.6)
                #4  0x00007fe53e57d17a start_thread (libpthread.so.0)
                #5  0x00007fe53d97ddc3 __clone (libc.so.6)
                
                Stack trace of thread 1956396:
                #0  0x00007fe53d97e0f7 epoll_wait (libc.so.6)
                #1  0x000055ce9e6d406f _ZN18EpollFDMultiplexer3runEP7timevali (dnsdist)
                #2  0x000055ce9e627ca2 _ZL15tcpClientThreadiiii (dnsdist)
                #3  0x00007fe53e2a2ba3 execute_native_thread_routine (libstdc++.so.6)
                #4  0x00007fe53e57d17a start_thread (libpthread.so.0)
                #5  0x00007fe53d97ddc3 __clone (libc.so.6)
                
                Stack trace of thread 1956404:
                #0  0x00007fe53d97e0f7 epoll_wait (libc.so.6)
                #1  0x000055ce9e6d3ea0 _ZN18EpollFDMultiplexer15getAvailableFDsERSt6vectorIiSaIiEEi (dnsdist)
                #2  0x000055ce9e655f6b _Z15responderThreadSt10shared_ptrI15DownstreamStateE (dnsdist)
                #3  0x000055ce9e361ba4 _ZNSt6thread11_State_implINS_8_InvokerISt5tupleIJPFvSt10shared_ptrI15DownstreamStateEES5_EEEEE6_M_runEv (dnsdist)
                #4  0x00007fe53e2a2ba3 execute_native_thread_routine (libstdc++.so.6)
                #5  0x00007fe53e57d17a start_thread (libpthread.so.0)
                #6  0x00007fe53d97ddc3 __clone (libc.so.6)
                
                Stack trace of thread 1956392:
                #0  0x00007fe53d97e0f7 epoll_wait (libc.so.6)
                #1  0x000055ce9e6d406f _ZN18EpollFDMultiplexer3runEP7timevali (dnsdist)
                #2  0x000055ce9e627ca2 _ZL15tcpClientThreadiiii (dnsdist)
                #3  0x00007fe53e2a2ba3 execute_native_thread_routine (libstdc++.so.6)
                #4  0x00007fe53e57d17a start_thread (libpthread.so.0)
                #5  0x00007fe53d97ddc3 __clone (libc.so.6)
                
                Stack trace of thread 1956399:
                #0  0x00007fe53d97e0f7 epoll_wait (libc.so.6)
                #1  0x000055ce9e6d406f _ZN18EpollFDMultiplexer3runEP7timevali (dnsdist)
                #2  0x000055ce9e627ca2 _ZL15tcpClientThreadiiii (dnsdist)
                #3  0x00007fe53e2a2ba3 execute_native_thread_routine (libstdc++.so.6)
                #4  0x00007fe53e57d17a start_thread (libpthread.so.0)
                #5  0x00007fe53d97ddc3 __clone (libc.so.6)
                
                Stack trace of thread 1956389:
                #0  0x00007fe53e5837e8 pthread_cond_timedwait@@GLIBC_2.3.2 (libpthread.so.0)
                #1  0x00007fe540bfa354 fstrm__iothr_thr (libfstrm.so.0)
                #2  0x00007fe53e57d17a start_thread (libpthread.so.0)
                #3  0x00007fe53d97ddc3 __clone (libc.so.6)
                
                Stack trace of thread 1956416:
                #0  0x00007fe53e5878b7 recvmsg (libpthread.so.0)
                #1  0x000055ce9e65d050 _ZL15udpClientThreadP11ClientState (dnsdist)
                #2  0x00007fe53e2a2ba3 execute_native_thread_routine (libstdc++.so.6)
                #3  0x00007fe53e57d17a start_thread (libpthread.so.0)
                #4  0x00007fe53d97ddc3 __clone (libc.so.6)
                
                Stack trace of thread 1956413:
                #0  0x00007fe53d97e0f7 epoll_wait (libc.so.6)
                #1  0x000055ce9e6d3ea0 _ZN18EpollFDMultiplexer15getAvailableFDsERSt6vectorIiSaIiEEi (dnsdist)
                #2  0x000055ce9e655f6b _Z15responderThreadSt10shared_ptrI15DownstreamStateE (dnsdist)
                #3  0x000055ce9e361ba4 _ZNSt6thread11_State_implINS_8_InvokerISt5tupleIJPFvSt10shared_ptrI15DownstreamStateEES5_EEEEE6_M_runEv (dnsdist)
                #4  0x00007fe53e2a2ba3 execute_native_thread_routine (libstdc++.so.6)
                #5  0x00007fe53e57d17a start_thread (libpthread.so.0)
                #6  0x00007fe53d97ddc3 __clone (libc.so.6)
                
                Stack trace of thread 1956412:
                #0  0x00007fe53d97e0f7 epoll_wait (libc.so.6)
                #1  0x000055ce9e6d3ea0 _ZN18EpollFDMultiplexer15getAvailableFDsERSt6vectorIiSaIiEEi (dnsdist)
                #2  0x000055ce9e655f6b _Z15responderThreadSt10shared_ptrI15DownstreamStateE (dnsdist)
                #3  0x000055ce9e361ba4 _ZNSt6thread11_State_implINS_8_InvokerISt5tupleIJPFvSt10shared_ptrI15DownstreamStateEES5_EEEEE6_M_runEv (dnsdist)
                #4  0x00007fe53e2a2ba3 execute_native_thread_routine (libstdc++.so.6)
                #5  0x00007fe53e57d17a start_thread (libpthread.so.0)
                #6  0x00007fe53d97ddc3 __clone (libc.so.6)
                
                Stack trace of thread 1956401:
                #0  0x00007fe53d97e0f7 epoll_wait (libc.so.6)
                #1  0x000055ce9e6d406f _ZN18EpollFDMultiplexer3runEP7timevali (dnsdist)
                #2  0x000055ce9e5f7148 _ZL15dohClientThreadi (dnsdist)
                #3  0x00007fe53e2a2ba3 execute_native_thread_routine (libstdc++.so.6)
                #4  0x00007fe53e57d17a start_thread (libpthread.so.0)
                #5  0x00007fe53d97ddc3 __clone (libc.so.6)
                
                Stack trace of thread 1956406:
                #0  0x00007fe53d97e0f7 epoll_wait (libc.so.6)
                #1  0x000055ce9e6d3ea0 _ZN18EpollFDMultiplexer15getAvailableFDsERSt6vectorIiSaIiEEi (dnsdist)
                #2  0x000055ce9e655f6b _Z15responderThreadSt10shared_ptrI15DownstreamStateE (dnsdist)
                #3  0x000055ce9e361ba4 _ZNSt6thread11_State_implINS_8_InvokerISt5tupleIJPFvSt10shared_ptrI15DownstreamStateEES5_EEEEE6_M_runEv (dnsdist)
                #4  0x00007fe53e2a2ba3 execute_native_thread_routine (libstdc++.so.6)
                #5  0x00007fe53e57d17a start_thread (libpthread.so.0)
                #6  0x00007fe53d97ddc3 __clone (libc.so.6)
                
                Stack trace of thread 1956420:
                #0  0x00007fe53e5878b7 recvmsg (libpthread.so.0)
                #1  0x000055ce9e65d050 _ZL15udpClientThreadP11ClientState (dnsdist)
                #2  0x00007fe53e2a2ba3 execute_native_thread_routine (libstdc++.so.6)
                #3  0x00007fe53e57d17a start_thread (libpthread.so.0)
                #4  0x00007fe53d97ddc3 __clone (libc.so.6)
                
                Stack trace of thread 1956388:
                #0  0x00007fe53e57e66d __GI___pthread_timedjoin_ex (libpthread.so.0)
                #1  0x00007fe53e2a2e27 _ZNSt6thread4joinEv (libstdc++.so.6)
                #2  0x000055ce9e2f7248 main (dnsdist)
                #3  0x00007fe53d8a4493 __libc_start_main (libc.so.6)
                #4  0x000055ce9e2f9a2e _start (dnsdist)
                
                Stack trace of thread 1956398:
                #0  0x00007fe53d97e0f7 epoll_wait (libc.so.6)
                #1  0x000055ce9e6d406f _ZN18EpollFDMultiplexer3runEP7timevali (dnsdist)
                #2  0x000055ce9e627ca2 _ZL15tcpClientThreadiiii (dnsdist)
                #3  0x00007fe53e2a2ba3 execute_native_thread_routine (libstdc++.so.6)
                #4  0x00007fe53e57d17a start_thread (libpthread.so.0)
                #5  0x00007fe53d97ddc3 __clone (libc.so.6)
                
                Stack trace of thread 1956400:
                #0  0x00007fe53d97e0f7 epoll_wait (libc.so.6)
                #1  0x000055ce9e6d406f _ZN18EpollFDMultiplexer3runEP7timevali (dnsdist)
                #2  0x000055ce9e627ca2 _ZL15tcpClientThreadiiii (dnsdist)
                #3  0x00007fe53e2a2ba3 execute_native_thread_routine (libstdc++.so.6)
                #4  0x00007fe53e57d17a start_thread (libpthread.so.0)
                #5  0x00007fe53d97ddc3 __clone (libc.so.6)
                
                Stack trace of thread 1956754:
                #0  0x00007fe53e5878b7 recvmsg (libpthread.so.0)
                #1  0x000055ce9e65d050 _ZL15udpClientThreadP11ClientState (dnsdist)
                #2  0x00007fe53e2a2ba3 execute_native_thread_routine (libstdc++.so.6)
                #3  0x00007fe53e57d17a start_thread (libpthread.so.0)
                #4  0x00007fe53d97ddc3 __clone (libc.so.6)
                
                Stack trace of thread 1956710:
                #0  0x00007fe53e5878b7 recvmsg (libpthread.so.0)
                #1  0x000055ce9e65d050 _ZL15udpClientThreadP11ClientState (dnsdist)
                #2  0x00007fe53e2a2ba3 execute_native_thread_routine (libstdc++.so.6)
                #3  0x00007fe53e57d17a start_thread (libpthread.so.0)
                #4  0x00007fe53d97ddc3 __clone (libc.so.6)
                
                Stack trace of thread 1956417:
                #0  0x00007fe53d97f767 accept4 (libc.so.6)
                #1  0x000055ce9e626a04 _Z17tcpAcceptorThreadP11ClientState (dnsdist)
                #2  0x00007fe53e2a2ba3 execute_native_thread_routine (libstdc++.so.6)
                #3  0x00007fe53e57d17a start_thread (libpthread.so.0)
                #4  0x00007fe53d97ddc3 __clone (libc.so.6)
                
                Stack trace of thread 1956876:
                #0  0x00007fe53e5878b7 recvmsg (libpthread.so.0)
                #1  0x000055ce9e65d050 _ZL15udpClientThreadP11ClientState (dnsdist)
                #2  0x00007fe53e2a2ba3 execute_native_thread_routine (libstdc++.so.6)
                #3  0x00007fe53e57d17a start_thread (libpthread.so.0)
                #4  0x00007fe53d97ddc3 __clone (libc.so.6)
                
                Stack trace of thread 1956419:
                #0  0x00007fe53d97f767 accept4 (libc.so.6)
                #1  0x000055ce9e626a04 _Z17tcpAcceptorThreadP11ClientState (dnsdist)
                #2  0x00007fe53e2a2ba3 execute_native_thread_routine (libstdc++.so.6)
                #3  0x00007fe53e57d17a start_thread (libpthread.so.0)
                #4  0x00007fe53d97ddc3 __clone (libc.so.6)
                
                Stack trace of thread 1956426:
                #0  0x00007fe53e5878b7 recvmsg (libpthread.so.0)
                #1  0x000055ce9e65d050 _ZL15udpClientThreadP11ClientState (dnsdist)
                #2  0x00007fe53e2a2ba3 execute_native_thread_routine (libstdc++.so.6)
                #3  0x00007fe53e57d17a start_thread (libpthread.so.0)
                #4  0x00007fe53d97ddc3 __clone (libc.so.6)
                
                Stack trace of thread 1956415:
                #0  0x00007fe53e586b07 accept (libpthread.so.0)
                #1  0x000055ce9e69a901 _Z7SAcceptiR12ComboAddress (dnsdist)
                #2  0x000055ce9e636589 _Z22dnsdistWebserverThreadiRK12ComboAddress (dnsdist)
                #3  0x00007fe53e2a2ba3 execute_native_thread_routine (libstdc++.so.6)
                #4  0x00007fe53e57d17a start_thread (libpthread.so.0)
                #5  0x00007fe53d97ddc3 __clone (libc.so.6)
                
                Stack trace of thread 1956963:
                #0  0x00007fe53d949d98 __nanosleep (libc.so.6)
                #1  0x00007fe53d949c9e sleep (libc.so.6)
                #2  0x000055ce9e6593ba _ZL11maintThreadv (dnsdist)
                #3  0x00007fe53e2a2ba3 execute_native_thread_routine (libstdc++.so.6)
                #4  0x00007fe53e57d17a start_thread (libpthread.so.0)
                #5  0x00007fe53d97ddc3 __clone (libc.so.6)
                
                Stack trace of thread 1956802:
                #0  0x00007fe53e5878b7 recvmsg (libpthread.so.0)
                #1  0x000055ce9e65d050 _ZL15udpClientThreadP11ClientState (dnsdist)
                #2  0x00007fe53e2a2ba3 execute_native_thread_routine (libstdc++.so.6)
                #3  0x00007fe53e57d17a start_thread (libpthread.so.0)
                #4  0x00007fe53d97ddc3 __clone (libc.so.6)
                
                Stack trace of thread 1956757:
                #0  0x00007fe53d97f767 accept4 (libc.so.6)
                #1  0x000055ce9e626a04 _Z17tcpAcceptorThreadP11ClientState (dnsdist)
                #2  0x00007fe53e2a2ba3 execute_native_thread_routine (libstdc++.so.6)
                #3  0x00007fe53e57d17a start_thread (libpthread.so.0)
                #4  0x00007fe53d97ddc3 __clone (libc.so.6)
                
                Stack trace of thread 1956506:
                #0  0x00007fe53e5878b7 recvmsg (libpthread.so.0)
                #1  0x000055ce9e65d050 _ZL15udpClientThreadP11ClientState (dnsdist)
                #2  0x00007fe53e2a2ba3 execute_native_thread_routine (libstdc++.so.6)
                #3  0x00007fe53e57d17a start_thread (libpthread.so.0)
                #4  0x00007fe53d97ddc3 __clone (libc.so.6)
                
                Stack trace of thread 1956430:
                #0  0x00007fe53e5878b7 recvmsg (libpthread.so.0)
                #1  0x000055ce9e65d050 _ZL15udpClientThreadP11ClientState (dnsdist)
                #2  0x00007fe53e2a2ba3 execute_native_thread_routine (libstdc++.so.6)
                #3  0x00007fe53e57d17a start_thread (libpthread.so.0)
                #4  0x00007fe53d97ddc3 __clone (libc.so.6)
                
                Stack trace of thread 1956427:
                #0  0x00007fe53d97f767 accept4 (libc.so.6)
                #1  0x000055ce9e626a04 _Z17tcpAcceptorThreadP11ClientState (dnsdist)
                #2  0x00007fe53e2a2ba3 execute_native_thread_routine (libstdc++.so.6)
                #3  0x00007fe53e57d17a start_thread (libpthread.so.0)
                #4  0x00007fe53d97ddc3 __clone (libc.so.6)
                
                Stack trace of thread 1956472:
                #0  0x00007fe53e5878b7 recvmsg (libpthread.so.0)
                #1  0x000055ce9e65d050 _ZL15udpClientThreadP11ClientState (dnsdist)
                #2  0x00007fe53e2a2ba3 execute_native_thread_routine (libstdc++.so.6)
                #3  0x00007fe53e57d17a start_thread (libpthread.so.0)
                #4  0x00007fe53d97ddc3 __clone (libc.so.6)
                
                Stack trace of thread 1956402:
                #0  0x00007fe53e586b07 accept (libpthread.so.0)
                #1  0x000055ce9e69a901 _Z7SAcceptiR12ComboAddress (dnsdist)
                #2  0x000055ce9e33c113 _Z13controlThreadi12ComboAddress (dnsdist)
                #3  0x000055ce9e5dbdaa _ZNSt6thread11_State_implINS_8_InvokerISt5tupleIJPFvi12ComboAddressEiS3_EEEEE6_M_runEv (dnsdist)
                #4  0x00007fe53e2a2ba3 execute_native_thread_routine (libstdc++.so.6)
                #5  0x00007fe53e57d17a start_thread (libpthread.so.0)
                #6  0x00007fe53d97ddc3 __clone (libc.so.6)
                
                Stack trace of thread 1956435:
                #0  0x00007fe53d97f767 accept4 (libc.so.6)
                #1  0x000055ce9e626a04 _Z17tcpAcceptorThreadP11ClientState (dnsdist)
                #2  0x00007fe53e2a2ba3 execute_native_thread_routine (libstdc++.so.6)
                #3  0x00007fe53e57d17a start_thread (libpthread.so.0)
                #4  0x00007fe53d97ddc3 __clone (libc.so.6)
                
                Stack trace of thread 1956431:
                #0  0x00007fe53d97f767 accept4 (libc.so.6)
                #1  0x000055ce9e626a04 _Z17tcpAcceptorThreadP11ClientState (dnsdist)
                #2  0x00007fe53e2a2ba3 execute_native_thread_routine (libstdc++.so.6)
                #3  0x00007fe53e57d17a start_thread (libpthread.so.0)
                #4  0x00007fe53d97ddc3 __clone (libc.so.6)
                
                Stack trace of thread 1956425:
                #0  0x00007fe53d97f767 accept4 (libc.so.6)
                #1  0x000055ce9e626a04 _Z17tcpAcceptorThreadP11ClientState (dnsdist)
                #2  0x00007fe53e2a2ba3 execute_native_thread_routine (libstdc++.so.6)
                #3  0x00007fe53e57d17a start_thread (libpthread.so.0)
                #4  0x00007fe53d97ddc3 __clone (libc.so.6)
                
                Stack trace of thread 1956873:
                #0  0x00007fe53d97f767 accept4 (libc.so.6)
                #1  0x000055ce9e626a04 _Z17tcpAcceptorThreadP11ClientState (dnsdist)
                #2  0x00007fe53e2a2ba3 execute_native_thread_routine (libstdc++.so.6)
                #3  0x00007fe53e57d17a start_thread (libpthread.so.0)
                #4  0x00007fe53d97ddc3 __clone (libc.so.6)
                
                Stack trace of thread 1956816:
                #0  0x00007fe53e5878b7 recvmsg (libpthread.so.0)
                #1  0x000055ce9e65d050 _ZL15udpClientThreadP11ClientState (dnsdist)
                #2  0x00007fe53e2a2ba3 execute_native_thread_routine (libstdc++.so.6)
                #3  0x00007fe53e57d17a start_thread (libpthread.so.0)
                #4  0x00007fe53d97ddc3 __clone (libc.so.6)
                
                Stack trace of thread 1956424:
                #0  0x00007fe53e5878b7 recvmsg (libpthread.so.0)
                #1  0x000055ce9e65d050 _ZL15udpClientThreadP11ClientState (dnsdist)
                #2  0x00007fe53e2a2ba3 execute_native_thread_routine (libstdc++.so.6)
                #3  0x00007fe53e57d17a start_thread (libpthread.so.0)
                #4  0x00007fe53d97ddc3 __clone (libc.so.6)
                
                Stack trace of thread 1956789:
                #0  0x00007fe53d97f767 accept4 (libc.so.6)
                #1  0x000055ce9e626a04 _Z17tcpAcceptorThreadP11ClientState (dnsdist)
                #2  0x00007fe53e2a2ba3 execute_native_thread_routine (libstdc++.so.6)
                #3  0x00007fe53e57d17a start_thread (libpthread.so.0)
                #4  0x00007fe53d97ddc3 __clone (libc.so.6)
                
                Stack trace of thread 1956763:
                #0  0x00007fe53d97f767 accept4 (libc.so.6)
                #1  0x000055ce9e626a04 _Z17tcpAcceptorThreadP11ClientState (dnsdist)
                #2  0x00007fe53e2a2ba3 execute_native_thread_routine (libstdc++.so.6)
                #3  0x00007fe53e57d17a start_thread (libpthread.so.0)
                #4  0x00007fe53d97ddc3 __clone (libc.so.6)
                
                Stack trace of thread 1956461:
                #0  0x00007fe53d97f767 accept4 (libc.so.6)
                #1  0x000055ce9e626a04 _Z17tcpAcceptorThreadP11ClientState (dnsdist)
                #2  0x00007fe53e2a2ba3 execute_native_thread_routine (libstdc++.so.6)
                #3  0x00007fe53e57d17a start_thread (libpthread.so.0)
                #4  0x00007fe53d97ddc3 __clone (libc.so.6)
                
                Stack trace of thread 1956457:
                #0  0x00007fe53d97f767 accept4 (libc.so.6)
                #1  0x000055ce9e626a04 _Z17tcpAcceptorThreadP11ClientState (dnsdist)
                #2  0x00007fe53e2a2ba3 execute_native_thread_routine (libstdc++.so.6)
                #3  0x00007fe53e57d17a start_thread (libpthread.so.0)
                #4  0x00007fe53d97ddc3 __clone (libc.so.6)
                
                Stack trace of thread 1956423:
                #0  0x00007fe53d97f767 accept4 (libc.so.6)
                #1  0x000055ce9e626a04 _Z17tcpAcceptorThreadP11ClientState (dnsdist)
                #2  0x00007fe53e2a2ba3 execute_native_thread_routine (libstdc++.so.6)
                #3  0x00007fe53e57d17a start_thread (libpthread.so.0)
                #4  0x00007fe53d97ddc3 __clone (libc.so.6)
                
                Stack trace of thread 1956743:
                #0  0x00007fe53d97f767 accept4 (libc.so.6)
                #1  0x000055ce9e626a04 _Z17tcpAcceptorThreadP11ClientState (dnsdist)
                #2  0x00007fe53e2a2ba3 execute_native_thread_routine (libstdc++.so.6)
                #3  0x00007fe53e57d17a start_thread (libpthread.so.0)
                #4  0x00007fe53d97ddc3 __clone (libc.so.6)
                
                Stack trace of thread 1956421:
                #0  0x00007fe53d97f767 accept4 (libc.so.6)
                #1  0x000055ce9e626a04 _Z17tcpAcceptorThreadP11ClientState (dnsdist)
                #2  0x00007fe53e2a2ba3 execute_native_thread_routine (libstdc++.so.6)
                #3  0x00007fe53e57d17a start_thread (libpthread.so.0)
                #4  0x00007fe53d97ddc3 __clone (libc.so.6)
                
                Stack trace of thread 1956597:
                #0  0x00007fe53d97f767 accept4 (libc.so.6)
                #1  0x000055ce9e626a04 _Z17tcpAcceptorThreadP11ClientState (dnsdist)
                #2  0x00007fe53e2a2ba3 execute_native_thread_routine (libstdc++.so.6)
                #3  0x00007fe53e57d17a start_thread (libpthread.so.0)
                #4  0x00007fe53d97ddc3 __clone (libc.so.6)
                
                Stack trace of thread 1956758:
                #0  0x00007fe53e5878b7 recvmsg (libpthread.so.0)
                #1  0x000055ce9e65d050 _ZL15udpClientThreadP11ClientState (dnsdist)
                #2  0x00007fe53e2a2ba3 execute_native_thread_routine (libstdc++.so.6)
                #3  0x00007fe53e57d17a start_thread (libpthread.so.0)
                #4  0x00007fe53d97ddc3 __clone (libc.so.6)
                
                Stack trace of thread 1956800:
                #0  0x00007fe53e5878b7 recvmsg (libpthread.so.0)
                #1  0x000055ce9e65d050 _ZL15udpClientThreadP11ClientState (dnsdist)
                #2  0x00007fe53e2a2ba3 execute_native_thread_routine (libstdc++.so.6)
                #3  0x00007fe53e57d17a start_thread (libpthread.so.0)
                #4  0x00007fe53d97ddc3 __clone (libc.so.6)
                
                Stack trace of thread 1956901:
                #0  0x00007fe53d97f767 accept4 (libc.so.6)
                #1  0x000055ce9e626a04 _Z17tcpAcceptorThreadP11ClientState (dnsdist)
                #2  0x00007fe53e2a2ba3 execute_native_thread_routine (libstdc++.so.6)
                #3  0x00007fe53e57d17a start_thread (libpthread.so.0)
                #4  0x00007fe53d97ddc3 __clone (libc.so.6)
                
                Stack trace of thread 1956753:
                #0  0x00007fe53d97f767 accept4 (libc.so.6)
                #1  0x000055ce9e626a04 _Z17tcpAcceptorThreadP11ClientState (dnsdist)
                #2  0x00007fe53e2a2ba3 execute_native_thread_routine (libstdc++.so.6)
                #3  0x00007fe53e57d17a start_thread (libpthread.so.0)
                #4  0x00007fe53d97ddc3 __clone (libc.so.6)
                
                Stack trace of thread 1956551:
                #0  0x00007fe53d97f767 accept4 (libc.so.6)
                #1  0x000055ce9e626a04 _Z17tcpAcceptorThreadP11ClientState (dnsdist)
                #2  0x00007fe53e2a2ba3 execute_native_thread_routine (libstdc++.so.6)
                #3  0x00007fe53e57d17a start_thread (libpthread.so.0)
                #4  0x00007fe53d97ddc3 __clone (libc.so.6)
                
                Stack trace of thread 1956626:
                #0  0x00007fe53e5878b7 recvmsg (libpthread.so.0)
                #1  0x000055ce9e65d050 _ZL15udpClientThreadP11ClientState (dnsdist)
                #2  0x00007fe53e2a2ba3 execute_native_thread_routine (libstdc++.so.6)
                #3  0x00007fe53e57d17a start_thread (libpthread.so.0)
                #4  0x00007fe53d97ddc3 __clone (libc.so.6)
                
                Stack trace of thread 1956815:
                #0  0x00007fe53d97f767 accept4 (libc.so.6)
                #1  0x000055ce9e626a04 _Z17tcpAcceptorThreadP11ClientState (dnsdist)
                #2  0x00007fe53e2a2ba3 execute_native_thread_routine (libstdc++.so.6)
                #3  0x00007fe53e57d17a start_thread (libpthread.so.0)
                #4  0x00007fe53d97ddc3 __clone (libc.so.6)
                
                Stack trace of thread 1956418:
                #0  0x00007fe53e5878b7 recvmsg (libpthread.so.0)
                #1  0x000055ce9e65d050 _ZL15udpClientThreadP11ClientState (dnsdist)
                #2  0x00007fe53e2a2ba3 execute_native_thread_routine (libstdc++.so.6)
                #3  0x00007fe53e57d17a start_thread (libpthread.so.0)
                #4  0x00007fe53d97ddc3 __clone (libc.so.6)
                
                Stack trace of thread 1956437:
                #0  0x00007fe53d97f767 accept4 (libc.so.6)
                #1  0x000055ce9e626a04 _Z17tcpAcceptorThreadP11ClientState (dnsdist)
                #2  0x00007fe53e2a2ba3 execute_native_thread_routine (libstdc++.so.6)
                #3  0x00007fe53e57d17a start_thread (libpthread.so.0)
                #4  0x00007fe53d97ddc3 __clone (libc.so.6)
                
                Stack trace of thread 1956429:
                #0  0x00007fe53d97f767 accept4 (libc.so.6)
                #1  0x000055ce9e626a04 _Z17tcpAcceptorThreadP11ClientState (dnsdist)
                #2  0x00007fe53e2a2ba3 execute_native_thread_routine (libstdc++.so.6)
                #3  0x00007fe53e57d17a start_thread (libpthread.so.0)
                #4  0x00007fe53d97ddc3 __clone (libc.so.6)
                
                Stack trace of thread 1956454:
                #0  0x00007fe53e5878b7 recvmsg (libpthread.so.0)
                #1  0x000055ce9e65d050 _ZL15udpClientThreadP11ClientState (dnsdist)
                #2  0x00007fe53e2a2ba3 execute_native_thread_routine (libstdc++.so.6)
                #3  0x00007fe53e57d17a start_thread (libpthread.so.0)
                #4  0x00007fe53d97ddc3 __clone (libc.so.6)
                
                Stack trace of thread 1956760:
                #0  0x00007fe53e5878b7 recvmsg (libpthread.so.0)
                #1  0x000055ce9e65d050 _ZL15udpClientThreadP11ClientState (dnsdist)
                #2  0x00007fe53e2a2ba3 execute_native_thread_routine (libstdc++.so.6)
                #3  0x00007fe53e57d17a start_thread (libpthread.so.0)
                #4  0x00007fe53d97ddc3 __clone (libc.so.6)
                
                Stack trace of thread 1956906:
                #0  0x00007fe53e5878b7 recvmsg (libpthread.so.0)
                #1  0x000055ce9e65d050 _ZL15udpClientThreadP11ClientState (dnsdist)
                #2  0x00007fe53e2a2ba3 execute_native_thread_routine (libstdc++.so.6)
                #3  0x00007fe53e57d17a start_thread (libpthread.so.0)
                #4  0x00007fe53d97ddc3 __clone (libc.so.6)
                
                Stack trace of thread 1956899:
                #0  0x00007fe53d97f767 accept4 (libc.so.6)
                #1  0x000055ce9e626a04 _Z17tcpAcceptorThreadP11ClientState (dnsdist)
                #2  0x00007fe53e2a2ba3 execute_native_thread_routine (libstdc++.so.6)
                #3  0x00007fe53e57d17a start_thread (libpthread.so.0)
                #4  0x00007fe53d97ddc3 __clone (libc.so.6)
                
                Stack trace of thread 1956432:
                #0  0x00007fe53e5878b7 recvmsg (libpthread.so.0)
                #1  0x000055ce9e65d050 _ZL15udpClientThreadP11ClientState (dnsdist)
                #2  0x00007fe53e2a2ba3 execute_native_thread_routine (libstdc++.so.6)
                #3  0x00007fe53e57d17a start_thread (libpthread.so.0)
                #4  0x00007fe53d97ddc3 __clone (libc.so.6)
                
                Stack trace of thread 1956966:
                #0  0x00007fe53d949d98 __nanosleep (libc.so.6)
                #1  0x00007fe53d949c9e sleep (libc.so.6)
                #2  0x000055ce9e655793 _ZL13secPollThreadv (dnsdist)
                #3  0x00007fe53e2a2ba3 execute_native_thread_routine (libstdc++.so.6)
                #4  0x00007fe53e57d17a start_thread (libpthread.so.0)
                #5  0x00007fe53d97ddc3 __clone (libc.so.6)

GDB output:

dnsdist_17_gdb.txt

If you need any further information please let me know ;)

[rgacogne]

Thanks, that's very helpful! I'm pretty sure we are providing debug packages, so it would be great if you could install the dnsdist debuginfo and perhaps debugsource package and try to get a new GDB output after that (it can be of an existing core, no need to get a new one!). In the meantime I'll see if I can figure it out without the symbols :)

[rgacogne]

On more question, are you by any chance setting tags on queries and/or responses?

[discostur]

@rgacogne yes, we are using dnstap for logging and set a response action:

dnstap = newFrameStreamUnixLogger("/var/run/dnsdist/dnstap.sock")
addResponseAction(AllRule(), DnstapLogResponseAction("name_of_dns_server", dnstap))

[discostur]

@rgacogne i found the debugsource / debuginfo packages in your repository:

https://repo.powerdns.com/centos/x86_64/8/dnsdist-17/debug/dnsdist-debugsource-1.7.0-1pdns.el8.x86_64.rpm
https://repo.powerdns.com/centos/x86_64/8/dnsdist-17/debug/dnsdist-debuginfo-1.7.0-1pdns.el8.x86_64.rpm

Can you provide me with further steps / how to generate a new GDB output? Thanks ;)

[rgacogne]

Hopefully you should be able to just install these packages and follow the same steps than earlier, only this time we should get more information like line numbers and perhaps even the content of variables.

[discostur]

@rgacogne looks better now / with more details ;)

dnsdist_17_gdb_v2.txt