Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

dnsdist segfaults in libh2o-evloop.so #9927

Closed
yannk opened this issue Jan 8, 2021 · 7 comments · Fixed by #9934
Closed

dnsdist segfaults in libh2o-evloop.so #9927

yannk opened this issue Jan 8, 2021 · 7 comments · Fixed by #9934
Assignees
Milestone

Comments

@yannk
Copy link

yannk commented Jan 8, 2021

dnsdist 1.5.1 (Lua 5.2.4)
Enabled features: cdb dns-over-tls(gnutls openssl) dns-over-https(DOH) dnscrypt ipcipher libsodium lmdb protobuf re2 recvmmsg/sendmmsg
  • Program: dnsdist
  • Issue type: Bug report

dnsdist crashes sporadically

Environment

  • Operating system: FreeBSD 12.2
  • Software version: dnsdist 1.5.1 (Lua 5.2.4)
  • Software source: self-compiled
  • tls 1.3
  • OpenSSL 1.1.1h-freebsd 22 Sep 2020

Other information

/usr/local/sbin/dnsdist:
        liblua-5.2.so => /usr/local/lib/liblua-5.2.so (0x801632000)
        libm.so.5 => /lib/libm.so.5 (0x80166b000)
        libedit.so.0 => /usr/local/lib/libedit.so.0 (0x80169d000)
        libsodium.so.23 => /usr/local/lib/libsodium.so.23 (0x8016d6000)
        libcdb.so.1 => /usr/local/lib/libcdb.so.1 (0x80177e000)
        libre2.so.6 => /usr/local/lib/libre2.so.6 (0x801784000)
        libssl.so.111 => /usr/lib/libssl.so.111 (0x8017e3000)
        libcrypto.so.111 => /lib/libcrypto.so.111 (0x80187b000)
        liblmdb.so.0 => /usr/local/lib/liblmdb.so.0 (0x801b6d000)
        libgnutls.so.30 => /usr/local/lib/libgnutls.so.30 (0x801b88000)
        libh2o-evloop.so.0.13 => /usr/local/lib/libh2o-evloop.so.0.13 (0x801d63000)
        libprotobuf.so.24 => /usr/local/lib/libprotobuf.so.24 (0x801dd5000)
        libthr.so.3 => /lib/libthr.so.3 (0x8020e7000)
        libc++.so.1 => /usr/lib/libc++.so.1 (0x802114000)
        libcxxrt.so.1 => /lib/libcxxrt.so.1 (0x8021e1000)
        libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x802203000)
        libc.so.7 => /lib/libc.so.7 (0x80221c000)
        libncurses.so.8 => /lib/libncurses.so.8 (0x802612000)
        libp11-kit.so.0 => /usr/local/lib/libp11-kit.so.0 (0x80266b000)
        libidn2.so.0 => /usr/local/lib/libidn2.so.0 (0x80278b000)
        libunistring.so.2 => /usr/local/lib/libunistring.so.2 (0x8027af000)
        libdl.so.1 => /usr/lib/libdl.so.1 (0x80296a000)
        libtasn1.so.6 => /usr/local/lib/libtasn1.so.6 (0x80296e000)
        libnettle.so.8 => /usr/local/lib/libnettle.so.8 (0x802984000)
        libhogweed.so.6 => /usr/local/lib/libhogweed.so.6 (0x8029c4000)
        libgmp.so.10 => /usr/local/lib/libgmp.so.10 (0x802a0b000)
        libintl.so.8 => /usr/local/lib/libintl.so.8 (0x802a91000)
        libz.so.6 => /lib/libz.so.6 (0x802a9f000)
        libffi.so.7 => /usr/local/lib/libffi.so.7 (0x802abb000)

--------------------------------------------------------------------------------
[R] Thread 101394 "dnsdist/doh" (faulted)
*  [  0] libh2o-evloop.so.0.13.6   h2o_set_header_by_str
        => movzx (%rdi), %eax
	-----------------------------------------------------------------------
	Signal: [11] SIGSEGV (segmentation violation)
	Signal address: 0
	-----------------------------------------------------------------------
   [  1] dnsdist                   OpenSSLTLSTicketKey::encrypt(unsigned char*, OpenSSLTLSTicketKey::encrypt, evp_cipher_ctx_st*, hmac_ctx_st*) const (libssl.cc:607)
   [  2] dnsdist                   NetmaskTree<bool>::TreeNode::TreeNode(Netmask const&) (./iputils.hh:672)
   [  3] libh2o-evloop.so.0.13.6   0x801d9fb99
   [  4] libh2o-evloop.so.0.13.6   0x801dbcdd9
   [  5] libh2o-evloop.so.0.13.6   0x801dbe4b9
   [  6] libh2o-evloop.so.0.13.6   0x801dbd851
   [  7] libh2o-evloop.so.0.13.6   0x801dbd00d
   [  8] libh2o-evloop.so.0.13.6   0x801dbb4de
   [  9] libh2o-evloop.so.0.13.6   0x801da2c50
   [ 10] libh2o-evloop.so.0.13.6   0x801d8efe2
   [ 11] libh2o-evloop.so.0.13.6   h2o_evloop_run
   [ 12] dnsdist                   SConnectWithTimeout(int, ComboAddress const&, int) (iputils.cc:71)
   [ 13] dnsdist                   DownstreamState::hash(void) (/usr/include/c++/v1/__split_buffer:0)
   [ 14] libthr.so.3               _pthread_create
   [ 15] 

[T] Thread 100284 "dnsdist"
   [  0] libthr.so.3               0x8020f9edc
   [  1] libc++.so.1               std::__1::time_get<wchar_t, std::__1::istreambuf_iterator<wchar_t, std::__1::char_traits<wchar_t> >>::do_get(std::__1::istreambuf_iterator<wchar_t, std::__1::char_traits<wchar_t> >, std::__1::istreambuf_iterator<wchar_t, std::__1::char_traits<wchar_t> >, std::__1::ios_base&, unsigned int&, tm*, char, char) const
   [  2] dnsdist                   main (dnsdist.cc:0)

[T] Thread 101123 "dnsdist/delayPi"
   [  0] libc.so.7                 0x802373fba
        => sti 
   [  1] dnsdist                   DNSProtoBufMessage::update(boost::uuids::uuid const&, ComboAddress const*, ComboAddress const, bool, unsigned short) (protobuf.cc:343)
   [  2] dnsdist                   void std::__1::__tree_remove<std::__1::__tree_node_base<void*>*>(void*, std::__1::__tree_remove<std::__1::__tree_node_base<void*>*>) (/usr/include/c++/v1/__tree:380)
   [  3] dnsdist                   _Z6genlogIJNSt3__112basic_stringIcNS0_11char_traitsIcEENS0_9allocatorIcEEEEPcEEviPKcDpT_ (./dolog.hh:85)
   [  4] dnsdist                   _ZNSt3__114__thread_proxyINS_5tupleIJNS_10unique_ptrINS_15__thread_structENS_14default_deleteIS3_EEEEPFvvEEEEEEPvSA_ (/usr/include/c++/v1/memory:2616)
   [  5] libthr.so.3               _pthread_create
   [  6] 

[T] Thread 101361 "dnsdist/control"
   [  0] libc.so.7                 0x8023618a8
        => add %al, (%rcx)
   [  1] dnsdist                   waitForMultiData(std::__1::set<int, std::__1::less<int>, std::__1::allocator<int> > const&, int, int, int*) (/usr/include/c++/v1/__tree:2136)
   [  2] dnsdist                   DynBlockRulesGroup::apply(timespec const&) (./dnsdist-dynblocks.hh:103)
   [  3] dnsdist                   std::__1::__tree_end_node<std::__1::__tree_node_base<void*>>*& std::__1::__tree_node_base<void*>*& std::__1::__tree<SuffixMatchTree<DynBlock>, std::__1::less<SuffixMatchTree<DynBlock>>, std::__1::allocator<SuffixMatchTree<DynBlock>> >::__find_equal<SuffixMatchTree<DynBlock>>(std::__1::__tree_const_iterator<SuffixMatchTree<DynBlock>, std::__1::__tree_node<SuffixMatchTree<DynBlock>, std::__1::__tree_node_base>*, long>(std::__1::__tree_node_base<void*>*, DynBlock const&) (./dnsname.hh:284)
   [  4] libthr.so.3               _pthread_create
   [  5] 

[T] Thread 101372 "dnsdist/respond"
   [  0] libc.so.7                 0x802373f98
        => mov %esi, -0x440(%rbp)
   [  1] dnsdist                   responderThread(std::__1::shared_ptr<DownstreamState>) (dnsdist.cc:226)
   [  2] dnsdist                   DownstreamState::reconnect(void) (/usr/local/include/boost/function/function_template.hpp:871)
   [  3] libthr.so.3               _pthread_create
   [  4] 

[T] Thread 101380 "dnsdist/respond"
   [  0] libc.so.7                 0x802373f98
        => mov %esi, -0x440(%rbp)
   [  1] dnsdist                   responderThread(std::__1::shared_ptr<DownstreamState>) (dnsdist.cc:226)
   [  2] dnsdist                   DownstreamState::reconnect(void) (/usr/local/include/boost/function/function_template.hpp:871)
   [  3] libthr.so.3               _pthread_create
   [  4] 

[T] Thread 101381 "dnsdist/respond"
   [  0] libc.so.7                 0x802373f98
        => mov %esi, -0x440(%rbp)
   [  1] dnsdist                   responderThread(std::__1::shared_ptr<DownstreamState>) (dnsdist.cc:226)
   [  2] dnsdist                   DownstreamState::reconnect(void) (/usr/local/include/boost/function/function_template.hpp:871)
   [  3] libthr.so.3               _pthread_create
   [  4] 

[T] Thread 101382 "dnsdist/respond"
   [  0] libc.so.7                 0x802373f98
        => mov %esi, -0x440(%rbp)
   [  1] dnsdist                   responderThread(std::__1::shared_ptr<DownstreamState>) (dnsdist.cc:226)
   [  2] dnsdist                   DownstreamState::reconnect(void) (/usr/local/include/boost/function/function_template.hpp:871)
   [  3] libthr.so.3               _pthread_create
   [  4] 

[T] Thread 101383 "dnsdist/respond"
   [  0] libc.so.7                 0x802373f98
        => mov %esi, -0x440(%rbp)
   [  1] dnsdist                   responderThread(std::__1::shared_ptr<DownstreamState>) (dnsdist.cc:226)
   [  2] dnsdist                   DownstreamState::reconnect(void) (/usr/local/include/boost/function/function_template.hpp:871)
   [  3] libthr.so.3               _pthread_create
   [  4] 

[T] Thread 101385 "dnsdist/respond"
   [  0] libc.so.7                 0x802373f98
        => mov %esi, -0x440(%rbp)
   [  1] dnsdist                   responderThread(std::__1::shared_ptr<DownstreamState>) (dnsdist.cc:226)
   [  2] dnsdist                   DownstreamState::reconnect(void) (/usr/local/include/boost/function/function_template.hpp:871)
   [  3] libthr.so.3               _pthread_create
   [  4] 

[T] Thread 101386 "dnsdist/respond"
   [  0] libc.so.7                 0x802373f98
        => mov %esi, -0x440(%rbp)
   [  1] dnsdist                   responderThread(std::__1::shared_ptr<DownstreamState>) (dnsdist.cc:226)
   [  2] dnsdist                   DownstreamState::reconnect(void) (/usr/local/include/boost/function/function_template.hpp:871)
   [  3] libthr.so.3               _pthread_create
   [  4] 

[T] Thread 101388 "dnsdist/respond"
   [  0] libc.so.7                 0x802373f98
        => mov %esi, -0x440(%rbp)
   [  1] dnsdist                   responderThread(std::__1::shared_ptr<DownstreamState>) (dnsdist.cc:226)
   [  2] dnsdist                   DownstreamState::reconnect(void) (/usr/local/include/boost/function/function_template.hpp:871)
   [  3] libthr.so.3               _pthread_create
   [  4] 

[T] Thread 101391 "dnsdist/respond"
   [  0] libc.so.7                 0x802373f98
        => mov %esi, -0x440(%rbp)
   [  1] dnsdist                   responderThread(std::__1::shared_ptr<DownstreamState>) (dnsdist.cc:226)
   [  2] dnsdist                   DownstreamState::reconnect(void) (/usr/local/include/boost/function/function_template.hpp:871)
   [  3] libthr.so.3               _pthread_create
   [  4] 

[T] Thread 101392 "dnsdist/respond"
   [  0] libc.so.7                 0x802373f98
        => mov %esi, -0x440(%rbp)
   [  1] dnsdist                   responderThread(std::__1::shared_ptr<DownstreamState>) (dnsdist.cc:226)
   [  2] dnsdist                   DownstreamState::reconnect(void) (/usr/local/include/boost/function/function_template.hpp:871)
   [  3] libthr.so.3               _pthread_create
   [  4] 

[T] Thread 101393 "dnsdist/webserv"
   [  0] libc.so.7                 0x8023618a8
        => add %al, (%rcx)
   [  1] dnsdist                   waitForMultiData(std::__1::set<int, std::__1::less<int>, std::__1::allocator<int> > const&, int, int, int*) (/usr/include/c++/v1/__tree:2136)
   [  2] dnsdist                   connectionThread(int, ComboAddress) (dnsdist-web.cc:936)
   [  3] dnsdist                   _ZZN10LuaContext6PusherINSt3__110shared_ptrI15DNSCryptContextEEvE4pushIS4_EENS_12PushedObjectEP9lua_StateOT_ENKUlS9_E0_clES9_ (./ext/luawrapper/include/LuaContext.hpp:1514)
   [  4] libthr.so.3               _pthread_create
   [  5] 

[T] Thread 101652 "dnsdist/doh"
   [  0] libc.so.7                 0x8023ba00a
        => test %cl, %cl
   [  1] libh2o-evloop.so.0.13.6   h2o_evloop_run
   [  2] dnsdist                   SConnectWithTimeout(int, ComboAddress const&, int) (iputils.cc:71)
   [  3] dnsdist                   DownstreamState::hash(void) (/usr/include/c++/v1/__split_buffer:0)
   [  4] libthr.so.3               _pthread_create
   [  5] 

[T] Thread 101653 "dnsdist/carbon"
   [  0] libc.so.7                 svis
        => add %al, (%rax)
   [  1] libc.so.7                 setproctitle_fast
   [  2] dnsdist                   void boost::algorithm::detail::find_format_all_impl2<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, boost::algorithm::detail::first_finderF<char const*, boost::algorithm::is_equal>, boost::algorithm::detail::const_formatF<boost::iterator_range<char const*> >, boost::iterator_range<std::__1::__wrap_iter<char*> >, boost::iterator_range<char const*>>(char&, std::__1::char_traits<char>, std::__1::allocator<char>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, char const*) (/usr/local/include/boost/algorithm/string/detail/find_format_all.hpp:0)
   [  3] dnsdist                   DownstreamState::hash(void) (./lock.hh:0)
   [  4] libthr.so.3               _pthread_create
   [  5] 

[T] Thread 101660 "dnsdist/main"
   [  0] libc.so.7                 svis
        => add %al, (%rax)
   [  1] libc.so.7                 setproctitle_fast
   [  2] dnsdist                   healthChecksThread(void) (dnsdist.cc:1687)
   [  3] dnsdist                   DownstreamState::hash(void) (./lock.hh:0)
   [  4] libthr.so.3               _pthread_create
   [  5] 

[T] Thread 101662 "dnsdist/healthC"
   [  0] libc.so.7                 svis
        => add %al, (%rax)
   [  1] libc.so.7                 setproctitle_fast
   [  2] dnsdist                   processUDPQuery(ClientState&, LocalHolders&, msghdr const*, ComboAddress const&, ComboAddress&, char*, unsigned short, unsigned long, mmsghdr*, unsigned int*, iovec*, cmsgbuf_aligned*) (/usr/include/c++/v1/string:1499)
   [  3] dnsdist                   DownstreamState::hash(void) (./lock.hh:0)
   [  4] libthr.so.3               _pthread_create
   [  5] 

[T] Thread 101671 "dnsdist/doh-cli"
   [  0] libc.so.7                 0x8023d1c88
        => add (%rax), %al
 ? [  1] dnsdist                   SConnectWithTimeout(int, ComboAddress const&, int) (/usr/include/c++/v1/__bit_reference:54)
 ? [  2] dnsdist                   connectionThread(int, ComboAddress) (/usr/include/c++/v1/ostream:869)
 ? [  3] libthr.so.3               _pthread_create
 ? [  4] 

[T] Thread 101673 "dnsdist/doh-cli"
   [  0] libc.so.7                 0x8023d1c88
        => add (%rax), %al
 ? [  1] dnsdist                   SConnectWithTimeout(int, ComboAddress const&, int) (/usr/include/c++/v1/__bit_reference:54)
 ? [  2] dnsdist                   connectionThread(int, ComboAddress) (/usr/include/c++/v1/ostream:869)
 ? [  3] libthr.so.3               _pthread_create
 ? [  4] 

@rgacogne
Copy link
Member

rgacogne commented Jan 8, 2021

Would you mind posting your configuration? The backtraces don't really make sense to me but if the crash really does happen in h2o_set_header_by_str(), we only call that directly if there are some custom response headers set.

@yannk
Copy link
Author

yannk commented Jan 8, 2021

Correct,

Here is the relevant configuration:

healthy   = newDOHResponseMapEntry('^/dohhealth$', 200, 'OK', {['Cache-Control']='max-age=15'})
unhealthy = newDOHResponseMapEntry('^/dohhealth$', 503, 'NOT_OK', {['Cache-Control']='max-age=15'})
foo       = newDOHResponseMapEntry('^/some/other/url$', 200, [[foo]], {['Cache-Control']='max-age=15'})

-- maintenance adds foo and the health endpoint and is called at most every second
function maintenance()
        h = healthy
        for key, s in pairs(getServers()) do
                if not s:isUp() then
                        h=unhealthy
                        break
                end
        end

        for i=0,getDOHFrontendCount()-1 do
                getDOHFrontend(i):setResponsesMap({h, foo})
        end
end

@rgacogne
Copy link
Member

rgacogne commented Jan 9, 2021

Right, that makes sense because I don't think we ever expected the response map to be set at runtime like that, and it's not protected by any kind of lock, so there is a race condition there that could completely explain the crash. I'll work on a patch next week.

@rgacogne rgacogne self-assigned this Jan 11, 2021
@rgacogne rgacogne added this to the dnsdist-1.5.x milestone Jan 11, 2021
@rgacogne
Copy link
Member

I can reproduce the crash, I'll have a fix soon.

@rgacogne
Copy link
Member

I opened a pull request (#9934) with a patch that fixes the issue for me, but it would be very much appreciated if you could test it and confirm whether the crashes vanish or not :)

@yannk
Copy link
Author

yannk commented Jan 12, 2021

I've deployed with the patch and observing for crashes. So far so good. Thank you for the prompt fix!

@rgacogne
Copy link
Member

Great, thanks a lot for the initial report and for testing the fix!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants