Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

dnsdist segfault (DoH) #7810

Closed
appliedprivacy opened this issue May 12, 2019 · 4 comments

Comments

Projects
None yet
2 participants
@appliedprivacy
Copy link
Contributor

commented May 12, 2019

  • Program: dnsdist
  • Issue type: Bug report

Short description

We run a public DoH service and switched to dnsdist today.
Minutes after pointing real traffic to it, it crashed (and got started again by systemd).

Environment

  • Operating system: Debian Buster
  • Software version:
dnsdist -V
dnsdist 1.4.0-alpha2 (Lua 5.1.4 [LuaJIT 2.1.0-beta3])
Enabled features: dns-over-tls(gnutls openssl) dns-over-https(DOH) dnscrypt ebpf fstrm ipcipher libsodium protobuf re2 recvmmsg/sendmmsg snmp systemd
  • Software source: PowerDNS repository

Steps to reproduce

We have no reproducer.

Expected behaviour

dnsdist should run

Actual behaviour

May 12 21:12:08  kernel: dnsdist/doh[11118]: segfault at 7fe0440af5a8 ip 0000560e7c0f1d98 sp 00007fe0405fccd8 error 7 in dnsdist[560e7bdeb000+327000]
May 12 21:12:08  kernel: Code: de 97 cf ff e9 a1 00 d7 ff 48 89 c5 e9 af 00 d7 ff 48 89 c5 e9 b7 00 d7 ff 48 89 c5 e9 bf 00 d7 ff 90 48 8b 07 48 85 c0 74 08 <48> c7 40 58 00 00 00 00 c3 90 66 66 2e 0f 1f 84 00 00 00 00 00 0f
May 12 21:12:08  systemd[1]: dnsdist.service: Main process exited, code=killed, status=11/SEGV
May 12 21:12:08  systemd[1]: dnsdist.service: Failed with result 'signal'.
May 12 21:12:10  systemd[1]: dnsdist.service: Service RestartSec=2s expired, scheduling restart.
May 12 21:12:10  systemd[1]: dnsdist.service: Scheduled restart job, restart counter is at 1.
May 12 21:12:10  systemd[1]: Stopped DNS Loadbalancer.
May 12 21:12:10  systemd[1]: Starting DNS Loadbalancer...
May 12 21:12:10  dnsdist[18854]: Configuration '/etc/dnsdist/dnsdist.conf' OK!
[...]
May 12 21:51:25  kernel: dnsdist/doh[18861]: segfault at 6d6f6303cc ip 0000561d9bb12d98 sp 00007f8a420dbcd8 error 6 in dnsdist[561d9b80c000+327000]
May 12 21:51:25  kernel: Code: de 97 cf ff e9 a1 00 d7 ff 48 89 c5 e9 af 00 d7 ff 48 89 c5 e9 b7 00 d7 ff 48 89 c5 e9 bf 00 d7 ff 90 48 8b 07 48 85 c0 74 08 <48> c7 40 58 00 00 00 00 c3 90 66 66 2e 0f 1f 84 00 00 00 00 00 0f
May 12 21:51:25  systemd[1]: dnsdist.service: Main process exited, code=killed, status=11/SEGV
May 12 21:51:25  systemd[1]: dnsdist.service: Failed with result 'signal'.
May 12 21:51:27  systemd[1]: dnsdist.service: Service RestartSec=2s expired, scheduling restart.
May 12 21:51:27  systemd[1]: dnsdist.service: Scheduled restart job, restart counter is at 2.
May 12 21:51:27  systemd[1]: Stopped DNS Loadbalancer.
May 12 21:51:27  systemd[1]: Starting DNS Loadbalancer...
May 12 21:51:27  dnsdist[31378]: Configuration '/etc/dnsdist/dnsdist.conf' OK!
[...]

Other information

config:

newServer({address="127.0.0.1"})
addDOHLocal("37.252.185.229", "/etc/letsencrypt/live/doh.appliedprivacy.net/fullchain.pem", "/etc/letsencrypt/live/doh.appliedprivacy.net/privkey.pem", "/query", {ciphers='ECDHE-RSA-CHACHA20-POLY1305:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256'})
addDOHLocal("2a00:63c1:a:229::2", "/etc/letsencrypt/live/doh.appliedprivacy.net/fullchain.pem", "/etc/letsencrypt/live/doh.appliedprivacy.net/privkey.pem", "/query", {ciphers='ECDHE-RSA-CHACHA20-POLY1305:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256'})
setACL({'0.0.0.0/0', '::/0'})
controlSocket('127.0.0.1:5199')
setConsoleACL('127.0.0.1/8')
setKey("-removed-")

@rgacogne rgacogne added this to the dnsdist-1.4.0 milestone May 12, 2019

@rgacogne

This comment has been minimized.

Copy link
Member

commented May 12, 2019

Thanks for reporting this! Would you be able to provide a core dump, a backtrace or perhaps a network capture of the incoming queries right before the crash?

@appliedprivacy

This comment has been minimized.

Copy link
Contributor Author

commented May 12, 2019

Thread 5 "dnsdist/doh" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff3abd700 (LWP 8310)]
on_generator_dispose (_self=0x7fffec02c4a0) at doh.cc:275
275     doh.cc: No such file or directory.
(gdb) backtrace
#0  on_generator_dispose (_self=0x7fffec02c4a0) at doh.cc:275
#1  0x00007ffff6ffb94c in h2o_mem_clear_pool () from /usr/lib/x86_64-linux-gnu/libh2o-evloop.so.0.13
#2  0x00007ffff702f14b in h2o_http2_stream_close () from /usr/lib/x86_64-linux-gnu/libh2o-evloop.so.0.13
#3  0x00007ffff702890f in ?? () from /usr/lib/x86_64-linux-gnu/libh2o-evloop.so.0.13
#4  0x00007ffff6ffe5d7 in ?? () from /usr/lib/x86_64-linux-gnu/libh2o-evloop.so.0.13
#5  0x00007ffff6ffe862 in ?? () from /usr/lib/x86_64-linux-gnu/libh2o-evloop.so.0.13
#6  0x00007ffff700087d in h2o_evloop_run () from /usr/lib/x86_64-linux-gnu/libh2o-evloop.so.0.13
#7  0x0000555555956484 in dohThread (cs=0x555555b31540) at /usr/include/c++/8/bits/shared_ptr_base.h:1018
#8  0x00007ffff6c05b2f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#9  0x00007ffff6cddfa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#10 0x00007ffff68e34cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
@rgacogne

This comment has been minimized.

Copy link
Member

commented May 13, 2019

Thank you for the backtrace, that was very helpful! We fixed a bug that looks a lot like this one in #7814,and although we can't be 100% sure it was the same bug, it seems quite likely. The PR has been merged and we are automatically building updated packages right now.
Would you by any chance be able to test the updated version when it's available? Thanks again!

@appliedprivacy

This comment has been minimized.

Copy link
Contributor Author

commented May 13, 2019

using the buster-dnsdist-master repo:

apt install dnsdist
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 dnsdist : Depends: libprotobuf17 but it is not going to be installed
           Depends: libre2-5 (>= 20131024+dfsg) but it is not going to be installed
           Depends: libsnmp30 (>= 5.7.3+dfsg) but it is not going to be installed
           Depends: libstdc++6 (>= 6) but it is not going to be installed
E: Unable to correct problems, you have held broken packages.

@zeha pointed out on the dnsdist mailing list that this issue is known and tracked as #7781

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.