Describe the bug
Unbound appears to leak memory when exposed to DoH traffic load.
To reproduce
Steps to reproduce the behavior:
- Configure unbound to handle DoH
- Query it repeatedly with "dig +https ..."
- Watch it bloat in size, eventually exhausting memory
- Decide to dig deeper and build with
gcc -fsanitize=leak
Result of number 4 is shown below.
Expected behavior
Unbound should not leak memory...
System:
- Unbound version: 1.22.0
- OS: NetBSD/amd64 10.99.12 of Mon Apr 7 09:53:29 UTC 2025 vintage
unbound -V output:
$ /usr/pkg/sbin/unbound -V
Version 1.22.0
Configure line: --enable-allsymbols --with-libexpat=/usr --with-libevent=/usr --enable-event-api --with-ssl=/usr --with-syslog-facility= --with-pidfile=/var/run/unbound/unbound.pid --sysconfdir=/usr/pkg/etc --enable-dnstap --with-libnghttp2=/usr/pkg --enable-gost --enable-sha2 --prefix=/usr/pkg --build=x86_64--netbsd --host=x86_64--netbsd --mandir=/usr/pkg/man --enable-option-checking=yes
Linked libs: pluggable-libevent 2.1.12-stable (it uses kqueue), OpenSSL 3.0.15 3 Sep 2024
Linked modules: dns64 respip validator iterator
BSD licensed, see LICENSE in source package for details.
Report bugs to unbound-bugs@nlnetlabs.nl or https://github.com/NLnetLabs/unbound/issues
$
$ pkg_info | grep nghttp2
nghttp2-1.65.0 Implementation of HTTP/2 in C
$
Additional information
The result of an unbound run with LSAN:
# /usr/pkg/sbin/unbound -d -c /usr/pkg/etc/unbound/unbound.conf
==28280==Installed the sigaction for signal 11
==28280==Installed the sigaction for signal 10
==28280==Installed the sigaction for signal 8
[1744187191] unbound[28280:0] warning: the ulimit(data seg size) is smaller than the expected memory usage (added size of caches). 268435456 < 3762184960 bytes
==21602==Processing thread 28280.
==21602==Stack at 0x7f7fffff0000-0x7f7ffffff000 (SP = 0x7f7fffffe458).
=================================================================
==28280==ERROR: LeakSanitizer: detected memory leaks
Direct leak of 3785952 byte(s) in 1396 object(s) allocated from:
#0 0x7f7ff7952e49 (/usr/lib/liblsan.so.3+0x8e49)
#1 0x7f7ff72a4e4a (/usr/pkg/lib/libnghttp2.so.14+0x9e4a)
#2 0x7f7ff72a54c7 (/usr/pkg/lib/libnghttp2.so.14+0xa4c7)
#3 0x7f7ff78d1333 (/usr/pkg/lib/libunbound.so.8+0xc6333)
#4 0x7f7ff77322db (/usr/lib/libevent.so.5+0x382db)
#5 0x7f7ff7732926 (/usr/lib/libevent.so.5+0x38926)
#6 0x7f7ff78cef95 (/usr/pkg/lib/libunbound.so.8+0xc3f95)
#7 0x17bbf (/usr/pkg/sbin/unbound+0x16bbf)
#8 0x7f7ff76f02e0 (/usr/lib/libpthread.so.1+0xd2e0)
Direct leak of 1296336 byte(s) in 478 object(s) allocated from:
#0 0x7f7ff7952e49 (/usr/lib/liblsan.so.3+0x8e49)
#1 0x7f7ff72a4e4a (/usr/pkg/lib/libnghttp2.so.14+0x9e4a)
#2 0x7f7ff72a54c7 (/usr/pkg/lib/libnghttp2.so.14+0xa4c7)
#3 0x7f7ff78d1333 (/usr/pkg/lib/libunbound.so.8+0xc6333)
#4 0x7f7ff77322db (/usr/lib/libevent.so.5+0x382db)
#5 0x7f7ff7732926 (/usr/lib/libevent.so.5+0x38926)
#6 0x7f7ff78cef95 (/usr/pkg/lib/libunbound.so.8+0xc3f95)
#7 0x18826 (/usr/pkg/sbin/unbound+0x17826)
#8 0x37747 (/usr/pkg/sbin/unbound+0x36747)
#9 0x140aa (/usr/pkg/sbin/unbound+0x130aa)
#10 0x7f7ff7ef5f57 (/usr/libexec/ld.elf_so+0xdf57)
Indirect leak of 22886024 byte(s) in 1396 object(s) allocated from:
#0 0x7f7ff7951b51 (/usr/lib/liblsan.so.3+0x7b51)
#1 0x7f7ff72a37af (/usr/pkg/lib/libnghttp2.so.14+0x87af)
#2 0x7f7ff72a3862 (/usr/pkg/lib/libnghttp2.so.14+0x8862)
#3 0x7f7ff72a3ac9 (/usr/pkg/lib/libnghttp2.so.14+0x8ac9)
#4 0x7f7ff72a518d (/usr/pkg/lib/libnghttp2.so.14+0xa18d)
#5 0x7f7ff72a54c7 (/usr/pkg/lib/libnghttp2.so.14+0xa4c7)
#6 0x7f7ff78d1333 (/usr/pkg/lib/libunbound.so.8+0xc6333)
#7 0x7f7ff77322db (/usr/lib/libevent.so.5+0x382db)
#8 0x7f7ff7732926 (/usr/lib/libevent.so.5+0x38926)
#9 0x7f7ff78cef95 (/usr/pkg/lib/libunbound.so.8+0xc3f95)
#10 0x17bbf (/usr/pkg/sbin/unbound+0x16bbf)
#11 0x7f7ff76f02e0 (/usr/lib/libpthread.so.1+0xd2e0)
Indirect leak of 7836332 byte(s) in 478 object(s) allocated from:
#0 0x7f7ff7951b51 (/usr/lib/liblsan.so.3+0x7b51)
#1 0x7f7ff72a37af (/usr/pkg/lib/libnghttp2.so.14+0x87af)
#2 0x7f7ff72a3862 (/usr/pkg/lib/libnghttp2.so.14+0x8862)
#3 0x7f7ff72a3ac9 (/usr/pkg/lib/libnghttp2.so.14+0x8ac9)
#4 0x7f7ff72a518d (/usr/pkg/lib/libnghttp2.so.14+0xa18d)
#5 0x7f7ff72a54c7 (/usr/pkg/lib/libnghttp2.so.14+0xa4c7)
#6 0x7f7ff78d1333 (/usr/pkg/lib/libunbound.so.8+0xc6333)
#7 0x7f7ff77322db (/usr/lib/libevent.so.5+0x382db)
#8 0x7f7ff7732926 (/usr/lib/libevent.so.5+0x38926)
#9 0x7f7ff78cef95 (/usr/pkg/lib/libunbound.so.8+0xc3f95)
#10 0x18826 (/usr/pkg/sbin/unbound+0x17826)
#11 0x37747 (/usr/pkg/sbin/unbound+0x36747)
#12 0x140aa (/usr/pkg/sbin/unbound+0x130aa)
#13 0x7f7ff7ef5f57 (/usr/libexec/ld.elf_so+0xdf57)
Indirect leak of 1429504 byte(s) in 1396 object(s) allocated from:
#0 0x7f7ff7953299 (/usr/lib/liblsan.so.3+0x9299)
#1 0x7f7ff72b0be4 (/usr/pkg/lib/libnghttp2.so.14+0x15be4)
#2 0x7f7ff72a5134 (/usr/pkg/lib/libnghttp2.so.14+0xa134)
#3 0x7f7ff72a54c7 (/usr/pkg/lib/libnghttp2.so.14+0xa4c7)
#4 0x7f7ff78d1333 (/usr/pkg/lib/libunbound.so.8+0xc6333)
#5 0x7f7ff77322db (/usr/lib/libevent.so.5+0x382db)
#6 0x7f7ff7732926 (/usr/lib/libevent.so.5+0x38926)
#7 0x7f7ff78cef95 (/usr/pkg/lib/libunbound.so.8+0xc3f95)
#8 0x17bbf (/usr/pkg/sbin/unbound+0x16bbf)
#9 0x7f7ff76f02e0 (/usr/lib/libpthread.so.1+0xd2e0)
Indirect leak of 1429504 byte(s) in 1396 object(s) allocated from:
#0 0x7f7ff7953299 (/usr/lib/liblsan.so.3+0x9299)
#1 0x7f7ff72b0ac9 (/usr/pkg/lib/libnghttp2.so.14+0x15ac9)
#2 0x7f7ff72a5117 (/usr/pkg/lib/libnghttp2.so.14+0xa117)
#3 0x7f7ff72a54c7 (/usr/pkg/lib/libnghttp2.so.14+0xa4c7)
#4 0x7f7ff78d1333 (/usr/pkg/lib/libunbound.so.8+0xc6333)
#5 0x7f7ff77322db (/usr/lib/libevent.so.5+0x382db)
#6 0x7f7ff7732926 (/usr/lib/libevent.so.5+0x38926)
#7 0x7f7ff78cef95 (/usr/pkg/lib/libunbound.so.8+0xc3f95)
#8 0x17bbf (/usr/pkg/sbin/unbound+0x16bbf)
#9 0x7f7ff76f02e0 (/usr/lib/libpthread.so.1+0xd2e0)
Indirect leak of 489472 byte(s) in 478 object(s) allocated from:
#0 0x7f7ff7953299 (/usr/lib/liblsan.so.3+0x9299)
#1 0x7f7ff72b0be4 (/usr/pkg/lib/libnghttp2.so.14+0x15be4)
#2 0x7f7ff72a5134 (/usr/pkg/lib/libnghttp2.so.14+0xa134)
#3 0x7f7ff72a54c7 (/usr/pkg/lib/libnghttp2.so.14+0xa4c7)
#4 0x7f7ff78d1333 (/usr/pkg/lib/libunbound.so.8+0xc6333)
#5 0x7f7ff77322db (/usr/lib/libevent.so.5+0x382db)
#6 0x7f7ff7732926 (/usr/lib/libevent.so.5+0x38926)
#7 0x7f7ff78cef95 (/usr/pkg/lib/libunbound.so.8+0xc3f95)
#8 0x18826 (/usr/pkg/sbin/unbound+0x17826)
#9 0x37747 (/usr/pkg/sbin/unbound+0x36747)
#10 0x140aa (/usr/pkg/sbin/unbound+0x130aa)
#11 0x7f7ff7ef5f57 (/usr/libexec/ld.elf_so+0xdf57)
Indirect leak of 489472 byte(s) in 478 object(s) allocated from:
#0 0x7f7ff7953299 (/usr/lib/liblsan.so.3+0x9299)
#1 0x7f7ff72b0ac9 (/usr/pkg/lib/libnghttp2.so.14+0x15ac9)
#2 0x7f7ff72a5117 (/usr/pkg/lib/libnghttp2.so.14+0xa117)
#3 0x7f7ff72a54c7 (/usr/pkg/lib/libnghttp2.so.14+0xa4c7)
#4 0x7f7ff78d1333 (/usr/pkg/lib/libunbound.so.8+0xc6333)
#5 0x7f7ff77322db (/usr/lib/libevent.so.5+0x382db)
#6 0x7f7ff7732926 (/usr/lib/libevent.so.5+0x38926)
#7 0x7f7ff78cef95 (/usr/pkg/lib/libunbound.so.8+0xc3f95)
#8 0x18826 (/usr/pkg/sbin/unbound+0x17826)
#9 0x37747 (/usr/pkg/sbin/unbound+0x36747)
#10 0x140aa (/usr/pkg/sbin/unbound+0x130aa)
#11 0x7f7ff7ef5f57 (/usr/libexec/ld.elf_so+0xdf57)
Indirect leak of 223360 byte(s) in 1396 object(s) allocated from:
#0 0x7f7ff7953299 (/usr/lib/liblsan.so.3+0x9299)
#1 0x7f7ff72a826e (/usr/pkg/lib/libnghttp2.so.14+0xd26e)
#2 0x7f7ff78d1376 (/usr/pkg/lib/libunbound.so.8+0xc6376)
#3 0x7f7ff77322db (/usr/lib/libevent.so.5+0x382db)
#4 0x7f7ff7732926 (/usr/lib/libevent.so.5+0x38926)
#5 0x7f7ff78cef95 (/usr/pkg/lib/libunbound.so.8+0xc3f95)
#6 0x17bbf (/usr/pkg/sbin/unbound+0x16bbf)
#7 0x7f7ff76f02e0 (/usr/lib/libpthread.so.1+0xd2e0)
Indirect leak of 76480 byte(s) in 478 object(s) allocated from:
#0 0x7f7ff7953299 (/usr/lib/liblsan.so.3+0x9299)
#1 0x7f7ff72a826e (/usr/pkg/lib/libnghttp2.so.14+0xd26e)
#2 0x7f7ff78d1376 (/usr/pkg/lib/libunbound.so.8+0xc6376)
#3 0x7f7ff77322db (/usr/lib/libevent.so.5+0x382db)
#4 0x7f7ff7732926 (/usr/lib/libevent.so.5+0x38926)
#5 0x7f7ff78cef95 (/usr/pkg/lib/libunbound.so.8+0xc3f95)
#6 0x18826 (/usr/pkg/sbin/unbound+0x17826)
#7 0x37747 (/usr/pkg/sbin/unbound+0x36747)
#8 0x140aa (/usr/pkg/sbin/unbound+0x130aa)
#9 0x7f7ff7ef5f57 (/usr/libexec/ld.elf_so+0xdf57)
Indirect leak of 67008 byte(s) in 1396 object(s) allocated from:
#0 0x7f7ff7953299 (/usr/lib/liblsan.so.3+0x9299)
#1 0x7f7ff72a3817 (/usr/pkg/lib/libnghttp2.so.14+0x8817)
#2 0x7f7ff72a3ac9 (/usr/pkg/lib/libnghttp2.so.14+0x8ac9)
#3 0x7f7ff72a518d (/usr/pkg/lib/libnghttp2.so.14+0xa18d)
#4 0x7f7ff72a54c7 (/usr/pkg/lib/libnghttp2.so.14+0xa4c7)
#5 0x7f7ff78d1333 (/usr/pkg/lib/libunbound.so.8+0xc6333)
#6 0x7f7ff77322db (/usr/lib/libevent.so.5+0x382db)
#7 0x7f7ff7732926 (/usr/lib/libevent.so.5+0x38926)
#8 0x7f7ff78cef95 (/usr/pkg/lib/libunbound.so.8+0xc3f95)
#9 0x17bbf (/usr/pkg/sbin/unbound+0x16bbf)
#10 0x7f7ff76f02e0 (/usr/lib/libpthread.so.1+0xd2e0)
Indirect leak of 33504 byte(s) in 1396 object(s) allocated from:
#0 0x7f7ff7953299 (/usr/lib/liblsan.so.3+0x9299)
#1 0x7f7ff72a82a9 (/usr/pkg/lib/libnghttp2.so.14+0xd2a9)
#2 0x7f7ff78d1376 (/usr/pkg/lib/libunbound.so.8+0xc6376)
#3 0x7f7ff77322db (/usr/lib/libevent.so.5+0x382db)
#4 0x7f7ff7732926 (/usr/lib/libevent.so.5+0x38926)
#5 0x7f7ff78cef95 (/usr/pkg/lib/libunbound.so.8+0xc3f95)
#6 0x17bbf (/usr/pkg/sbin/unbound+0x16bbf)
#7 0x7f7ff76f02e0 (/usr/lib/libpthread.so.1+0xd2e0)
Indirect leak of 22944 byte(s) in 478 object(s) allocated from:
#0 0x7f7ff7953299 (/usr/lib/liblsan.so.3+0x9299)
#1 0x7f7ff72a3817 (/usr/pkg/lib/libnghttp2.so.14+0x8817)
#2 0x7f7ff72a3ac9 (/usr/pkg/lib/libnghttp2.so.14+0x8ac9)
#3 0x7f7ff72a518d (/usr/pkg/lib/libnghttp2.so.14+0xa18d)
#4 0x7f7ff72a54c7 (/usr/pkg/lib/libnghttp2.so.14+0xa4c7)
#5 0x7f7ff78d1333 (/usr/pkg/lib/libunbound.so.8+0xc6333)
#6 0x7f7ff77322db (/usr/lib/libevent.so.5+0x382db)
#7 0x7f7ff7732926 (/usr/lib/libevent.so.5+0x38926)
#8 0x7f7ff78cef95 (/usr/pkg/lib/libunbound.so.8+0xc3f95)
#9 0x18826 (/usr/pkg/sbin/unbound+0x17826)
#10 0x37747 (/usr/pkg/sbin/unbound+0x36747)
#11 0x140aa (/usr/pkg/sbin/unbound+0x130aa)
#12 0x7f7ff7ef5f57 (/usr/libexec/ld.elf_so+0xdf57)
Indirect leak of 11472 byte(s) in 478 object(s) allocated from:
#0 0x7f7ff7953299 (/usr/lib/liblsan.so.3+0x9299)
#1 0x7f7ff72a82a9 (/usr/pkg/lib/libnghttp2.so.14+0xd2a9)
#2 0x7f7ff78d1376 (/usr/pkg/lib/libunbound.so.8+0xc6376)
#3 0x7f7ff77322db (/usr/lib/libevent.so.5+0x382db)
#4 0x7f7ff7732926 (/usr/lib/libevent.so.5+0x38926)
#5 0x7f7ff78cef95 (/usr/pkg/lib/libunbound.so.8+0xc3f95)
#6 0x18826 (/usr/pkg/sbin/unbound+0x17826)
#7 0x37747 (/usr/pkg/sbin/unbound+0x36747)
#8 0x140aa (/usr/pkg/sbin/unbound+0x130aa)
#9 0x7f7ff7ef5f57 (/usr/libexec/ld.elf_so+0xdf57)
Indirect leak of 11168 byte(s) in 1396 object(s) allocated from:
#0 0x7f7ff7953299 (/usr/lib/liblsan.so.3+0x9299)
#1 0x7f7ff72a3272 (/usr/pkg/lib/libnghttp2.so.14+0x8272)
#2 0x7f7ff72a828a (/usr/pkg/lib/libnghttp2.so.14+0xd28a)
#3 0x7f7ff78d1376 (/usr/pkg/lib/libunbound.so.8+0xc6376)
#4 0x7f7ff77322db (/usr/lib/libevent.so.5+0x382db)
#5 0x7f7ff7732926 (/usr/lib/libevent.so.5+0x38926)
#6 0x7f7ff78cef95 (/usr/pkg/lib/libunbound.so.8+0xc3f95)
#7 0x17bbf (/usr/pkg/sbin/unbound+0x16bbf)
#8 0x7f7ff76f02e0 (/usr/lib/libpthread.so.1+0xd2e0)
Indirect leak of 11168 byte(s) in 1396 object(s) allocated from:
#0 0x7f7ff7953299 (/usr/lib/liblsan.so.3+0x9299)
#1 0x7f7ff72a3272 (/usr/pkg/lib/libnghttp2.so.14+0x8272)
#2 0x7f7ff72a83be (/usr/pkg/lib/libnghttp2.so.14+0xd3be)
#3 0x7f7ff78d1376 (/usr/pkg/lib/libunbound.so.8+0xc6376)
#4 0x7f7ff77322db (/usr/lib/libevent.so.5+0x382db)
#5 0x7f7ff7732926 (/usr/lib/libevent.so.5+0x38926)
#6 0x7f7ff78cef95 (/usr/pkg/lib/libunbound.so.8+0xc3f95)
#7 0x17bbf (/usr/pkg/sbin/unbound+0x16bbf)
#8 0x7f7ff76f02e0 (/usr/lib/libpthread.so.1+0xd2e0)
Indirect leak of 3824 byte(s) in 478 object(s) allocated from:
#0 0x7f7ff7953299 (/usr/lib/liblsan.so.3+0x9299)
#1 0x7f7ff72a3272 (/usr/pkg/lib/libnghttp2.so.14+0x8272)
#2 0x7f7ff72a83be (/usr/pkg/lib/libnghttp2.so.14+0xd3be)
#3 0x7f7ff78d1376 (/usr/pkg/lib/libunbound.so.8+0xc6376)
#4 0x7f7ff77322db (/usr/lib/libevent.so.5+0x382db)
#5 0x7f7ff7732926 (/usr/lib/libevent.so.5+0x38926)
#6 0x7f7ff78cef95 (/usr/pkg/lib/libunbound.so.8+0xc3f95)
#7 0x18826 (/usr/pkg/sbin/unbound+0x17826)
#8 0x37747 (/usr/pkg/sbin/unbound+0x36747)
#9 0x140aa (/usr/pkg/sbin/unbound+0x130aa)
#10 0x7f7ff7ef5f57 (/usr/libexec/ld.elf_so+0xdf57)
Indirect leak of 3824 byte(s) in 478 object(s) allocated from:
#0 0x7f7ff7953299 (/usr/lib/liblsan.so.3+0x9299)
#1 0x7f7ff72a3272 (/usr/pkg/lib/libnghttp2.so.14+0x8272)
#2 0x7f7ff72a828a (/usr/pkg/lib/libnghttp2.so.14+0xd28a)
#3 0x7f7ff78d1376 (/usr/pkg/lib/libunbound.so.8+0xc6376)
#4 0x7f7ff77322db (/usr/lib/libevent.so.5+0x382db)
#5 0x7f7ff7732926 (/usr/lib/libevent.so.5+0x38926)
#6 0x7f7ff78cef95 (/usr/pkg/lib/libunbound.so.8+0xc3f95)
#7 0x18826 (/usr/pkg/sbin/unbound+0x17826)
#8 0x37747 (/usr/pkg/sbin/unbound+0x36747)
#9 0x140aa (/usr/pkg/sbin/unbound+0x130aa)
#10 0x7f7ff7ef5f57 (/usr/libexec/ld.elf_so+0xdf57)
SUMMARY: LeakSanitizer: 40107348 byte(s) leaked in 16866 allocation(s).
#
Looking at this reveals that there are two places in libunbound.so which are involved in these allocations:
#3 0x7f7ff78d1333 (/usr/pkg/lib/libunbound.so.8+0xc6333)
and
#2 0x7f7ff78d1376 (/usr/pkg/lib/libunbound.so.8+0xc6376)
and looking at this with gdb (yes, I've turned off address space randomization for this executable):
(gdb) b main
Breakpoint 1 at 0x36290: file ./daemon/unbound.c, line 774.
(gdb) r
Starting program: /usr/pkg/sbin/unbound
Breakpoint 1, main (argc=1, argv=0x7f7fffffe758) at ./daemon/unbound.c:774
774 {
(gdb) i li *0x7f7ff78d1333
Line 3159 of "./util/netevent.c"
starts at address 0x7f7ff78d1327 <comm_point_tcp_accept_callback+173>
and ends at 0x7f7ff78d133f <comm_point_tcp_accept_callback+197>.
(gdb) list util/netevent.c:3159
3154 /** Create http2 session server. Per connection, after TCP accepted.*/
3155 static int http2_session_server_create(struct http2_session* h2_session)
3156 {
3157 log_assert(h2_session->callbacks);
3158 h2_session->is_drop = 0;
3159 if(nghttp2_session_server_new(&h2_session->session,
3160 h2_session->callbacks,
3161 h2_session) == NGHTTP2_ERR_NOMEM) {
3162 log_err("failed to create nghttp2 session server");
3163 return 0;
(gdb) i li *0x7f7ff78d1376
Line 3177 of "./util/netevent.c"
starts at address 0x7f7ff78d1367 <comm_point_tcp_accept_callback+237>
and ends at 0x7f7ff78d1377 <comm_point_tcp_accept_callback+253>.
(gdb) l 3177
3172 int ret;
3173 nghttp2_settings_entry settings[1] = {
3174 {NGHTTP2_SETTINGS_MAX_CONCURRENT_STREAMS,
3175 h2_session->c->http2_max_streams}};
3176
3177 ret = nghttp2_submit_settings(h2_session->session, NGHTTP2_FLAG_NONE,
3178 settings, 1);
3179 if(ret) {
3180 verbose(VERB_QUERY, "http2: submit_settings failed, "
3181 "error: %s", nghttp2_strerror(ret));
(gdb)
Describe the bug
Unbound appears to leak memory when exposed to DoH traffic load.
To reproduce
Steps to reproduce the behavior:
gcc -fsanitize=leakResult of number 4 is shown below.
Expected behavior
Unbound should not leak memory...
System:
unbound -Voutput:Additional information
The result of an unbound run with LSAN:
Looking at this reveals that there are two places in
libunbound.sowhich are involved in these allocations:and
and looking at this with gdb (yes, I've turned off address space randomization for this executable):