Unbound-1.13.1 crashed by SIGABRT

[iruzanov]

Hello, Wouter!

I am actively using unbound-1.13.1 (with our DNSTAP patches, issue #367). And sometimes my unbound is crashing under highload, massive recursive TCP-requests. Any abnormal terminations caused by services/outside_network.c code. And now i have one of such core dumps:
(gdb) bt
#0 0x0000000800955c2a in thr_kill () from /lib/libc.so.7
#1 0x0000000800954084 in raise () from /lib/libc.so.7
#2 0x00000008008ca279 in abort () from /lib/libc.so.7
#3 0x0000000800464641 in ?? () from /usr/local/lib/libevent-2.1.so.7
#4 0x0000000800464939 in event_errx () from /usr/local/lib/libevent-2.1.so.7
#5 0x000000080045ec54 in evmap_io_del_ () from /usr/local/lib/libevent-2.1.so.7
#6 0x0000000800457e8f in event_del_nolock_ () from /usr/local/lib/libevent-2.1.so.7
#7 0x000000080045ada8 in event_del () from /usr/local/lib/libevent-2.1.so.7
#8 0x000000000030e25b in ub_event_del (ev=) at ./util/ub_event.c:395
#9 comm_point_close (c=0xdc97b7c00) at ./util/netevent.c:3860
#10 0x0000000000315bab in decommission_pending_tcp (outnet=, pend=0xdc9494980)
at ./services/outside_network.c:945
#11 0x00000000003147d6 in reuse_cb_and_decommission (outnet=0x18e75, pend=0x6, error=-2)
at ./services/outside_network.c:986
#12 0x0000000000317491 in outnet_tcptimer (arg=0xee67c2300) at ./services/outside_network.c:2033
#13 0x000000080045e0ed in ?? () from /usr/local/lib/libevent-2.1.so.7
#14 0x000000080045a09c in event_base_loop () from /usr/local/lib/libevent-2.1.so.7
#15 0x000000000024dc54 in thread_start (arg=0x8014c0800) at ./util/ub_event.c:280
#16 0x0000000800780fac in ?? () from /lib/libthr.so.3
#17 0x0000000000000000 in ?? ()
Backtrace stopped: Cannot access memory at address 0x7fffdf7fa000
(gdb)

If we enter frame 12 (outnet_tcptimer) and do print pend structure, we will see the following:
(gdb) print pend
$15 = (struct pending_tcp *) 0x6
(gdb) print *pend
Cannot access memory at address 0x6
(gdb)
And this corrupt pend structure is passing to reuse_cb_and_decommission() function (frame 11) and higher in the stacktrace output above.

In the outnet_tcptimer() function we can see the following code (in services/outside_network.c):
/* it was in use /
struct pending_tcp pend=(struct pending_tcp*)w->next_waiting;

But the structure w->next_waiting is of type waiting_tcp:
(gdb) print w->next_waiting
$18 = (struct waiting_tcp *) 0xdc9494980
(gdb)

So my question - is the types casting correct in outnet_tcptimer() function? And does this corrupt pend structure cause event_errx() in libevent?
If it might help, i found structure of pending_tcp type in w structure:
(gdb) print w->outnet->tcp_free
$23 = (struct pending_tcp *) 0xdc9494980
(gdb)
(gdb) print *w->outnet->tcp_free
$24 = {next_free = 0xdc9493e40, pi = 0xd7da2c000, c = 0xdc97b7c00, query = 0x0, reuse = {node = {parent = 0xdc94953a0,
left = 0x3287d0 <rbtree_null_node>, right = 0x3287d0 <rbtree_null_node>, key = 0x0, color = 1 '\001'}, addr = {
ss_len = 0 '\000', ss_family = 2 '\002', __ss_pad1 = "\000\065X\320\017\067", __ss_align = 0,
__ss_pad2 = "\000\000\000\000\000\000\000\016", '\000' <repeats 103 times>}, addrlen = 16, is_ssl = 0,
lru_next = 0xdc9494ae0, lru_prev = 0x0, item_on_lru_list = 0, pending = 0xdc9494980, cp_more_read_again = 0,
cp_more_write_again = 0, tree_by_id = {root = 0x3287d0 <rbtree_null_node>, count = 0,
cmp = 0x3133e0 <reuse_id_cmp>}, write_wait_first = 0x0, write_wait_last = 0x0, outnet = 0xd7d805000}}
(gdb)

Big thank you in advance!

PS I did not send core-file itself because of 31GB in size of the file.

[gthess]

Hello,

Thanks for the backtrace! This seems to be the same as #411 and #439 and the backtrace will hopefully help narrow down the cause of the corruption.
In this case w->next_waiting is supposed to be casted to struct pending_tcp* because of the w->on_tcp_waiting_list check above.
So the error is not there per say; rather at an earlier point the w struct was corrupted.

[gthess]

I also see the cause of the crash for the other issues, namely:

reuse = {node = {parent = 0xdc94953a0,
left = 0x3287d0 <rbtree_null_node>, right = 0x3287d0 <rbtree_null_node>, key = 0x0, color = 1 '\001'}

there is a node in the reuse tree with key=NULL. That is the case that shouldn't be happening in the first place.

No scratch that, this is just a node and having key=NULL just means that this node is not part of the tree.

[gthess]

Btw, can you somehow reproduce the issue reliably-ish?

[iruzanov]

I can see in my case that the key=NULL too:
(gdb) print w->outnet->tcp_free.reuse.node
$27 = {parent = 0xdc94953a0, left = 0x3287d0 <rbtree_null_node>, right = 0x3287d0 <rbtree_null_node>, key = 0x0,
color = 1 '\001'}
(gdb)

[iruzanov]

By the way, i have another core dump where the key is NULL:
Core was generated by `/usr/local/sbin/unbound -c /srvs/unbound/etc/unbound/unbound.conf'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00000000003133ea in reuse_id_cmp (key1=0xf194e0500, key2=0x0) at ./services/outside_network.c:174
174 if(w1->id < w2->id)
[Current thread is 1 (LWP 100527)]
(gdb) bt
#0 0x00000000003133ea in reuse_id_cmp (key1=0xf194e0500, key2=0x0) at ./services/outside_network.c:174
#1 0x00000000002c2f1a in rbtree_insert (rbtree=0xd810e9ab8, data=0xf194e0528) at ./util/rbtree.c:241
#2 0x0000000000314b08 in reuse_tree_by_id_insert (reuse=0xd810e99e0, w=0xf194e0500)
at ./services/outside_network.c:400
#3 use_free_buffer (outnet=0xc72751000) at ./services/outside_network.c:743
#4 0x000000000031431f in outnet_tcp_cb (c=, arg=, error=,
reply_info=) at ./services/outside_network.c:1125
#5 0x000000080045fd7d in ?? () from /usr/local/lib/libevent-2.1.so.7
#6 0x000000080045bd2c in event_base_loop () from /usr/local/lib/libevent-2.1.so.7
#7 0x000000000024dc54 in thread_start (arg=0x8014c8000) at ./util/ub_event.c:280
#8 0x0000000800783fac in ?? () from /lib/libthr.so.3
#9 0x0000000000000000 in ?? ()
Backtrace stopped: Cannot access memory at address 0x7fffdf1f7000
(gdb)
(gdb) fr 1
#1 0x00000000002c2f1a in rbtree_insert (rbtree=0xd810e9ab8, data=0xf194e0528) at ./util/rbtree.c:241
241 if ((r = rbtree->cmp(data->key, node->key)) == 0) {
(gdb) print *node
$14 = {parent = 0xf0686c928, left = 0x3287d0 <rbtree_null_node>, right = 0x3287d0 <rbtree_null_node>, key = 0x0,
color = 0 '\000'}
(gdb)

This node is appearing in while(node != RBTREE_NULL) loop (source is util/rbtree.c in rbtree_insert() function) and program is abnormally terminating when we call rbtree->cmp(data->key, node->key) in frame 0.

[iruzanov]

Btw, can you somehow reproduce the issue reliably-ish?

It would be hard. Unbound crashes under massive TCP-requests when server generating huge amount of iterative requests to upstreams. So i will need to research this traffic first to reproduce the issue in my sandbox.

[gthess]

There is a possible fix on master branch (ff6b527) for this.
It would be great if you could test and provide feedback!

[gthess]

Hi @iruzanov, just checking if you were able to test with the aforementioned fix.

[iruzanov]

Hello colleagues!
Please excuse me for long delay! I'm just very "loaded" by other tasks right now ;)

In will definitely test your patch on my load Unbound resolvers in this week. And then i will report you about results! Thank you very much for help!

[iruzanov]

Today i have patched one of my loaded Unbound resolvers. My plan is:

• to monitor the service during of two working days
• to report you about results
• to apply your patch to one more loaded resolvers in next week
• to report you about results

Thank you very much for big help!

[iruzanov]

Unbound patched on one of the most loaded servers is still running without any core dumps. Looks good. In days i'm going to patch still more loaded servers.

[gthess]

Thanks for letting us know, very good news!
Normally would you expect a core dump by now?