Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Bgpd crash on shutdown when have VRFs and leaked routes #2184

Closed
louberger opened this issue May 8, 2018 · 8 comments
Closed

Bgpd crash on shutdown when have VRFs and leaked routes #2184

louberger opened this issue May 8, 2018 · 8 comments
Labels
bug

Comments

@louberger
Copy link
Member

louberger commented May 8, 2018 •
edited
Loading

As seen in https://github.com/FRRouting/topotests/tree/master/bgp_l3vpn_to_bgp_vrf

Examination of the debug outputs show that a peer goes down in one thread while shutdown is taking place in the other thread. This results in the crash. Experimentation confirms this...
@qlyoung
Copy link
Member

qlyoung commented May 8, 2018

Examination of the debug outputs show that appear goes down in one thread...

I think there's a typo there, not sure what is being referred to.

@louberger
Copy link
Member Author

louberger commented May 8, 2018 via email

@qlyoung
Copy link
Member

qlyoung commented May 8, 2018

Oh, you must have been trying to say "a peer" and not "appear". Thanks for the clarification.

@louberger louberger mentioned this issue May 9, 2018
Merged
@qlyoung qlyoung added the bug label May 13, 2018
@qlyoung qlyoung closed this as completed May 13, 2018
@louberger
Copy link
Member Author

race condition avoided, but still have crash, see #2198 -- which is a band aid around a double free.

@louberger louberger reopened this May 13, 2018
@louberger louberger changed the title Bgpd crash on shutdown due to race condition Bgpd crash on shutdown when have VRFs and leaked routes May 13, 2018
@louberger
Copy link
Member Author

Asan shows the issue as being on vrf leaked routes:
ubuntu17-1> cat /tmp/topotests/bgp_l3vpn_to_bgp_vrf.test_bgp_l3vpn_to_bgp_vrf/r4/bgpd.err

==4520==ERROR: AddressSanitizer: heap-use-after-free on address 0x6070002109a8 at pc 0x55c5995068dd bp 0x7fff56442010 sp 0x7fff56442000
READ of size 4 at 0x6070002109a8 thread T0
#0 0x55c5995068dc in path_nh_map /home/lberger/Code/github/labn/working/bgpd/bgp_nht.c:772
#1 0x55c5995068dc in bgp_unlink_nexthop /home/lberger/Code/github/labn/working/bgpd/bgp_nht.c:113
#2 0x55c59947a40a in bgp_info_free /home/lberger/Code/github/labn/working/bgpd/bgp_route.c:211
#3 0x55c59947a40a in bgp_info_unlock /home/lberger/Code/github/labn/working/bgpd/bgp_route.c:241
#4 0x55c59947a6c8 in bgp_info_reap /home/lberger/Code/github/labn/working/bgpd/bgp_route.c:285
#5 0x55c59947a88c in bgp_clear_route_table /home/lberger/Code/github/labn/working/bgpd/bgp_route.c:3914
#6 0x55c599483df4 in bgp_clear_route /home/lberger/Code/github/labn/working/bgpd/bgp_route.c:3962
#7 0x55c599483f93 in bgp_clear_route_all /home/lberger/Code/github/labn/working/bgpd/bgp_route.c:3980
#8 0x55c5994611a8 in bgp_fsm_change_status /home/lberger/Code/github/labn/working/bgpd/bgp_fsm.c:926
#9 0x55c599448d95 in peer_delete /home/lberger/Code/github/labn/working/bgpd/bgpd.c:2192
#10 0x55c59944d326 in bgp_delete /home/lberger/Code/github/labn/working/bgpd/bgpd.c:3309
#11 0x55c59943a711 in bgp_exit /home/lberger/Code/github/labn/working/bgpd/bgp_main.c:186
#12 0x55c59943a711 in sigint /home/lberger/Code/github/labn/working/bgpd/bgp_main.c:152
#13 0x55c599606311 in quagga_sigevent_process lib/sigevent.c:103
#14 0x55c59961d806 in thread_fetch lib/thread.c:1359
#15 0x55c5995e8164 in frr_run lib/libfrr.c:878
#16 0x55c599434a2e in main /home/lberger/Code/github/labn/working/bgpd/bgp_main.c:418
#17 0x7fbcbe3981c0 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x211c0)
#18 0x55c59943a029 in _start (/usr/lib/frr/bgpd+0x120029)

0x6070002109a8 is located 56 bytes inside of 72-byte region [0x607000210970,0x6070002109b8)
freed by thread T0 here:
#0 0x7fbcbf3f17b8 in __interceptor_free (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xde7b8)
#1 0x55c5994d615b in bgp_nexthop_cache_reset /home/lberger/Code/github/labn/working/bgpd/bgp_nexthop.c:85
#2 0x55c5994d615b in bgp_scan_finish /home/lberger/Code/github/labn/working/bgpd/bgp_nexthop.c:655
#3 0x55c59944742a in bgp_free /home/lberger/Code/github/labn/working/bgpd/bgpd.c:3403
#4 0x55c59944d657 in bgp_delete /home/lberger/Code/github/labn/working/bgpd/bgpd.c:3348
#5 0x55c59943a711 in bgp_exit /home/lberger/Code/github/labn/working/bgpd/bgp_main.c:186
#6 0x55c59943a711 in sigint /home/lberger/Code/github/labn/working/bgpd/bgp_main.c:152
#7 0x55c599606311 in quagga_sigevent_process lib/sigevent.c:103
#8 0x55c59961d806 in thread_fetch lib/thread.c:1359
#9 0x55c5995e8164 in frr_run lib/libfrr.c:878
#10 0x55c599434a2e in main /home/lberger/Code/github/labn/working/bgpd/bgp_main.c:418
#11 0x7fbcbe3981c0 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x211c0)

previously allocated by thread T0 here:
#0 0x7fbcbf3f1d38 in __interceptor_calloc (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xded38)
#1 0x55c5995ed858 in qcalloc lib/memory.c:69
#2 0x55c5994d429e in bnc_new /home/lberger/Code/github/labn/working/bgpd/bgp_nexthop.c:65
#3 0x55c59950730c in bgp_find_or_add_nexthop /home/lberger/Code/github/labn/working/bgpd/bgp_nht.c:194
#4 0x55c599487885 in bgp_update /home/lberger/Code/github/labn/working/bgpd/bgp_route.c:3357
#5 0x55c599489547 in bgp_nlri_parse_ip /home/lberger/Code/github/labn/working/bgpd/bgp_route.c:4277
#6 0x55c5994b89bd in bgp_update_receive /home/lberger/Code/github/labn/working/bgpd/bgp_packet.c:1558
#7 0x55c5994bbe89 in bgp_process_packet /home/lberger/Code/github/labn/working/bgpd/bgp_packet.c:2223
#8 0x55c59961ea35 in thread_call lib/thread.c:1534
#9 0x55c5995e8171 in frr_run lib/libfrr.c:879
#10 0x55c599434a2e in main /home/lberger/Code/github/labn/working/bgpd/bgp_main.c:418
#11 0x7fbcbe3981c0 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x211c0)

SUMMARY: AddressSanitizer: heap-use-after-free /home/lberger/Code/github/labn/working/bgpd/bgp_nht.c:772 in path_nh_map
Shadow bytes around the buggy address:
0x0c0e8003a0e0: fd fd fd fa fa fa fa fa fd fd fd fd fd fd fd fd
0x0c0e8003a0f0: fd fa fa fa fa fa fd fd fd fd fd fd fd fd fd fd
0x0c0e8003a100: fa fa fa fa fd fd fd fd fd fd fd fd fd fa fa fa
0x0c0e8003a110: fa fa fd fd fd fd fd fd fd fd fd fa fa fa fa fa
0x0c0e8003a120: fd fd fd fd fd fd fd fd fd fa fa fa fa fa fd fd
=>0x0c0e8003a130: fd fd fd fd fd[fd]fd fa fa fa fa fa fd fd fd fd
0x0c0e8003a140: fd fd fd fd fd fa fa fa fa fa fd fd fd fd fd fd
0x0c0e8003a150: fd fd fd fa fa fa fa fa fd fd fd fd fd fd fd fd
0x0c0e8003a160: fd fa fa fa fa fa fd fd fd fd fd fd fd fd fd fa
0x0c0e8003a170: fa fa fa fa fd fd fd fd fd fd fd fd fd fa fa fa
0x0c0e8003a180: fa fa fd fd fd fd fd fd fd fd fd fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable: 00
Partially addressable: 01 02 03 04 05 06 07
Heap left redzone: fa
Freed heap region: fd
Stack left redzone: f1
Stack mid redzone: f2
Stack right redzone: f3
Stack after return: f5
Stack use after scope: f8
Global redzone: f9
Global init order: f6
Poisoned by user: f7
Container overflow: fc
Array cookie: ac
Intra object redzone: bb
ASan internal: fe
Left alloca redzone: ca
Right alloca redzone: cb
==4520==ABORTING

@louberger
Copy link
Member Author

Current version of asan errors:
r4: bgpd triggered an exception by AddressSanitizer
ERROR: AddressSanitizer: heap-use-after-free on address 0x60e00004ead0 at pc 0x55cee50f3cd7 bp 0x7ffd7dea8720 sp 0x7ffd7dea8710
READ of size 4 at 0x60e00004ead0 thread T0
#0 0x55cee50f3cd6 in route_unlock_node ../lib/table.h:238
#1 0x55cee50f3e7b in bgp_unlock_node ../bgpd/bgp_table.h:132
#2 0x55cee50f687b in bgp_info_free /home/lberger/Code/github/labn/working/bgpd/bgp_route.c:211
#3 0x55cee50f6ac8 in bgp_info_unlock /home/lberger/Code/github/labn/working/bgpd/bgp_route.c:244
#4 0x55cee50f6dc3 in bgp_info_reap /home/lberger/Code/github/labn/working/bgpd/bgp_route.c:288
#5 0x55cee510b3f7 in bgp_clear_route_table /home/lberger/Code/github/labn/working/bgpd/bgp_route.c:3918
#6 0x55cee510b5c5 in bgp_clear_route /home/lberger/Code/github/labn/working/bgpd/bgp_route.c:3966
#7 0x55cee510b75e in bgp_clear_route_all /home/lberger/Code/github/labn/working/bgpd/bgp_route.c:3984
#8 0x55cee50cf693 in bgp_fsm_change_status /home/lberger/Code/github/labn/working/bgpd/bgp_fsm.c:926
#9 0x55cee5096413 in peer_delete /home/lberger/Code/github/labn/working/bgpd/bgpd.c:2227
#10 0x55cee50a0537 in bgp_delete /home/lberger/Code/github/labn/working/bgpd/bgpd.c:3334
#11 0x55cee507d680 in bgp_exit /home/lberger/Code/github/labn/working/bgpd/bgp_main.c:185
#12 0x55cee507d56f in sigint /home/lberger/Code/github/labn/working/bgpd/bgp_main.c:151
#13 0x55cee539e8a0 in quagga_sigevent_process lib/sigevent.c:103
#14 0x55cee53cbe46 in thread_fetch lib/thread.c:1359
#15 0x55cee5369af6 in frr_run lib/libfrr.c:878
#16 0x55cee507e342 in main /home/lberger/Code/github/labn/working/bgpd/bgp_main.c:425
#17 0x7fbc031a31c0 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x211c0)
#18 0x55cee507cfc9 in _start (/usr/lib/frr/bgpd+0x207fc9)

0x60e00004ead0 is located 80 bytes inside of 160-byte region [0x60e00004ea80,0x60e00004eb20)
freed by thread T0 here:
#0 0x7fbc041fc7b8 in __interceptor_free (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xde7b8)
#1 0x55cee537309e in qfree lib/memory.c:91
#2 0x55cee5199e06 in bgp_node_destroy /home/lberger/Code/github/labn/working/bgpd/bgp_table.c:80
#3 0x55cee53b9fa8 in route_node_free lib/table.c:90
#4 0x55cee53ba1cc in route_table_free lib/table.c:126
#5 0x55cee53b9d93 in route_table_finish lib/table.c:59
#6 0x55cee5199ccb in bgp_table_unlock /home/lberger/Code/github/labn/working/bgpd/bgp_table.c:47
#7 0x55cee5199d67 in bgp_table_finish /home/lberger/Code/github/labn/working/bgpd/bgp_table.c:56
#8 0x55cee510c062 in bgp_cleanup_routes /home/lberger/Code/github/labn/working/bgpd/bgp_route.c:4109
#9 0x55cee50a058c in bgp_delete /home/lberger/Code/github/labn/working/bgpd/bgpd.c:3345
#10 0x55cee507d680 in bgp_exit /home/lberger/Code/github/labn/working/bgpd/bgp_main.c:185
#11 0x55cee507d56f in sigint /home/lberger/Code/github/labn/working/bgpd/bgp_main.c:151
#12 0x55cee539e8a0 in quagga_sigevent_process lib/sigevent.c:103
#13 0x55cee53cbe46 in thread_fetch lib/thread.c:1359
#14 0x55cee5369af6 in frr_run lib/libfrr.c:878
#15 0x55cee507e342 in main /home/lberger/Code/github/labn/working/bgpd/bgp_main.c:425
#16 0x7fbc031a31c0 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x211c0)

previously allocated by thread T0 here:
#0 0x7fbc041fcd38 in __interceptor_calloc (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xded38)
#1 0x55cee5372fa4 in qcalloc lib/memory.c:72
#2 0x55cee5199dba in bgp_node_create /home/lberger/Code/github/labn/working/bgpd/bgp_table.c:68
#3 0x55cee53b9e06 in route_node_new lib/table.c:65
#4 0x55cee53b9e24 in route_node_set lib/table.c:74
#5 0x55cee53bb8d7 in route_node_get lib/table.c:327
#6 0x55cee50f3ff3 in bgp_node_get ../bgpd/bgp_table.h:184
#7 0x55cee50f633e in bgp_afi_node_get /home/lberger/Code/github/labn/working/bgpd/bgp_route.c:127
#8 0x55cee5105cb2 in bgp_update /home/lberger/Code/github/labn/working/bgpd/bgp_route.c:2871
#9 0x55cee51853c7 in bgp_nlri_parse_vpn /home/lberger/Code/github/labn/working/bgpd/bgp_mplsvpn.c:227
#10 0x55cee515e4e6 in bgp_nlri_parse /home/lberger/Code/github/labn/working/bgpd/bgp_packet.c:302
#11 0x55cee5165550 in bgp_update_receive /home/lberger/Code/github/labn/working/bgpd/bgp_packet.c:1553
#12 0x55cee5169883 in bgp_process_packet /home/lberger/Code/github/labn/working/bgpd/bgp_packet.c:2223
#13 0x55cee53ccced in thread_call lib/thread.c:1534
#14 0x55cee5369adc in frr_run lib/libfrr.c:879
#15 0x55cee507e342 in main /home/lberger/Code/github/labn/working/bgpd/bgp_main.c:425
#16 0x7fbc031a31c0 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x211c0)

SUMMARY: AddressSanitizer: heap-use-after-free ../lib/table.h:238 in route_unlock_node
Shadow bytes around the buggy address:
0x0c1c80001d00: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c1c80001d10: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd
0x0c1c80001d20: fd fd fd fd fd fd fd fd fd fd fd fd fa fa fa fa
0x0c1c80001d30: fa fa fa fa fd fd fd fd fd fd fd fd fd fd fd fd
0x0c1c80001d40: fd fd fd fd fd fd fd fd fa fa fa fa fa fa fa fa
=>0x0c1c80001d50: fd fd fd fd fd fd fd fd fd fd[fd]fd fd fd fd fd
0x0c1c80001d60: fd fd fd fd fa fa fa fa fa fa fa fa fd fd fd fd
0x0c1c80001d70: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c1c80001d80: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd
0x0c1c80001d90: fd fd fd fd fd fd fd fd fd fd fd fd fa fa fa fa
0x0c1c80001da0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable: 00
Partially addressable: 01 02 03 04 05 06 07
Heap left redzone: fa
Freed heap region: fd
Stack left redzone: f1
Stack mid redzone: f2
Stack right redzone: f3
Stack after return: f5
Stack use after scope: f8
Global redzone: f9
Global init order: f6
Poisoned by user: f7
Container overflow: fc
Array cookie: ac
Intra object redzone: bb
ASan internal: fe
Left alloca redzone: ca
Right alloca redzone: cb

@donaldsharp
Copy link
Member

#2335 might fix this

@ton31337
Copy link
Member

Probably fixed as Donald says. If not, let's reopen on demand.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@ton31337 @louberger @qlyoung @donaldsharp