new Data plane code crashes in CI

[donaldsharp]

AssertionError: r4: zebra crashed. Core file found - Backtrace follows: [New LWP 8651] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Core was generated by /usr/lib/frr/zebra'. Program terminated with signal SIGABRT, Aborted. #0 0x00007f471eac3428 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54 #0 0x00007f471eac3428 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54 #1 0x00007f471eac502a in __GI_abort () at abort.c:89 #2 0x00007f471f4dcbda in core_handler (signo=11, siginfo=0x7ffc00bcadb0, context=0x7ffc00bcac80) at lib/sigevent.c:249 #3 <signal handler called> #4 dplane_ctx_route_init (re=0x55f812fa4ae0, rn=0x55f812fa5e20, op=DPLANE_OP_ROUTE_DELETE, ctx=0x55f812fa4550) at zebra/zebra_dplane.c:602 #5 dplane_route_update_internal (rn=0x55f812fa5e20, re=0x55f812fa4ae0, old_re=old_re@entry=0x0, op=op@entry=DPLANE_OP_ROUTE_DELETE) at zebra/zebra_dplane.c:719 #6 0x000055f811bac755 in dplane_route_delete (rn=rn@entry=0x55f812fa5e20, re=re@entry=0x55f812fa4ae0) at zebra/zebra_dplane.c:813 #7 0x000055f811bb8c9a in rib_uninstall_kernel (rn=rn@entry=0x55f812fa5e20, re=0x55f812fa4ae0) at zebra/zebra_rib.c:1198 #8 0x000055f811bbb953 in rib_close_table (table=<optimized out>) at zebra/zebra_rib.c:3250 #9 0x000055f811bbbef1 in zebra_router_free_table (zrt=0x55f812cbb170) at zebra/zebra_router.c:142 #10 zebra_router_terminate () at zebra/zebra_router.c:156 #11 0x000055f811ba03a4 in zebra_finalize (dummy=<optimized out>) at zebra/main.c:205 #12 0x00007f471f4e9b10 in thread_call (thread=thread@entry=0x7ffc00bcb550) at lib/thread.c:1595 #13 0x00007f471f4c41c8 in frr_run (master=0x55f812c145d0) at lib/libfrr.c:947 #14 0x000055f811b92de6 in main (argc=1, argv=0x7ffc00bcb938) at zebra/main.c:470 E AssertionError: r4: zebra crashed. Core file found - Backtrace follows: [New LWP 8651] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Core was generated by /usr/lib/frr/zebra'.
Program terminated with signal SIGABRT, Aborted.
#0 0x00007f471eac3428 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
#0 0x00007f471eac3428 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
#1 0x00007f471eac502a in __GI_abort () at abort.c:89
#2 0x00007f471f4dcbda in core_handler (signo=11, siginfo=0x7ffc00bcadb0, context=0x7ffc00bcac80) at lib/sigevent.c:249
#3
#4 dplane_ctx_route_init (re=0x55f812fa4ae0, rn=0x55f812fa5e20, op=DPLANE_OP_ROUTE_DELETE, ctx=0x55f812fa4550) at zebra/zebra_dplane.c:602
#5 dplane_route_update_internal (rn=0x55f812fa5e20, re=0x55f812fa4ae0, old_re=old_re@entry=0x0, op=op@entry=DPLANE_OP_ROUTE_DELETE) at zebra/zebra_dplane.c:719
#6 0x000055f811bac755 in dplane_route_delete (rn=rn@entry=0x55f812fa5e20, re=re@entry=0x55f812fa4ae0) at zebra/zebra_dplane.c:813
#7 0x000055f811bb8c9a in rib_uninstall_kernel (rn=rn@entry=0x55f812fa5e20, re=0x55f812fa4ae0) at zebra/zebra_rib.c:1198
#8 0x000055f811bbb953 in rib_close_table (table=) at zebra/zebra_rib.c:3250
#9 0x000055f811bbbef1 in zebra_router_free_table (zrt=0x55f812cbb170) at zebra/zebra_router.c:142
#10 zebra_router_terminate () at zebra/zebra_router.c:156
#11 0x000055f811ba03a4 in zebra_finalize (dummy=) at zebra/main.c:205
#12 0x00007f471f4e9b10 in thread_call (thread=thread@entry=0x7ffc00bcb550) at lib/thread.c:1595
#13 0x00007f471f4c41c8 in frr_run (master=0x55f812c145d0) at lib/libfrr.c:947
#14 0x000055f811b92de6 in main (argc=1, argv=0x7ffc00bcb938) at zebra/main.c:470

[mjstapp]

Looks like the changes that added the 'zebra_router' component overlapped with the async dataplane changes in a way that wasn't a merge issue, but is a semantic sort of issue. The zebra shutdown path is calling 'rib_close_table()' more than once, and that wasn't safe. I've opened PR 3621 to try to make that path safe.