FRR crash using macvlan interface in a lxc/lxd/incus container #15370
Comments
Linux kernel can leave a macvlan interface attached to a removed link-interface. This situation is buggy. However, it must not result in a zebra crash. > 6 0x0000559d77a329d3 in zebra_vxlan_macvlan_up (ifp=0x559d798b8e00) at /root/frr/zebra/zebra_vxlan.c:4676 > 4676 link_zif = link_ifp->info; > (gdb) list > 4671 struct interface *link_ifp, *link_if; > 4672 > 4673 zif = ifp->info; > 4674 assert(zif); > 4675 link_ifp = zif->link; > 4676 link_zif = link_ifp->info; > 4677 assert(link_zif); > 4678 > (gdb) p zif->link > $2 = (struct interface *) 0x0 > (gdb) p zif->link_ifindex > $3 = 15 Fix the crash by returning when the macvlan link-interface is not found. No need to go further because the macvlan interface is not operational. Link: FRRouting#15370 Signed-off-by: Louis Scalbert <louis.scalbert@6wind.com>
|
Could you give us the "ip link show" output for all the namespaces ? |
A macvlan interface can have its underlying link-interface in another namespace (aka. netns). However, by default, zebra does not know the interface from the other namespaces. It results in a crash the pointer to the link interface is NULL. > 6 0x0000559d77a329d3 in zebra_vxlan_macvlan_up (ifp=0x559d798b8e00) at /root/frr/zebra/zebra_vxlan.c:4676 > 4676 link_zif = link_ifp->info; > (gdb) list > 4671 struct interface *link_ifp, *link_if; > 4672 > 4673 zif = ifp->info; > 4674 assert(zif); > 4675 link_ifp = zif->link; > 4676 link_zif = link_ifp->info; > 4677 assert(link_zif); > 4678 > (gdb) p zif->link > $2 = (struct interface *) 0x0 > (gdb) p zif->link_ifindex > $3 = 15 Fix the crash by returning when the macvlan link-interface is in another namespace. No need to go further because any vxlan under the macvlan interface would not be accessible by zebra. Link: FRRouting#15370 Signed-off-by: Louis Scalbert <louis.scalbert@6wind.com>
|
@joolli please test the fix. It does not crash anymore when the macvlan link-interface is in another namespace |
|
@louis-6wind I want to but I have barely basic knowledge of git... Do you know of instructions on how I would get your patch? |
|
You can do |
|
I finally managed to test. I won't take as long next time. |
|
Please provide us the full backtrace. Redo the crash. |
A macvlan interface can have its underlying link-interface in another namespace (aka. netns). However, by default, zebra does not know the interface from the other namespaces. It results in a crash the pointer to the link interface is NULL. > 6 0x0000559d77a329d3 in zebra_vxlan_macvlan_up (ifp=0x559d798b8e00) at /root/frr/zebra/zebra_vxlan.c:4676 > 4676 link_zif = link_ifp->info; > (gdb) list > 4671 struct interface *link_ifp, *link_if; > 4672 > 4673 zif = ifp->info; > 4674 assert(zif); > 4675 link_ifp = zif->link; > 4676 link_zif = link_ifp->info; > 4677 assert(link_zif); > 4678 > (gdb) p zif->link > $2 = (struct interface *) 0x0 > (gdb) p zif->link_ifindex > $3 = 15 Fix the crash by returning when the macvlan link-interface is in another namespace. No need to go further because any vxlan under the macvlan interface would not be accessible by zebra. Link: #15370 Signed-off-by: Louis Scalbert <louis.scalbert@6wind.com> (cherry picked from commit 44e6e38)
A macvlan interface can have its underlying link-interface in another namespace (aka. netns). However, by default, zebra does not know the interface from the other namespaces. It results in a crash the pointer to the link interface is NULL. > 6 0x0000559d77a329d3 in zebra_vxlan_macvlan_up (ifp=0x559d798b8e00) at /root/frr/zebra/zebra_vxlan.c:4676 > 4676 link_zif = link_ifp->info; > (gdb) list > 4671 struct interface *link_ifp, *link_if; > 4672 > 4673 zif = ifp->info; > 4674 assert(zif); > 4675 link_ifp = zif->link; > 4676 link_zif = link_ifp->info; > 4677 assert(link_zif); > 4678 > (gdb) p zif->link > $2 = (struct interface *) 0x0 > (gdb) p zif->link_ifindex > $3 = 15 Fix the crash by returning when the macvlan link-interface is in another namespace. No need to go further because any vxlan under the macvlan interface would not be accessible by zebra. Link: #15370 Signed-off-by: Louis Scalbert <louis.scalbert@6wind.com> (cherry picked from commit 44e6e38)
A macvlan interface can have its underlying link-interface in another namespace (aka. netns). However, by default, zebra does not know the interface from the other namespaces. It results in a crash the pointer to the link interface is NULL. > 6 0x0000559d77a329d3 in zebra_vxlan_macvlan_up (ifp=0x559d798b8e00) at /root/frr/zebra/zebra_vxlan.c:4676 > 4676 link_zif = link_ifp->info; > (gdb) list > 4671 struct interface *link_ifp, *link_if; > 4672 > 4673 zif = ifp->info; > 4674 assert(zif); > 4675 link_ifp = zif->link; > 4676 link_zif = link_ifp->info; > 4677 assert(link_zif); > 4678 > (gdb) p zif->link > $2 = (struct interface *) 0x0 > (gdb) p zif->link_ifindex > $3 = 15 Fix the crash by returning when the macvlan link-interface is in another namespace. No need to go further because any vxlan under the macvlan interface would not be accessible by zebra. Link: #15370 Signed-off-by: Louis Scalbert <louis.scalbert@6wind.com> (cherry picked from commit 44e6e38)
A macvlan interface can have its underlying link-interface in another namespace (aka. netns). However, by default, zebra does not know the interface from the other namespaces. It results in a crash the pointer to the link interface is NULL. > 6 0x0000559d77a329d3 in zebra_vxlan_macvlan_up (ifp=0x559d798b8e00) at /root/frr/zebra/zebra_vxlan.c:4676 > 4676 link_zif = link_ifp->info; > (gdb) list > 4671 struct interface *link_ifp, *link_if; > 4672 > 4673 zif = ifp->info; > 4674 assert(zif); > 4675 link_ifp = zif->link; > 4676 link_zif = link_ifp->info; > 4677 assert(link_zif); > 4678 > (gdb) p zif->link > $2 = (struct interface *) 0x0 > (gdb) p zif->link_ifindex > $3 = 15 Fix the crash by returning when the macvlan link-interface is in another namespace. No need to go further because any vxlan under the macvlan interface would not be accessible by zebra. Link: #15370 Signed-off-by: Louis Scalbert <louis.scalbert@6wind.com> (cherry picked from commit 44e6e38)
|
I've been unable to provide a backtrace. The crash does not happen as consistently now (at least the debug output doesn't show) but FRR restarts every time. |
|
You should modify the sysctl on the host on which the dockers are running on |
|
I modified /proc/sys/kernel/core_pattern on the host. I'm using LXD/incus, but it should be the same. |
|
Finally got it: |
|
|
|
The coredump resides on the host and I executed coredumpctl there. I'll see if I can get it to the container |
|
I couldn't get the coredump to the container on first glance, I installed the patched version onto the host. Hope that suffices: (gdb) bt |
|
then |
|
(gdb) frame 5 |
|
not the same backtrace |
|
(gdb) bt |
|
Can you do in |
|
|
and |
|
|
p zif->link |
|
For some reason, link_nsid is not set. Try this patch instead 7861f4f |
|
Is it a part of the |
|
nope but you can apply the patch manually |
|
Compiled with the patch applied and I do not get a crash now when the macvlan interface comes up |
|
fixed in #15010 |
Hi, I have a BGP-unnumbered setup going to two juniper QFX5120. That is working okay except, sometimes peering takes a long time to form (don't know if that's related). FRR is running in a LXD/incus container with two macvlan interfaces going to each switch. Those macvlan interfaces are configured with a vlan tag in the profile for the container. This container is running as a router-reflector and whenever one of the switches are restarted, zebra crashes and FRR restarts, causing an outage.
This initially happened on FRR 8.4.2, which I got from Debian stable
Then I installed FRR 9.1 from frr-stable, from the FRR repo
The problem persisted. Both versions on kernel 6.1.0-12-amd64.
The system is Debian 12.1
To Reproduce
Possibly the same issue:
#13523
Expected behavior
FRR to stay up and keep other bgp-unnumbered peer sessions going.
Versions
The text was updated successfully, but these errors were encountered: