-
Notifications
You must be signed in to change notification settings - Fork 1.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
FRR crash using macvlan interface in a lxc/lxd/incus container #15370
Comments
Linux kernel can leave a macvlan interface attached to a removed link-interface. This situation is buggy. However, it must not result in a zebra crash. > 6 0x0000559d77a329d3 in zebra_vxlan_macvlan_up (ifp=0x559d798b8e00) at /root/frr/zebra/zebra_vxlan.c:4676 > 4676 link_zif = link_ifp->info; > (gdb) list > 4671 struct interface *link_ifp, *link_if; > 4672 > 4673 zif = ifp->info; > 4674 assert(zif); > 4675 link_ifp = zif->link; > 4676 link_zif = link_ifp->info; > 4677 assert(link_zif); > 4678 > (gdb) p zif->link > $2 = (struct interface *) 0x0 > (gdb) p zif->link_ifindex > $3 = 15 Fix the crash by returning when the macvlan link-interface is not found. No need to go further because the macvlan interface is not operational. Link: FRRouting#15370 Signed-off-by: Louis Scalbert <louis.scalbert@6wind.com>
Could you give us the "ip link show" output for all the namespaces ? |
A macvlan interface can have its underlying link-interface in another namespace (aka. netns). However, by default, zebra does not know the interface from the other namespaces. It results in a crash the pointer to the link interface is NULL. > 6 0x0000559d77a329d3 in zebra_vxlan_macvlan_up (ifp=0x559d798b8e00) at /root/frr/zebra/zebra_vxlan.c:4676 > 4676 link_zif = link_ifp->info; > (gdb) list > 4671 struct interface *link_ifp, *link_if; > 4672 > 4673 zif = ifp->info; > 4674 assert(zif); > 4675 link_ifp = zif->link; > 4676 link_zif = link_ifp->info; > 4677 assert(link_zif); > 4678 > (gdb) p zif->link > $2 = (struct interface *) 0x0 > (gdb) p zif->link_ifindex > $3 = 15 Fix the crash by returning when the macvlan link-interface is in another namespace. No need to go further because any vxlan under the macvlan interface would not be accessible by zebra. Link: FRRouting#15370 Signed-off-by: Louis Scalbert <louis.scalbert@6wind.com>
@joolli please test the fix. It does not crash anymore when the macvlan link-interface is in another namespace |
@louis-6wind I want to but I have barely basic knowledge of git... Do you know of instructions on how I would get your patch? |
You can do
|
I finally managed to test. I won't take as long next time.
|
Please provide us the full backtrace.
Redo the crash.
|
A macvlan interface can have its underlying link-interface in another namespace (aka. netns). However, by default, zebra does not know the interface from the other namespaces. It results in a crash the pointer to the link interface is NULL. > 6 0x0000559d77a329d3 in zebra_vxlan_macvlan_up (ifp=0x559d798b8e00) at /root/frr/zebra/zebra_vxlan.c:4676 > 4676 link_zif = link_ifp->info; > (gdb) list > 4671 struct interface *link_ifp, *link_if; > 4672 > 4673 zif = ifp->info; > 4674 assert(zif); > 4675 link_ifp = zif->link; > 4676 link_zif = link_ifp->info; > 4677 assert(link_zif); > 4678 > (gdb) p zif->link > $2 = (struct interface *) 0x0 > (gdb) p zif->link_ifindex > $3 = 15 Fix the crash by returning when the macvlan link-interface is in another namespace. No need to go further because any vxlan under the macvlan interface would not be accessible by zebra. Link: #15370 Signed-off-by: Louis Scalbert <louis.scalbert@6wind.com> (cherry picked from commit 44e6e38)
A macvlan interface can have its underlying link-interface in another namespace (aka. netns). However, by default, zebra does not know the interface from the other namespaces. It results in a crash the pointer to the link interface is NULL. > 6 0x0000559d77a329d3 in zebra_vxlan_macvlan_up (ifp=0x559d798b8e00) at /root/frr/zebra/zebra_vxlan.c:4676 > 4676 link_zif = link_ifp->info; > (gdb) list > 4671 struct interface *link_ifp, *link_if; > 4672 > 4673 zif = ifp->info; > 4674 assert(zif); > 4675 link_ifp = zif->link; > 4676 link_zif = link_ifp->info; > 4677 assert(link_zif); > 4678 > (gdb) p zif->link > $2 = (struct interface *) 0x0 > (gdb) p zif->link_ifindex > $3 = 15 Fix the crash by returning when the macvlan link-interface is in another namespace. No need to go further because any vxlan under the macvlan interface would not be accessible by zebra. Link: #15370 Signed-off-by: Louis Scalbert <louis.scalbert@6wind.com> (cherry picked from commit 44e6e38)
A macvlan interface can have its underlying link-interface in another namespace (aka. netns). However, by default, zebra does not know the interface from the other namespaces. It results in a crash the pointer to the link interface is NULL. > 6 0x0000559d77a329d3 in zebra_vxlan_macvlan_up (ifp=0x559d798b8e00) at /root/frr/zebra/zebra_vxlan.c:4676 > 4676 link_zif = link_ifp->info; > (gdb) list > 4671 struct interface *link_ifp, *link_if; > 4672 > 4673 zif = ifp->info; > 4674 assert(zif); > 4675 link_ifp = zif->link; > 4676 link_zif = link_ifp->info; > 4677 assert(link_zif); > 4678 > (gdb) p zif->link > $2 = (struct interface *) 0x0 > (gdb) p zif->link_ifindex > $3 = 15 Fix the crash by returning when the macvlan link-interface is in another namespace. No need to go further because any vxlan under the macvlan interface would not be accessible by zebra. Link: #15370 Signed-off-by: Louis Scalbert <louis.scalbert@6wind.com> (cherry picked from commit 44e6e38)
A macvlan interface can have its underlying link-interface in another namespace (aka. netns). However, by default, zebra does not know the interface from the other namespaces. It results in a crash the pointer to the link interface is NULL. > 6 0x0000559d77a329d3 in zebra_vxlan_macvlan_up (ifp=0x559d798b8e00) at /root/frr/zebra/zebra_vxlan.c:4676 > 4676 link_zif = link_ifp->info; > (gdb) list > 4671 struct interface *link_ifp, *link_if; > 4672 > 4673 zif = ifp->info; > 4674 assert(zif); > 4675 link_ifp = zif->link; > 4676 link_zif = link_ifp->info; > 4677 assert(link_zif); > 4678 > (gdb) p zif->link > $2 = (struct interface *) 0x0 > (gdb) p zif->link_ifindex > $3 = 15 Fix the crash by returning when the macvlan link-interface is in another namespace. No need to go further because any vxlan under the macvlan interface would not be accessible by zebra. Link: #15370 Signed-off-by: Louis Scalbert <louis.scalbert@6wind.com> (cherry picked from commit 44e6e38)
I've been unable to provide a backtrace. The crash does not happen as consistently now (at least the debug output doesn't show) but FRR restarts every time. |
You should modify the sysctl on the host on which the dockers are running on |
I modified /proc/sys/kernel/core_pattern on the host. I'm using LXD/incus, but it should be the same. |
Finally got it: |
|
The coredump resides on the host and I executed coredumpctl there. I'll see if I can get it to the container |
I couldn't get the coredump to the container on first glance, I installed the patched version onto the host. Hope that suffices: (gdb) bt |
then
|
(gdb) frame 5 |
not the same backtrace |
(gdb) bt |
Can you do in
|
|
and
|
|
p zif->link |
For some reason, link_nsid is not set. Try this patch instead 7861f4f |
Is it a part of the |
nope but you can apply the patch manually |
Compiled with the patch applied and I do not get a crash now when the macvlan interface comes up |
fixed in #15010 |
Hi, I have a BGP-unnumbered setup going to two juniper QFX5120. That is working okay except, sometimes peering takes a long time to form (don't know if that's related). FRR is running in a LXD/incus container with two macvlan interfaces going to each switch. Those macvlan interfaces are configured with a vlan tag in the profile for the container. This container is running as a router-reflector and whenever one of the switches are restarted, zebra crashes and FRR restarts, causing an outage.
This initially happened on FRR 8.4.2, which I got from Debian stable
Then I installed FRR 9.1 from frr-stable, from the FRR repo
The problem persisted. Both versions on kernel 6.1.0-12-amd64.
The system is Debian 12.1
To Reproduce
Possibly the same issue:
#13523
Expected behavior
FRR to stay up and keep other bgp-unnumbered peer sessions going.
Versions
The text was updated successfully, but these errors were encountered: