Description
bgp_srv6l3vpn_to_bgp_vrf3 topotests is frequently failing on a new setup with ASAN on Debian 13 and newer 6.18 kernel (Not sure which part here might impact it, but the test runs fine on older Debian 12 setups)
The test fails with some routes not getting installed in kernel
Version
Current frr master (git sha 961a1e802)
How to reproduce
Topotests on Debian 13 with ASAN on new kernel. Ping me if access to a testbed is needed. The test fails approx 1..3 times per 10 runs.
Expected behavior
Test passing
Actual behavior
Test fails frequently
Additional context
Spent a bit time with Claude Code debugging it and here is what Claude found. Sorry, not understanding the code enough to verify this, so someone with better understanding of this code needs to look at it:
The failure
test_rib intermittently fails (~1/10, more under CPU load) because a VPN-imported VRF route is missing from a router's zebra table. The specific missing prefix varies (e.g. r1 missing 192.168.2.0/24, or r2 missing 192.168.1.0/24 + 192.168.3.0/24) — but it's always a route a router learned over the SRv6 L3VPN and should have imported into its VRF. Once it's stuck, it never recovers, so the 15s poll times out and all 10 tests cascade-fail.
Root cause (proven with instrumented bgpd)
A startup race in leak_update_nexthop_valid() (bgpd/bgp_mplsvpn.c). When a router imports a VPN route into a VRF, this check (added Nov-2024 by commit 779077b) rejects the route if the nexthop VRF's loopback is down:
ifp = if_get_vrf_loopback(bgp_nexthop->vrf_id);
if (ifp && !if_is_up(ifp))
return false;
For VPN→VRF imports of remote routes, bgp_nexthop is the default instance, so this gates on lo. In a fresh netns lo starts down and zebra brings it up shortly after start. If the BGP routes arrive in that tiny window, the import is rejected.
Instrumentation captured the exact failing moment on the failing router:
192.168.1.0/24 loopback=lo up=0 (in VRF default)
192.168.1.0/24 EARLYRET loopback lo down (in VRF default)
The return false happens before the nexthop is registered for tracking, so the route is created invalid, never installed to zebra, and never recovers because:
- bgp_process on a VPN route doesn't re-leak to VRFs, and
- bgp_ifp_up only re-triggers vpn_leak_postchange_all() for if_is_vrf interfaces — not plain lo (deliberately, per commit c21c597: "the up/down state of the lo loopback interface does not determine the availability of the default vrf-lite").
So 779077b contradicts c21c597 for the default VRF: it reintroduced exactly the lo-gating that c21c597 had removed, but without any recovery path.
The fix (bgpd/bgp_mplsvpn.c)
Scope the loopback-down check to non-default VRFs:
if (bgp_nexthop->vrf_id != VRF_DEFAULT) {
ifp = if_get_vrf_loopback(bgp_nexthop->vrf_id);
if (ifp && !if_is_up(ifp))
return false;
}
A genuinely unreachable default-VRF nexthop is still caught by normal NHT (bgp_find_or_add_nexthop, which resolves via eth0 — the logs confirm the SID nexthop resolves via 2001::1 if 11, independent of lo). Non-default VRFs keep the check (and they do have a recovery path via bgp_ifp_up/if_is_vrf). The fix is applied in your local working tree.
Validation
- Baseline (unfixed): reproduced the failure repeatedly; instrumentation proved the up=0 → EARLYRET mechanism on the exact missing routes.
- Fixed: 12/12 pass under moderate load + 5/5 clean final runs + forced-lo-down attempts — zero failures of the lo race across ~25 runs. (Note: under pathological 5-spinner load a few runs failed on unrelated convergence timeouts — up=0 count 0 — which is a separate test-robustness matter, not this bug.)
The ASAN topotest on Debian 13 now runs a clean fixed bgpd; test configs and the test file were restored to their original state.
One caveat worth flagging: the same vrf_id == VRF_UNKNOWN early-return just above the fixed line could in principle have a similar "no recovery" property, but it didn't trigger here (the default instance always has a valid vrf_id) — It is left unchanged for now.
Checklist
Description
bgp_srv6l3vpn_to_bgp_vrf3 topotests is frequently failing on a new setup with ASAN on Debian 13 and newer 6.18 kernel (Not sure which part here might impact it, but the test runs fine on older Debian 12 setups)
The test fails with some routes not getting installed in kernel
Version
How to reproduce
Topotests on Debian 13 with ASAN on new kernel. Ping me if access to a testbed is needed. The test fails approx 1..3 times per 10 runs.
Expected behavior
Test passing
Actual behavior
Test fails frequently
Additional context
Spent a bit time with Claude Code debugging it and here is what Claude found. Sorry, not understanding the code enough to verify this, so someone with better understanding of this code needs to look at it:
The failure
test_rib intermittently fails (~1/10, more under CPU load) because a VPN-imported VRF route is missing from a router's zebra table. The specific missing prefix varies (e.g. r1 missing 192.168.2.0/24, or r2 missing 192.168.1.0/24 + 192.168.3.0/24) — but it's always a route a router learned over the SRv6 L3VPN and should have imported into its VRF. Once it's stuck, it never recovers, so the 15s poll times out and all 10 tests cascade-fail.
Root cause (proven with instrumented bgpd)
A startup race in leak_update_nexthop_valid() (bgpd/bgp_mplsvpn.c). When a router imports a VPN route into a VRF, this check (added Nov-2024 by commit 779077b) rejects the route if the nexthop VRF's loopback is down:
For VPN→VRF imports of remote routes, bgp_nexthop is the default instance, so this gates on lo. In a fresh netns lo starts down and zebra brings it up shortly after start. If the BGP routes arrive in that tiny window, the import is rejected.
Instrumentation captured the exact failing moment on the failing router:
The return false happens before the nexthop is registered for tracking, so the route is created invalid, never installed to zebra, and never recovers because:
So 779077b contradicts c21c597 for the default VRF: it reintroduced exactly the lo-gating that c21c597 had removed, but without any recovery path.
The fix (bgpd/bgp_mplsvpn.c)
Scope the loopback-down check to non-default VRFs:
A genuinely unreachable default-VRF nexthop is still caught by normal NHT (bgp_find_or_add_nexthop, which resolves via eth0 — the logs confirm the SID nexthop resolves via 2001::1 if 11, independent of lo). Non-default VRFs keep the check (and they do have a recovery path via bgp_ifp_up/if_is_vrf). The fix is applied in your local working tree.
Validation
The ASAN topotest on Debian 13 now runs a clean fixed bgpd; test configs and the test file were restored to their original state.
One caveat worth flagging: the same vrf_id == VRF_UNKNOWN early-return just above the fixed line could in principle have a similar "no recovery" property, but it didn't trigger here (the default instance always has a valid vrf_id) — It is left unchanged for now.
Checklist