evpn-linux: ADR-0059 slice 3b — wire FDB-NHG onto the apply path

[lance0]

Summary

First slice in the ADR-0059 chain with operational behavior change. Multi-homed Type 2 routes now program FDB nexthop groups via NDA_NH_ID in the kernel; traffic to a multi-homed MAC fans out across every observed alias VTEP on the encap path. Slice 4 (M40 manual containerlab smoke against FRR) closes out the validation story.

This PR is the wiring follow-up to slice 3a (#87) which shipped the foundation infrastructure with zero behavior change. The user-supplied research (member-order canonicalization, atomic replace semantics, CVE test scope, bridge fdb show regex, ip nexthop show format, Docker capabilities, per-VTEP NH key, RTNLGRP_NEIGH coverage gaps) is folded into the implementation per-section.

What's in this PR

• compute_diff Pass 1b (crates/evpn-linux/src/diff.rs) — emits InstallFdbNhg / UpdateFdbNhgMembers / RemoveFdbNhg based on RemoteMacEntry::alias_group_key + last_applied + GroupOwnedMap state. IPv6 alias members fall back to single-dst AddRemoteFdb with a tracing::warn! per (VNI, MAC) — slice 2's NexthopSocket rejects v6 gateways, so this is graceful degradation until the v6 fixture follow-up. 6 transition tests cover the owned↔desired matrix: single-dst → FDB-NHG, FDB-NHG → single-dst, group-key drift, N→1 drain, NotReady drain, IPv6 fallback.

• NexthopOps impls: LinuxDataplane delegates to slice 2's NexthopSocket + slice 3a's linux::fdb_nhg with the CVE-2025-39851 guard inline; InMemoryDataplane records add_nexthop_member / add_nexthop_group / del_nexthop / install_fdb_nhg_row / remove_fdb_nhg_row / dump_owned_nexthops calls for unit-testing the coordinator without netlink.

• Reconcile actor coordinator (crates/evpn-linux/src/reconcile.rs::apply_nhg_op) — intercepts the three FDB-NHG op variants before Dataplane::apply, calls NexthopOps methods in ADR-0059 §5 invariant 1+2 order:

  • Install: per-VTEP members → group (create or NLM_F_REPLACE) → FDB row.
  • Update: add new members → group REPLACE (atomic alias-set swap) → GC removed per-VTEP members whose ref-count hits zero.
  • Remove: FDB row first → if last MAC, delete group + unref per-VTEP members; release allocator slots for anything cleaned up.

• NexthopSocket::dump_owned (crates/evpn-linux/src/linux/nexthop_raw/socket.rs) — RTM_GETNEXTHOP multipart parser. Tolerates NHA_GROUP_TYPE / NHA_OP_FLAGS per research §1 (kernel emits them on dump even when we don't set them on add); filters by tag bits + NHA_FDB presence. 7 unit tests covering member-decode, group-decode, unknown-attr tolerance, tag-bit filtering, non-FDB skip, NLMSG_DONE termination, continuation.

• Startup adoption + deferred stale cleanup: first reconcile_once pass dumps tagged nexthops, reserves their IDs in NhIdAllocator (prevents NLM_F_REPLACE collisions with a prior daemon's state), and tracks them in adopted_unreferenced. After the first apply phase, anything still unreferenced by GroupOwnedMap is true-stale and gets deleted + released. Gated to one shot via adoption_done. Per research §8, this is startup-only; periodic RTM_GETNEXTHOP drift recovery (forced NHG deletion doesn't emit RTNLGRP_NEIGH) is deferred to slice 3.5.

• DataplaneOpKind variants from slice 3a are now actually emitted: InstallFdbNhg { mac }, UpdateFdbNhgMembers { esi, ethernet_tag }, RemoveFdbNhg { mac }. Each carries the discriminator needed for operator reporting; group-level update uses (esi, ethernet_tag) since there's no per-MAC identity (the vni lives on AppliedOp).

• Docker-runnable netns integration test (crates/evpn-linux/tests/netns_fdb_nhg.rs): round-trip install + verification via ip nexthop show (member format id %u via %s fdb, group format id %u group %u/%u fdb per research §5) + bridge fdb show (regex (^|\\s)nhid\\s+${id}(\\s|$) per research §4) + dump_owned_nexthops round-trip + idempotent del + CVE-2025-39851 guard negative path (per research §3: don't try to crash the kernel; assert learning=on → install returns InvalidArgument referencing the CVE, no FDB row programmed).

  Harness extended at crates/evpn-linux/tests/docker/run-netns-tests.sh with fdb_nhg / fdb_nhg_roundtrip / fdb_nhg_cve selectors. Existing cap set (NET_ADMIN + SYS_ADMIN + apparmor=unconfined) is correct per research §6.

Design notes / deviations from the approved plan

• Adoption happens in the first reconcile_once pass rather than at LinuxDataplane::connect() time. Keeps allocator + refcount state in ActorState (per ADR §7 boundary) without changing connect()'s signature. Net behavior identical — adoption completes before any InstallFdbNhg is emitted in compute_diff Pass 1b.

• Per-VTEP NH key is IpAddr only (not (VNI, IpAddr)) per research §7 (FRR's zebra_evpn_l2_nh_find keys by VTEP IP). Group key stays VNI-scoped because share_l2_nhg defaults off.

• Member set canonicalization: encoder produces sorted+deduped (aliasing::group_members already does this); GroupOwnedMap stores in BTreeSet<IpAddr>; diff compares order-insensitively. Per research §1, kernel preserves stored order on dump but canonicalization avoids false drift.

What's not in this PR (slice 3.5 / slice 4 / future)

• M40 manual containerlab smoke against FRR — slice 4.
• Periodic RTM_GETNEXTHOP drift recovery — slice 3.5. Forced NHG deletion doesn't emit RTNLGRP_NEIGH (research §8); requires periodic dump or RTNLGRP_NEXTHOP multicast (latter explicit ADR out-of-scope).
• IPv6 alias members on FDB-NHG — slice 2.5 follow-up. Diff falls back to single-dst with warn today.
• apply_aliasing_ecmp operator off-switch — slice 3b ships enforcement-on; toggle is slice 3.5+ if operators ask.

Stats

• 13 files changed: +1999 / -57 lines (largest contributor: the reconcile coordinator + parser).
• 227 lib tests passing (+13 new on this branch: 7 dump-parser + 6 transition). All pre-existing diff / l3_diff / linux::* / reconcile tests still green.
• 21 integration / smoke buckets unchanged.

Test plan

• cargo fmt --all -- --check
• cargo clippy --workspace --all-targets -- -D warnings
• cargo test --workspace --no-fail-fast (full sweep)
• RUSTDOCFLAGS=\"-D warnings\" cargo doc --workspace --no-deps
• cargo +1.92 check --workspace --all-targets --locked (MSRV)
• CI: build matrix + interop pass on push.
• Manual netns smoke (Docker, no sudo):

  crates/evpn-linux/tests/docker/run-netns-tests.sh fdb_nhg

References

• ADR-0059 (merged docs: ADR-0059 — EVPN aliasing dataplane via FDB nexthop groups #83): design.
• Slice 1 (evpn: ADR-0059 slice 1 — RemoteMacEntry alias_group_key + projection AF guard #84): portable intent (alias_group_key).
• Slice 2 (evpn-linux: ADR-0059 slice 2 — nexthop_raw raw-netlink FDB-NHG primitive #86): nexthop_raw netlink primitive.
• Slice 3a (evpn-linux: ADR-0059 slice 3a — FDB-NHG state types + apply primitive + CVE guard #87): state types + apply primitive + CVE guard.
• CVE-2025-39851 mainline fix 6ead38147ebb.

[Copilot]

Pull request overview

Wires ADR-0059 slice 3b into the EVPN Linux dataplane apply path: multi-homed Type 2 MACs are now programmed using kernel FDB nexthop groups (NHGs) with startup adoption of tagged NHIDs and an end-to-end privileged netns integration test.

Changes:

• Extend compute_diff to emit FDB-NHG ops (InstallFdbNhg / UpdateFdbNhgMembers / RemoveFdbNhg) and add IPv6 graceful fallback.
• Add reconcile-actor coordination for FDB-NHG lifecycle (allocator + refcounts + adoption/cleanup).
• Implement NexthopOps for Linux + in-memory dataplanes and add RTM_GETNEXTHOP dump support + Docker/netns harness updates.

Reviewed changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
src/evpn_dataplane.rs	Tightens supervisor spawn generic bound to require NexthopOps.
ROADMAP.md	Updates roadmap status for ADR-0059 slice chain.
CHANGELOG.md	Adds unreleased entry describing slice 3b operational change and wiring.
crates/evpn-linux/src/diff.rs	Emits FDB-NHG ops (Pass 1b), handles transitions, adds tests.
crates/evpn-linux/src/reconcile.rs	Adds NHG coordinator apply path, startup adoption + stale cleanup, actor state.
crates/evpn-linux/src/linux/mod.rs	Adds dedicated nexthop socket and NexthopOps impl for LinuxDataplane.
crates/evpn-linux/src/linux/fdb_nhg.rs	Updates module docs to reflect slice 3b wiring status.
crates/evpn-linux/src/linux/nexthop_raw/encode.rs	Adds RTM_GETNEXTHOP dump encoder.
crates/evpn-linux/src/linux/nexthop_raw/socket.rs	Implements multipart dump parsing + dump_owned() with unit tests.
crates/evpn-linux/src/linux/nexthop_raw/uapi.rs	Adds missing UAPI consts needed for dump tolerance.
crates/evpn-linux/src/in_memory.rs	Implements NexthopOps in the in-memory dataplane for coordinator tests.
crates/evpn-linux/tests/netns_fdb_nhg.rs	Adds privileged netns integration test for FDB-NHG programming + CVE guard.
crates/evpn-linux/tests/docker/run-netns-tests.sh	Adds selectors to run the new netns test binary and subtests.

Comments suppressed due to low confidence (2)

crates/evpn-linux/src/reconcile.rs:917

• cleanup_unreferenced_adoptions releases NHID allocator slots even when del_nexthop(id) fails. If the kernel object is still present, releasing the ID can allow a later allocation to reuse it and inadvertently REPLACE/overwrite a still-live kernel nexthop. Safer approach: only release(id) after a confirmed successful delete (including idempotent ENOENT), and keep failed IDs in adopted_unreferenced so cleanup can retry on the next pass (which likely also implies not flipping adoption_done yet).

            if let Err(e) = self.dataplane.del_nexthop(id).await {
                tracing::debug!(?e, id, "adoption cleanup: del_nexthop failed (best-effort)");
            }
            self.state.nh_id_alloc.release(id);
        }

crates/evpn-linux/src/reconcile.rs:1419

• In the RemoveFdbNhg path, del_nexthop errors are ignored (let _ = ...) but the allocator slot is still released immediately afterward. If the delete failed and the kernel object remains, the allocator may later reuse the same ID and NLM_F_REPLACE could overwrite a still-live nexthop/group. Consider releasing IDs only after a confirmed successful delete (including idempotent ENOENT), and preserving enough state to retry cleanup on the next reconcile pass when deletion fails.

                RefDelta::GroupShouldDelete { id, members } => {
                    let _ = dataplane.del_nexthop(id).await;
                    state.nh_id_alloc.release(id);
                    for ip in members {
                        if let Some(vtep_id) = state.groups.record_member_unref(ip, *group_key) {
                            let _ = dataplane.del_nexthop(vtep_id).await;
                            state.nh_id_alloc.release(vtep_id);
                        }

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 4 comments.

[Copilot]

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 3 comments.

Comments suppressed due to low confidence (2)

crates/evpn-linux/src/reconcile.rs:1472

• Same issue as above: this GC loop ignores del_nexthop errors but still releases the allocator slot. Releasing on failure can lead to reusing an ID that is still present in the kernel. Prefer releasing only on successful delete (del already treats ENOENT as Ok), and keep failed deletions reserved + queued for retry.

            for ip in removed {
                if let Some(vtep_id) = state.groups.record_member_unref(ip, *group_key) {
                    let _ = dataplane.del_nexthop(vtep_id).await;
                    state.nh_id_alloc.release(vtep_id);
                }

crates/evpn-linux/src/reconcile.rs:1493

• On group teardown, del_nexthop errors are ignored but the group/member IDs are released anyway. If a delete fails (e.g., transient netlink error), the allocator can hand out an ID that still exists in the kernel. Recommend releasing IDs only after del_nexthop succeeds, and retaining enough state to retry deletion later while keeping the IDs reserved.

                RefDelta::GroupShouldDelete { id, members } => {
                    let _ = dataplane.del_nexthop(id).await;
                    state.nh_id_alloc.release(id);
                    for ip in members {
                        if let Some(vtep_id) = state.groups.record_member_unref(ip, *group_key) {
                            let _ = dataplane.del_nexthop(vtep_id).await;
                            state.nh_id_alloc.release(vtep_id);
                        }

[Copilot]

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 3 comments.

[Copilot]

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 3 comments.

Comments suppressed due to low confidence (1)

crates/evpn-linux/src/reconcile.rs:1554

• If add_nexthop_group(...).await? fails, any new per-VTEP members installed above remain present but (since record_group_member_change is not reached) are not attached to the group in GroupOwnedMap. A permanent failure would leave these members orphaned and allocator slots reserved. Consider best-effort rollback/enqueue of member IDs created in this Update op when the group REPLACE step fails.

            // Compute new member-id list and REPLACE the group.
            let member_ids: Vec<u32> = members
                .iter()
                .map(|ip| state.groups.vtep_nh(ip).expect("just installed").id)
                .collect();
            let Some(g_id) = state.groups.group(group_key).map(|g| g.id) else {
                // `compute_diff` Pass 1b only emits `UpdateFdbNhgMembers`
                // when the group exists in `GroupOwnedMap`, so hitting
                // this branch means `owned` and `groups` have drifted
                // out of sync — an internal state inconsistency.
                // Surface it as a transient error (classified `Other`
                // → retried) so it shows up in metrics rather than
                // silently clearing retry state.
                return Err(crate::error::DataplaneError::Other(format!(
                    "ADR-0059: UpdateFdbNhgMembers for {group_key:?} but group missing \
                     from GroupOwnedMap (owned/groups state drift)",
                )));
            };
            dataplane.add_nexthop_group(g_id, &member_ids).await?;
            let new_members: BTreeSet<_> = members.iter().copied().collect();

[Copilot]

Pull request overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated 4 comments.

[Copilot]

Pull request overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated 3 comments.

Comments suppressed due to low confidence (1)

crates/evpn-linux/src/in_memory.rs:351

• Similarly, remove_fdb_nhg_row only updates fdb_nhg_rows; if you make install_fdb_nhg_row update state.kernel for snapshot consistency, this should also remove/clear the corresponding KernelSnapshot FDB entry so compute_diff converges correctly in in-memory actor tests.

    async fn remove_fdb_nhg_row(
        &mut self,
        vni: EvpnInstanceId,
        mac: MacAddress,
    ) -> Result<(), DataplaneError> {
        let mut state = self.state.lock().expect("poisoned");
        // Idempotent — slice 3b coordinator may issue this on
        // already-removed rows during stale cleanup.
        state.fdb_nhg_rows.remove(&(vni, mac));
        Ok(())

[Copilot]

Pull request overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated 4 comments.

[Copilot]

Pull request overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (1)

crates/evpn-linux/src/reconcile.rs:1066

• cleanup_unreferenced_adoptions deletes stale nexthops in ascending ID order. Since member IDs (0x3000_....) sort before group IDs (0x4000_....), this will try to delete members while their (stale) group still exists, which the kernel typically rejects; that forces at least one extra reconcile pass and keeps adoption_done false even if the group delete succeeds later in the same loop. Consider deleting stale NHGs before members (using the stored KernelNexthopKind/tag bits to order), so cleanup can succeed in a single pass and avoid avoidable warnings/retries.

    async fn cleanup_unreferenced_adoptions(&mut self) -> bool {
        // Drain to avoid holding the borrow across the await.
        let stale: Vec<u32> = self.state.adopted_unreferenced.keys().copied().collect();
        let mut all_ok = true;
        for id in stale {
            // del_nexthop is idempotent on ENOENT (slice 2). Only
            // release + drop tracking on Ok — on Err, leave the slot
            // reserved + the entry in the map so the next reconcile
            // pass retries.
            match self.dataplane.del_nexthop(id).await {
                Ok(()) => {
                    self.state.nh_id_alloc.release(id);
                    self.state.adopted_unreferenced.remove(&id);
                }
                Err(e) => {
                    tracing::warn!(
                        ?e,
                        id,
                        "adoption cleanup: del_nexthop failed; leaving reserved for next pass"
                    );
                    all_ok = false;
                }
            }
        }
        all_ok