fix: Use reduced SRv6 encap so eBPF ingress stops dropping packets - #298
Merged
Conversation
RouteEgressAdd built its egress SRv6 route with SEG6_IPTUN_MODE_ENCAP, which the kernel always wraps in a full Segment Routing Header (RFC 8754) even for this function's single-segment case. The eBPF uSID ingress datapath introduced in #283 (galactic_usid_ingress, internal/plumbing/ebpf/prog/usid.c) replaced the old kernel seg6local ingress route model and requires the outer IPv6 header's Next Header to name the inner packet's AF directly (IPIP=4 or IPv6-in-IPv6=41) -- any Routing Header in between, including this one, is dropped as DROP_REASON_UNEXPECTED_NEXTHDR. Every packet this route encapsulated was silently black-holed on arrival at the destination node, breaking all cross-node/cross-cluster pod connectivity. Switched Mode to SEG6_IPTUN_MODE_ENCAP_RED (value 3 from the kernel UAPI's mode enum; the vendored vishvananda/netlink has never picked up this constant, but SEG6Encap.Mode is a plain int forwarded to the kernel with no validation, so no library patch is needed). For a single segment, reduced mode omits the SRH entirely and sets the outer Next Header directly to the inner packet's protocol -- confirmed empirically against this repo's containerlab lab, for both IPv4 and IPv6 inner payloads, before and after this change. Fixes #297 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
scotwells
previously approved these changes
Aug 7, 2026
scotwells
left a comment
scotwells
left a comment
Contributor
There was a problem hiding this comment.
What kind of observability do we have around galactic where we could detect this? Assume we have metrics around packets dropped?
…ot tests Two unrelated pre-existing CI failures surfaced on this PR, the first one since #283 (feat(ebpf): usid TC-BPF program) actually ran the "Build"/"Unit Tests (root)" jobs against the merged result of that PR -- neither job's checks ever recorded against #283's own commits, only against its "Publish Docker Image" runs, so both gaps below shipped to main unnoticed: 1. internal/plumbing/ebpf/prog/usid_bpfeb.o and usid_bpfel.o were stale relative to usid.c under the Build job's pinned clang-18/ llvm-18 toolchain (task build's build:ebpf step silently skips regeneration whenever clang isn't installed, per its own comment -- whatever environment last committed these apparently didn't have it, or had a different clang version, either of which produces different BPF object bytes for the same source per doc.go's own BPF2GO_CC rationale). Regenerated both via the exact pinned toolchain (clang-18, in an ubuntu:24.04 container matching the runner) so they match what the Build job's drift check expects; no change to usid.c or the generated .go bindings. 2. internal/cni's bgp_ebpf_test.go creates a real netlink.Vrf link (vrf.Add), which fails with "operation not supported" on the Actions runner: the kernel doesn't autoload the vrf module via request_module() from inside the netlink path there, and nothing in the Unit Tests (root) job loaded it explicitly (scripts/ci.sh's e2etest case already does `sudo modprobe vrf` for the same underlying reason -- this job just never had the equivalent). Added the same modprobe step here. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The previous commit's `sudo modprobe vrf` still failed on the Actions runner: "Module vrf not found in directory /lib/modules/<kernel>" -- the module isn't built into this runner's kernel or preinstalled, it lives in linux-modules-extra-<kernel-version>, which needs an explicit apt install first. Pinned to `uname -r` rather than the unversioned meta-package for the same reason scripts/ci.sh's e2etest case (which already does this exact install) pins it: an unversioned package can resolve to a newer kernel's modules than the one actually running. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
scotwells
approved these changes
Aug 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Pods in different clusters couldn't reach each other over the SRv6 data plane — every packet arrived at the destination node and was silently dropped, with no error visible anywhere in the control plane.
The egress route installer was building its SRv6 route with a non-reduced encap mode, which the kernel always wraps in a full Segment Routing Header. The eBPF ingress program that replaced the old kernel
seg6localroute model in #283 requires no such header for the single-segment case this installer always produces, and drops anything else. This switches the egress route to reduced encap, which the kernel renders as a bare encapsulation with no extra header — matching what the ingress side expects. Confirmed on the wire, for both IPv4 and IPv6 inner payloads, that this produces exactly the format the ingress side requires, and that cross-cluster ping now succeeds end-to-end.Test plan
go build ./...and full unit test suite passgalactic-routerwith the fix, all 6 cross-cluster pod-to-pod ping pairs went from 100% to 0% packet lossFixes #297