ci: run real-agent datapath e2e nightly + on relevant PRs - #36
Closed
syscod3 wants to merge 19 commits into
Closed
Conversation
capability.Check now opens /dev/kvm (O_RDWR) instead of stat, catching permission-blocked-but-existing devices (#34). e2e_suite_test.go conditionally drops the no-agent nodeSelector and sets IMP_SCALE_TO_ZERO=true when IMP_E2E_REAL_AGENT=true, enabling a real Firecracker agent in the suite. scaletozero_datapath_test.go un-pends the wake-on-traffic datapath spec: second always-on VM execs a ping into the suspended target's overlay IP via the agent's vsock guest-exec API, asserting Suspended -> Resuming -> Running. New CI job "E2E Datapath (Kind, real agent)" (workflow_dispatch only): Kind cluster with /dev/kvm passed through to the node, Firecracker binary + public quickstart kernel staged via docker cp (no imp-specific guest kernel exists yet), runs the datapath-labeled spec. UNVALIDATED end to end until this CI job actually runs green.
Helm --set types agent.extraEnv[0].value=true as bool via server-side
apply; DaemonSet env values must be strings. Found by the first real
CI run of the datapath e2e job (helm install imp failed:
"expected string, got &value.valueUnstructured{Value:true}").
First real run got the pinger ImpVM stuck at phase=Pending for the full 5m timeout with no visibility into why (operator/agent logs, events). Cluster is deleted immediately after the job, so add a failure-only dump step before teardown.
impvm_scheduler.go's schedule() filters candidate nodes by node label
imp/enabled=true before anything else runs. The datapath test never set
it (nor removed the Kind control-plane taint), so both VMs sat
Unschedulable ("No eligible node with available capacity") for the
entire 5m Eventually window. The existing "Scheduling filter" smoke
test already does this same node prep; datapath test now mirrors it.
Both pinger and target VMs reached Starting then their Firecracker
process exited ("VM process exited and was marked Failed") on the first
real-agent run, but the CI-level diagnostics step ran after go test
already returned, by which point Ginkgo's own AfterSuite had already
helm-uninstalled the chart and killed the agent/operator pods — no logs
captured. Dump agent+operator logs from inside AfterSuite instead, before
the uninstall, whenever IMP_E2E_REAL_AGENT=true.
Ginkgo's default reporter only prints a suite node's GinkgoWriter buffer when that node fails. AfterSuite succeeded, so the agent/operator log dump from the previous commit never showed up in CI output even though it ran. fmt.Println bypasses that suppression.
Agent logs (now visible after the previous fix) show the actual root cause of every prior failure: ghcr.io/syscode-labs/test:latest is a placeholder used across e2e fixtures that was never actually pulled before, because every prior e2e run disabled the agent. With a real agent it fails outright: "DENIED: requested access to the resource is denied". Switch to public docker.io/library/busybox:latest — imp synthesizes /sbin/init from the image's CMD/ENTRYPOINT itself (internal/agent/rootfs/init.go), so any real image with a CMD works; busybox's default CMD (sh) blocks on stdin, keeping PID1 alive, and it ships a ping applet the pinger VM needs anyway.
Agent logs from the real-agent e2e run show the actual blocker: "start machine: failed to stat kernel image path, \"/var/lib/imp/vmlinux\": stat /var/lib/imp/vmlinux: no such file or directory". FC_KERNEL was always set as an env var, but unlike FC_BIN (firecracker-bin hostPath mount, fixed for #33) there was never a matching hostPath volume for the kernel — the path only ever existed on the node, never inside the agent container. Every real deployment would hit this; it just never surfaced because the agent has never run for real until this e2e run. Mirrors the firecracker-bin mount exactly. Chart unit tests pass (`helm unittest charts/imp`, 51/51); added a case for the new mount.
busybox's default CMD is a bare "sh" — with no interactive console
attached it hits EOF on stdin almost immediately and exits, so the
guest VM finished (ephemeral lifecycle default) within seconds of
boot: agent logs show finishFailed/finishSucceeded firing right after
Running, clearing the driver's process-map entry, so the later idle
suspend's Snapshot() found nothing ("VM ... is not running on this
node") and retried in a loop until timeout. nginx:alpine's default CMD
(nginx -g "daemon off;") blocks forever without needing stdin, and its
Alpine base still ships busybox ping for the guest-exec step.
Latest run: idle detection correctly transitioned target to Suspending, but Snapshot() immediately failed with "VM ... is not running on this node" and never recovered. Root cause: the imp-agent pod's container actually restarted right at that moment (kubectl events show a second Created/Started for the same pod, timestamped exactly when the errors begin), wiping the driver's in-memory procs map — a real reliability gap, not a test fixture issue. kubectl logs (current container) can't show why, since the crash was in the container that got replaced. Add pod status (RESTARTS column) + --previous logs to the AfterSuite dump so the next run captures the actual crash/exit reason.
Found the actual restart cause: previous-container agent logs end abruptly mid-stream with no panic/fatal/shutdown line — the signature of an external SIGKILL, not a Go crash. Firecracker runs as a child process of the agent, so guest RAM counts against the agent container's own cgroup. Chart default agent.resources.limits.memory is 128Mi, assuming no hosted VMs; two 256MiB-class VMs alone blow past that, so kubelet OOM-kills the container almost immediately after a real microVM boots, wiping the driver's in-memory state. Override to 1Gi for the real-agent e2e path. (The 128Mi chart default is itself a real footgun for any actual Firecracker deployment — worth its own follow-up, not fixed here to keep this change scoped.)
Suspend/resume itself now works end to end (OOM fix held) — the run got past suspend and failed on the next step: vmNodeName() returned empty. status.NodeName exists on the ImpVMStatus type with a doc comment claiming it's "the node where the VM is running", but nothing in the codebase actually writes it — only spec.nodeName is set, by the scheduler (impvm_controller.go). My own test bug, not a product bug.
Diagnosed the actual open research question: the wake ping never resolves because the target's TAP is torn down on suspend (handleSuspending frees memory; only the VTEP is kept, for cross-node routing). The AF_PACKET wake hook is also bound to ETH_P_IP only, not ARP — so even if it were listening for ARP it wouldn't see it. A same-node sender with no prior ARP entry can never emit the IP frame in the first place; there's nothing to catch. This matches the real intended use case: a client with an already-warm ARP entry (from earlier traffic) sends the IP packet directly on the next attempt, no fresh ARP round-trip needed. Ping the target once while still Running to warm the pinger's neighbor cache, matching that scenario, before letting it idle-suspend. Also refactors the raw net/http exec call into execInVM(), decoding the NDJSON response to an exit code + combined output, so both the warm-up ping (asserted) and the wake ping (logged for diagnostics) can inspect actual ping results, not just the HTTP transport status.
Failed instantly (11ms) after both VMs hit Running: status.phase reflects the host starting Firecracker, not that the guest kernel has finished booting and brought its network up. The single-shot warm-up ping fired before the guest was ready. Wrap in Eventually (up to 1m, 3s interval) instead — the original code never hit this race because it always waited through the full 15s+ idle-suspend window first, by which point the guest was long since up.
Warm-up ping failed consistently for the full 60s retry window with exit=1 and empty stdout/stderr — not a timeout pattern. Root cause: internal/guest/server.go's Exec RPC calls exec.CommandContext(command[0], ...) directly, which Go resolves via the guest-agent process's own PATH env var. The guest-agent is launched by /.imp/init as "/.imp/guest-agent &" (absolute path, no PATH lookup needed) from a PID1 shell that never had PATH set (WithEnv only writes /.imp/env when explicit env vars are configured, which this test doesn't set) — so "ping" by bare name can never resolve, hitting the *exec.Error branch that sets exitCode=1 with no output before the process ever starts. Use /bin/ping (present via alpine's busybox) instead of relying on PATH.
The scaletozero e2e run got all the way to the wake ping (suspend worked, target reached Suspended cleanly) but the target never transitioned to Resuming within the test's 2m window — and this code path had zero logging anywhere, so there was no way to tell from cluster logs whether the AF_PACKET hook saw nothing, saw traffic that didn't match, or something else. Purely additive, no logic change: - afpacketSource.Run: logs socket open/failure, then a throttled liveness line (first frame + every 10s) with frame count and last dst IP, so a validation run can tell "the socket sees nothing" from "the socket sees traffic, none of it matched." - wakeRegistry.register/onDstIP: logs VM registration, and each of the three onDstIP outcomes (matched+enqueued, matched+already-signalled, matched+channel-full) at Info; a match against nothing registered stays silent by design, since unrelated overlay traffic would otherwise flood this log continuously in a busy cluster. internal/agent unit tests (scaletozero_test.go, scaletozero_reconciler_test.go) pass unchanged — no behavior touched, existing coverage still green.
Root-caused via a real-KVM CI run + Fable investigation: the socket opened with a specific protocol (ETH_P_IP) registers into the kernel's ptype_base hash, which __netif_receive_skb_core only invokes if the receiving device's rx_handler returns RX_HANDLER_PASS. A bridge port's rx_handler (br_handle_frame) returns RX_HANDLER_CONSUMED for every switched/flooded frame -- i.e. every ordinary VM-to-VM frame on an impbr-* bridge -- so ptype_base delivery never happens for exactly the traffic this hook exists to see. This exactly matched the evidence: the diagnostic run captured thousands of frames on lo/the pod's own CNI IP (neither bridged, no rx_handler) but zero frames from imp's own overlay subnet, and zero wake matches, across the whole run. Fix: bind with ETH_P_ALL (registers into ptype_all, invoked before rx_handler -- same mechanism tcpdump uses to see switched traffic) and filter to IPv4 in userspace via the EtherType field instead of at socket() time.
Root-caused via Fable investigation, confirmed against real CI job logs (job 90608981396): "delete FDB entry <mac> ...: address family not supported by protocol" is EAFNOSUPPORT, hit deterministically every run. When the VXLAN interface attaches to the bridge (attachToBridge), Linux auto-creates a "self permanent" FDB entry for the port's own MAC with no tunnel destination. SyncFDB's stale-entry sweep isn't caught by the existing all-zeros-MAC skip, tries to delete it anyway, and NeighDel serializes a 0-length NDA_DST from the nil IP, which the kernel's VXLAN driver rejects. Not just log noise: the delete loop runs before the add loop and returns on first error, so SyncFDB could never reach NeighAdd once the interface was bridged -- meaning real VTEP entries plausibly never got added either. Not a Kind/nested-container artifact: NeighAdd, NeighList, and bridge attach all succeed in the same environment: the failure is data-shape driven (empty attribute), not a permission or capability gap, and would reproduce identically on Talos nodes. Fix: skip entries with no resolvable destination IP in the delete sweep, alongside the existing all-zeros-broadcast-MAC skip.
Was manual-dispatch-only (never ran automatically). Splits it into its own workflow file matching the existing nightly-e2e.yml / nightly-e2e-cilium-heavy.yml convention: schedule (04:00 UTC daily, staggered after the 02:00/03:00 nightly jobs), plus a pull_request paths filter so PRs touching the fragile suspend/resume/wake code trigger it automatically. E2E Smoke (Kind) in ci.yml is unchanged and stays the fast always-on push/PR gate. Adds a Telegram notify-on-failure step (reusing the org's existing bot pattern from oci-free-tier-monitor) for schedule/workflow_dispatch runs -- skips cleanly until TELEGRAM_BOT_TOKEN/TELEGRAM_CHAT_ID repo secrets are added (not yet configured on this repo).
Contributor
Author
|
Superseded by a rebuilt branch to fix a spurious merge conflict caused by the old branch forking before PR #35 merged. See replacement PR. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The real-agent datapath e2e job (validated end-to-end in #35) was manual-dispatch-only -- nothing runs it unless someone remembers to. This moves it to a routine cadence so a regression in the wake-on-traffic/suspend-resume path is caught automatically, without slowing down the fast push/PR gate everyone hits on every commit.
What changed
Test plan