Skip to content

agent config e2e: advertise Kind control-plane VM-bridge node IP#468

Merged
plombardi89 merged 1 commit into
mainfrom
fix/agent-config-e2e-kindnet-node-ip
Jul 13, 2026
Merged

agent config e2e: advertise Kind control-plane VM-bridge node IP#468
plombardi89 merged 1 commit into
mainfrom
fix/agent-config-e2e-kindnet-node-ip

Conversation

@plombardi89

Copy link
Copy Markdown
Collaborator

Summary

Fixes the agent config e2e flake in #467 where worker nodes stay NetworkPluginNotReady and the job times out after 720s.

Kindnet on the QEMU worker nodes installs the control-plane pod CIDR route via the control-plane Node's InternalIP. By default that is the Kind container's Docker-network address (for example 172.18.0.2), which is not on-link from the VM bridge subnet the workers live on. The ip route add then fails:

Adding route ... Dst: 10.244.0.0/24 Gw: 172.18.0.2
Failed to reconcile routes, retrying after error: network is unreachable
panic: Maximum retries reconciling node routes: network is unreachable

Containerd never gets a CNI config and the worker never becomes Ready.

Change

Add a configure-kind-node-ip step that reconfigures the Kind control-plane kubelet to advertise its VM-bridge address (${VM_SUBNET}.2, already assigned to eth-e2e) as the Node InternalIP, before any worker joins. That makes the pod-CIDR next hop on-link for every worker, so route reconciliation succeeds.

  • hack/agent/e2e-kind/e2e.py: new idempotent configure-kind-node-ip command (rewrites KUBELET_KUBEADM_ARGS, restarts kubelet, waits for the Node to advertise the new IP).
  • .github/actions/agent-e2e-kind-control-plane/action.yaml: run it immediately after the Kind container is attached to the VM bridge.
  • hack/agent/e2e-kind/run-local.sh: same call for local runs.

CONTROL_PLANE_ENDPOINT is intentionally left on the Docker IP (covered by the API cert SANs); only the Node's advertised InternalIP (a routing next hop, not a TLS endpoint) changes.

Verification

  • Static: python3 -m py_compile, AST/command-registration checks, bash -n, PyYAML parse, CI step-ordering assertion.
  • Isolated repro (network namespace): ip route add 10.244.0.0/24 via <docker-ip> fails; via <bridge-ip> succeeds.
  • Real Kind cluster end-to-end: ran the actual configure-kind-node-ip command; the control-plane Node advertised 192.168.110.2 within 3s; a worker netns on the VM subnet reached that IP (ping) and installed the pod-CIDR route via it, while the old Docker-IP route still failed.

Notes

The existing crash-looping-Kindnet-pod deletion workaround in e2e.py is left in place as a safety net; this change removes the cause.

Kindnet on QEMU worker nodes installs the control-plane pod CIDR route via
the control-plane Node InternalIP. By default that is the Kind container's
Docker-network address (e.g. 172.18.0.2), which is not on-link from the VM
bridge subnet the workers live on. The route add then fails with
'network is unreachable', kindnet panics reconciling node routes, and the
worker stays NetworkPluginNotReady until the 720s wait times out.

Add a configure-kind-node-ip step that reconfigures the control-plane
kubelet to advertise its VM-bridge address (VM_SUBNET.2, assigned to eth-e2e)
before any worker joins, making the next hop on-link. Wire it into the CI
composite action (after the bridge attach) and run-local.sh. Leaves kindnet's
CONTROL_PLANE_ENDPOINT on the Docker IP, which the API cert SANs cover.

Fixes #467
@plombardi89
plombardi89 requested a review from a team July 13, 2026 21:13
@plombardi89
plombardi89 added this pull request to the merge queue Jul 13, 2026
Merged via the queue into main with commit a6b54b7 Jul 13, 2026
25 checks passed
@plombardi89
plombardi89 deleted the fix/agent-config-e2e-kindnet-node-ip branch July 13, 2026 21:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants