Fix Kubernetes control-plane restart after IP rotation

[mxschmitt]

Summary

Fix restart of newly created Kubernetes nodes after their container VM gets a
new IPv4 address.

Closes #2156.

Bug before

On a clean 1.3.0 installation, with no workload or custom DNS:

container k8s create --name k8s-repro
container stop k8s-repro
container k8s start --name k8s-repro

The first restart changed 192.168.64.2 -> 192.168.64.3 and failed in the
kind entrypoint:

INFO: detected IPv4 address: 192.168.64.3
INFO: detected old IPv4 address: 192.168.64.2
error: unable to read config from "/kind/kubeadm.conf":
open /kind/kubeadm.conf: no such file or directory

There are two independent causes:

1. The plugin ran kubeadm with /etc/kubernetes/kubeadm-config.yaml, but a
   kind node expects its provider to create /kind/kubeadm.conf for restart
   reconciliation. The file was never created; kubeadm did not consume it.
2. Without controlPlaneEndpoint, kubeadm persisted the current vmnet address
   in clients such as admin.conf and kube-proxy. Preserving only the kind file
   let kind repair the API server, but kube-proxy still dialed the retired
   address and CoreDNS stayed unready.

Fix and behavior afterwards

• Write the generated configuration directly to /kind/kubeadm.conf and run
  kubeadm with that same file, matching kind's provider contract.
• Set controlPlaneEndpoint to 127.0.0.1:6443 for clients inside the current
  single-node control-plane/worker VM.

On restart, kind rewrites address-dependent advertise and certificate state to
the new VM address. Persistent in-VM clients keep using loopback, so they do not
retain a rotating vmnet address. The node still advertises its real current
address, and host kubeconfig generation remains unchanged.

This intentionally does not add sticky IP allocation, DNS setup, a proxy, or a
start-time reconciliation hook. Existing clusters created without
/kind/kubeadm.conf are not migrated by this change.

Loopback is appropriate for the plugin's current one-VM topology. Separate
workers or multiple control planes would require a stable shared DNS, VIP, or
load-balancer endpoint instead.

Testing

• Built the rebased patch on current main with make container.
• swift test --filter K8sBootstrapTests: 2/2 passed.
• TestK8sRunSerial.testRestartAfterAddressRotation passed twice against the
  rebuilt daemon, including 192.168.64.5 -> 192.168.64.7 after the test used
  an intervening container to advance the rotating allocator.
• The regression verifies /kind/kubeadm.conf, the loopback endpoint, actual
  address rotation, kind's rewritten advertiseAddress, successful
  k8s start, and a Ready node. Cleanup removes both test containers.
• A separate restarted-cluster smoke test loaded local Caddy and Alpine images,
  resolved the Service through CoreDNS, and fetched it successfully.
• Strict Swift format lint and git diff --check passed.

[mxschmitt]

Long-term design question

This PR intentionally fixes the current single-node plugin with the smallest
address-independent bootstrap change: new in-VM Kubernetes clients use
127.0.0.1:6443, while kind continues to reconcile the node's changing
advertise address.

I would welcome maintainer guidance on the longer-term abstraction before
multi-control-plane support is added. At that point, loopback would no longer
be sufficient and the plugin would need a stable shared endpoint, for example:

• a container/runtime-provided DNS identity,
• a local API load balancer with a dynamic backend set, or
• a narrowly defined endpoint-reconciliation API.

After a clean-room and ablation retest, I removed the earlier start-time
ConfigMap migration. It did not provide a valid upgrade path for stock legacy
clusters and was not needed by newly created clusters, so keeping it would
have made this fix broader than the demonstrated defect.

[mxschmitt]

@jglogan, could you take a look when you have a chance? This fixes a clean
single-node container k8s stop/start failure after expected VM IP rotation.
The patch aligns bootstrap with kind's /kind/kubeadm.conf contract, uses a
stable in-VM endpoint for the current one-VM topology, and includes the exact
restart regression. Since you reviewed much of the original K8s plugin, your
input would be especially helpful.

[guanchzhou]

Independent check of this PR against the kube-proxy half of #2156, on Apple Silicon, container 1.3.1 daemon, plugin built from this branch (86735d0).

Threw away cluster k8s-2158 (2 CPU / 4 GiB). Did not migrate a pre-existing cluster; that is out of scope here as documented.

Create wrote /kind/kubeadm.conf with controlPlaneEndpoint: 127.0.0.1:6443. In-cluster kube-proxy ConfigMap was already:

server: https://127.0.0.1:6443

Stop, run an intervening Alpine container so vmnet could not reuse the address, then k8s start:

• node IP 192.168.64.8 → 192.168.64.10
• kind rewrote advertiseAddress to 192.168.64.10
• controlPlaneEndpoint stayed 127.0.0.1:6443
• kube-proxy ConfigMap still https://127.0.0.1:6443
• kube-proxy logs: zero no route to host
• apiserver.crt still present (this is the crash-loop we hit on an unpatched 1.3.1 restart)
• node Ready; a busybox pod fetched gitVersion: v1.35.5 from https://10.96.0.1:443/version

So for new clusters this PR does resolve the silent Service-routing break. It does not, and does not claim to, fix clusters created before /kind/kubeadm.conf existed.

Note for anyone reproducing on 1.3.1: this branch still embeds iptables-nft in the node prep script (#2120). The guest kernel here has no nftables, so create needed the same equal-length iptables-nft → iptables binary patch used on stock 1.3.1. Unrelated to the IP-rotation fix.