pkg/kube: fix multus NAD apply ordering and unreachable retry back-off

[eriknordmark]

Description

Two small independent fixes to how cluster-init.sh applies the multus manifest.

The NetworkAttachmentDefinition CRD and an instance of it go in one apply. pkg/kube/multus-daemonset.yaml declares the CRD and the singleton network-instance-attachment object of that kind, and apply_multus_cni feeds the whole file to one kubectl apply. kubectl does not wait for a CRD to be established before creating a custom resource of that kind, so the instance can lose the race and fail with no matches for kind "NetworkAttachmentDefinition" while the CRD and the daemonset apply normally — leaving pods that attach to a network instance with no NAD to reference. Now the CRD is established first and the manifest re-applied to create whatever lost the race; apply is idempotent, so a run that did not hit the race does no extra work. The same commit stops sending kubectl's output to /dev/null on both failure paths, which is what made this slow to identify: the log said only Apply Multus, has failed, jump out now.

I hit this race on a branch carrying #5971 (the Go kube-init port), not on master. Master has the same shape — one manifest holding CRD and CR, one un-gated apply — so the race is available to it, but I have not observed it there. Treat that commit as hardening.

That said, the race is not hypothetical. The Go port in #5971 hit it independently and documents it in kubectlApply: "The retry loop is REQUIRED for manifests that combine a CRD and a custom resource of that CRD in the same file (e.g. multus-daemonset.yaml ships a NetworkAttachmentDefinition CRD plus a NAD instance): kubectl's API-discovery cache is built at process start, so the first apply creates the CRD but errors on the CR with 'no matches for kind'." It solves the same problem a different way — classifying that error as transient and retrying with backoff, rather than waiting for the CRD to be established. Either approach works; the shell path currently does neither.

The multus retry back-off was unreachable. An unconditional continue sat immediately after the apply_multus_cni call, so the block handling a failed apply — a log line plus a ten second sleep — could never run, and a failing apply retried in a tight loop while logging nothing. Moving the continue below that block restores both; the loop still restarts after every attempt, as before.

How to test and validate this PR

Both changes are in the EVE-k cluster bring-up path, so validation is on an EVE-k device.

No regression. Bring up an EVE-k node from scratch and confirm multus still initializes: /var/lib/multus_initialized appears, the log shows Done applying Multus, kubectl get crd network-attachment-definitions.k8s.cni.cncf.io is Established, and kubectl -n eve-kube-app get net-attach-def network-instance-attachment exists. Deploy an app on a network instance and confirm it gets its interface.

First fix. The race is hard to force on demand. The reachable checks are that the NAD is present after bring-up on a node where it previously came up missing, and that a failing apply now logs the object and reason. To exercise the new failure branch, make the CRD unable to establish (e.g. an invalid CRD schema) and confirm bring-up logs Apply Multus, NAD CRD not established: <kubectl output> and retries instead of proceeding as if multus were up.

Second fix. Make apply_multus_cni fail deterministically (e.g. make /etc/multus-daemonset-new.yaml unreadable) and watch the log emit Failed to apply multus cni, wait a while about every ten seconds, where before it spun silently.

shellcheck -x pkg/kube/cluster-init.sh is clean, and each commit is syntax-checked independently.

Changelog notes

Fixed cluster bring-up on Kubernetes-enabled devices sometimes leaving the network-attachment definition missing, which prevented applications from attaching to network instances. Failures while setting up multus are now logged with the underlying reason and retried at a sane interval instead of in a tight loop.

PR Backports

The template's list does not yet include 17.0-stable, which is where these belong.

• 17.0-stable: To be backported.
• 16.0-stable: No.
• 14.5-stable: No.
• 13.4-stable: No.

Checklist

• I've provided a proper description
• I've added the proper documentation
• I've tested my PR on amd64 device
• I've tested my PR on arm64 device
• I've written the test verification instructions
• I've set the proper labels to this PR

And the last but not least:

• I've checked the boxes above, or I've provided a good reason why I didn't
  check them.

No documentation change: no existing doc describes the apply ordering or the retry
cadence. Not run on amd64/arm64 hardware: both changes are architecture-independent
shell in the EVE-k bring-up path, with the device steps above for whoever runs them.

[naiming-zededa]

LGTM

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 23.31%. Comparing base (5d69266) to head (72bdad3).
⚠️ Report is 32 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #6242      +/-   ##
==========================================
+ Coverage   22.93%   23.31%   +0.37%     
==========================================
  Files         510      520      +10     
  Lines       93473    95189    +1716     
==========================================
+ Hits        21440    22189     +749     
- Misses      70292    71075     +783     
- Partials     1741     1925     +184     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.