[1.21 release] Can't not start multus DaemonSet in an air-gapped environment

[bk201]

Environmental Info:
RKE2 Version: v1.21.2+rke2r1

Node(s) CPU architecture, OS, and Version:
A KVM VM. On RancherOSv2.

Cluster Configuration:
Single node server provision.

Describe the bug:
Multus DaemonSet is not running in an air-gapped environment.

Steps To Reproduce:

• Try Installed RKE2 in air-gapped environment.
• Prepare and copy these image tarballs to /var/lib/rancher/rke2/agent/images/:

  rke2-images.linux-amd64.tar.zst
  rke2-images-multus.linux-amd64.tar.zst
  

• Start rke-server with the config:

  cni: multus,canal
  

• Wait for kube-multus-xxx to be provisioned.

Expected behavior:
Status of pod kube-multus-ds-pn577 should be running.

Actual behavior:
Status of pod kube-multus-ds-pn577 is Init:ImagePullBackOff.

Additional context / logs:

$ kubectl describe -n kube-system pod kube-multus-ds-pn577
...
Init Containers:
  cni-plugins:
    Container ID:
    Image:          rancher/hardened-cni-plugins:v0.9.1-build20210414
    Image ID:
    Port:           <none>
    Host Port:      <none>
    State:          Waiting
      Reason:       ImagePullBackOff
    Ready:          False
...
Events:
  Type     Reason     Age                  From               Message
  ----     ------     ----                 ----               -------
  Normal   Scheduled  11m                  default-scheduler  Successfully assigned kube-system/kube-multus-ds-pn577 to node1
  Warning  Failed     11m                  kubelet            Failed to pull image "rancher/hardened-cni-plugins:v0.9.1-build20210414": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/rancher/hardened-cni-plugins:v0.9.1-build20210414": failed to resolve reference "docker.io/rancher/hardened-cni-plugins:v0.9.1-build20210414": failed to do request: Head "https://registry-1.docker.io/v2/rancher/hardened-cni-plugins/manifests/v0.9.1-build20210414": dial tcp: lookup registry-1.docker.io on 127.0.0.1:53: read udp 127.0.0.1:40327->127.0.0.1:53: read: connection refused

Looks like image rancher/hardened-cni-plugins is missing in the building script.

[brandond]

@manuelbuil can you take a look at this?

[manuelbuil]

Environmental Info:
RKE2 Version: v1.21.2+rke2r1

Node(s) CPU architecture, OS, and Version:
A KVM VM. On RancherOSv2.

Cluster Configuration:
Single node server provision.

Describe the bug:
Multus DaemonSet is not running in an air-gapped environment.

Steps To Reproduce:

* Try Installed RKE2 in air-gapped environment.

* Prepare and copy these image tarballs to `/var/lib/rancher/rke2/agent/images/`:
  ```
  rke2-images.linux-amd64.tar.zst
  rke2-images-multus.linux-amd64.tar.zst
  ```

* Start rke-server with the config:
  ```
  cni: multus,canal
  ```

* Wait for `kube-multus-xxx` to be provisioned.

Expected behavior:
Status of pod kube-multus-ds-pn577 should be running.

Actual behavior:
Status of pod kube-multus-ds-pn577 is Init:ImagePullBackOff.

Additional context / logs:

$ kubectl describe -n kube-system pod kube-multus-ds-pn577
...
Init Containers:
  cni-plugins:
    Container ID:
    Image:          rancher/hardened-cni-plugins:v0.9.1-build20210414
    Image ID:
    Port:           <none>
    Host Port:      <none>
    State:          Waiting
      Reason:       ImagePullBackOff
    Ready:          False
...
Events:
  Type     Reason     Age                  From               Message
  ----     ------     ----                 ----               -------
  Normal   Scheduled  11m                  default-scheduler  Successfully assigned kube-system/kube-multus-ds-pn577 to node1
  Warning  Failed     11m                  kubelet            Failed to pull image "rancher/hardened-cni-plugins:v0.9.1-build20210414": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/rancher/hardened-cni-plugins:v0.9.1-build20210414": failed to resolve reference "docker.io/rancher/hardened-cni-plugins:v0.9.1-build20210414": failed to do request: Head "https://registry-1.docker.io/v2/rancher/hardened-cni-plugins/manifests/v0.9.1-build20210414": dial tcp: lookup registry-1.docker.io on 127.0.0.1:53: read udp 127.0.0.1:40327->127.0.0.1:53: read: connection refused

Looks like image rancher/hardened-cni-plugins is missing in the building script.

Yes, that's the problem! We forgot to add hardened-cni-plugins image. Thanks for the report :)

[brandond]

reopening; missed QA testing

[rancher-max]

Validated on v1.21.3+rke2r1

Can deploy airgap with multus using any method: private registry, tarball