Cannot install windows in airgap using private registry

[rancher-max]

Environmental Info:
RKE2 Version:

v1.21.3-rc5+rke2r2

Node(s) CPU architecture, OS, and Version:

WindowsBuildLabEx                                       : 17763.1.amd64fre.rs5_release.180914-1434
WindowsCurrentVersion                                   : 6.3
WindowsEditionId                                        : ServerDatacenter
WindowsInstallationType                                 : Server
WindowsInstallDateFromRegistry                          : 7/24/2021 6:08:30 AM
WindowsProductId                                        : 00430-00000-00000-AA230
WindowsProductName                                      : Windows Server 2019 Datacenter
WindowsRegisteredOrganization                           : Amazon.com
WindowsRegisteredOwner                                  : EC2
WindowsSystemRoot                                       : C:\Windows
WindowsVersion                                          : 1809

Cluster Configuration:

1 airgapped server node (using calico and strictaffinity), trying to add 1 windows agent but fails

Describe the bug:

Windows fails to install in an airgap environment when using the private registry. Error shown is:

PS C:\Users\Administrator> rke2 agent
I0813 20:49:24.471957    2804 ipsec_windows.go:20] ipsec is not supported on this platform
time="2021-08-13T20:49:24Z" level=warning msg="not running in CIS mode"
time="2021-08-13T20:49:24Z" level=info msg="Starting rke2 agent v1.21.3-rc5+rke2r2 (fca5af9e52c5b0969c520a1a6320b3a5c93f8909)"
time="2021-08-13T20:49:24Z" level=info msg="Running load balancer 127.0.0.1:6444 -> [172.31.7.42:9345]"
time="2021-08-13T20:49:44Z" level=info msg="Running load balancer 127.0.0.1:6443 -> [172.31.7.42:6443]"
time="2021-08-13T20:49:45Z" level=error msg="Failed to write C:\\Users\\ADMINI~1\\AppData\\Local\\Temp\\2\\rke2-resolv.conf: open C:\\Users\\ADMINI~1\\AppData\\Local\\Temp\\2\\rke2-resolv.conf: Access is denied."
time="2021-08-13T20:49:45Z" level=info msg="Checking local image archives in C:\\var\\lib\\rancher\\rke2\\agent\\images for index.docker.io/rancher/rke2-runtime:v1.21.3-rc5-rke2r2"
time="2021-08-13T20:49:45Z" level=error msg="Failed to load runtime image index.docker.io/rancher/rke2-runtime:v1.21.3-rc5-rke2r2: no local image available for index.docker.io/rancher/rke2-runtime:v1.21.3-rc5-rke2r2: not found in any file in C:\\var\\lib\\rancher\\rke2\\agent\\images: image not found"
time="2021-08-13T20:49:45Z" level=info msg="Checking local image archives in C:\\var\\lib\\rancher\\rke2\\agent\\images for index.docker.io/rancher/rke2-runtime:v1.21.3-rc5-rke2r2"
time="2021-08-13T20:49:45Z" level=error msg="Failed to load runtime image index.docker.io/rancher/rke2-runtime:v1.21.3-rc5-rke2r2: no local image available for index.docker.io/rancher/rke2-runtime:v1.21.3-rc5-rke2r2: not found in any file in C:\\var\\lib\\rancher\\rke2\\agent\\images: image not found"
time="2021-08-13T20:49:45Z" level=info msg="Using private registry config file at /etc/rancher/rke2/registries.yaml"
time="2021-08-13T20:49:45Z" level=info msg="Pulling runtime image index.docker.io/rancher/rke2-runtime:v1.21.3-rc5-rke2r2"
time="2021-08-13T20:49:45Z" level=fatal msg="failed to get runtime image index.docker.io/rancher/rke2-runtime:v1.21.3-rc5-rke2r2: Get \"https://<redacted_registry>/v2/\": failed to get TLS config for endpoint <redacted_registry>: failed to get system cert pool: crypto/x509: system root pool is not available on Windows"

Steps To Reproduce:

• Install airgapped server node with calico as cni
• Setup the registries.yaml tarball into C:\etc\rancher\rke2\ on windows airgapped node:

---
mirrors:
  docker.io:
    endpoint:
      - https://<redacted_registry>
  k8s.gcr.io:
    endpoint:
      - https://<redacted_registry>
configs:
  <redacted_registry>:
    auth:
      username: <redacted_username>
      password: <redacted_password>
    tls:
      cert_file: C:\Users\Administrator\certs\cert.pem
      key_file:  C:\Users\Administrator\certs\key.pem
      ca_file: C:\Users\Administrator\certs\ca.pem

• Setup config.yaml file with correct server and token
• Run rke2 agent

Expected behavior:

rke2 should install correctly

Actual behavior:

See the error at the end of the logs posted: failed to get system cert pool: crypto/x509: system root pool is not available on Windows. I believe this might be an issue with containerd?

Additional context / logs:

N/A

[rancher-max]

Moving this to test as it seems like the "epic" to:

• hardcode linker flag for windows runtime image #2671
• [release-1.22] hardcode linker flag for windows runtime image #2672
• [release-1.21] hardcode linker flag for windows runtime image #2673

[rancher-max]

Validated in v1.21.11-rc6+rke2r1, v1.22.8-rc6+rke2r1, and v1.23.5-rc5+rke2r1

• Windows is able to be installed in an airgap environment now using private registry method
• Confirmed that install is successful on both Windows 2022 and Windows 2019 servers.
• My steps were:

1. Start an airgapped server node running calico pointing to a private registry
2. Join an airgapped linux agent node
3. Setup windows node to use the private registry:
  a. Trust the cert on the node: `Import-Certificate -FilePath "C:\Users\Administrator\ca.pem" -CertStoreLocation cert:\CurrentUser\Root`
  b. Setup `c:\etc\rancher\rke2\config.yaml` to have the server and token information to join
  c. Setup `c:\etc\rancher\rke2\registries.yaml` to have the proper mirror(s) and config(s).
4. Run `rke2.exe agent`