provide arguments to k3s for providing environmental variables to containerd

[dxlr8r]

We want to use a proxy for fetcing images, but we do not want the complexity of managing no_proxy for other processes spawned by k3s (like etcd).

Usually you alter containerd's proxy settings, through environmental variables, when executing containerd from the init system. But we prefer to use the built in containerd in k3s, which is spawned from k3s, and not init. Meaning k3s needs some method to supply the required variables to containerd.

Suggested implementation:

k3s {agent,server} --containerd-env http_proxy=... https_proxy=... no_proxy=... 

[brandond]

I had some code to do something like this sitting around from a while back, take a look at the PR and see what you think.

[dxlr8r]

Looks awesome!

[rancher-max]

Confirmed containerd environment variables get passed through using master branch commitid ad1a40a96c400c17780cf9455f5da330d690194c

Simple steps:

1. Install k3s but don't start service: curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="--write-kubeconfig-mode 644" INSTALL_K3S_SKIP_START=true INSTALL_K3S_COMMIT=ad1a40a96c400c17780cf9455f5da330d690194c sh -
2. Edit file /etc/systemd/system/k3s.service to include the following:

[Service]
...
Environment="CONTAINERD_HTTP_PROXY=http://this.is.my.fake.proxy/"
Environment="CONTAINERD_HTTPS_PROXY=https://this.is.my.fake.proxy/"
...

3. Since that proxy doesn't actually exist, we expect failures. We can see multiple ways, showing two below:

$ journalctl -eu k3s -f
...
k3s[3282]: E0831 20:06:20.495632    3282 pod_workers.go:747] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"helm-install-traefik-crd--1-5ltf8_kube-system(fae1b559-9e34-47e8-b3c0-96339a63e4e6)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"helm-install-traefik-crd--1-5ltf8_kube-system(fae1b559-9e34-47e8-b3c0-96339a63e4e6)\\\": rpc error: code = Unknown desc = failed to get sandbox image \\\"rancher/mirrored-pause:3.1\\\": failed to pull image \\\"rancher/mirrored-pause:3.1\\\": failed to pull and unpack image \\\"docker.io/rancher/mirrored-pause:3.1\\\": failed to resolve reference \\\"docker.io/rancher/mirrored-pause:3.1\\\": failed to do request: Head \\\"https://registry-1.docker.io/v2/rancher/mirrored-pause/manifests/3.1\\\": proxyconnect tcp: dial tcp: lookup this.is.my.fake.proxy: no such host\"" pod="kube-system/helm-install-traefik-crd--1-5ltf8" podUID=fae1b559-9e34-47e8-b3c0-96339a63e4e6
...

# or 

$ kubectl get nodes,pods -A
NAME                    STATUS   ROLES                  AGE     VERSION
node/ip-172-31-28-150   Ready    control-plane,master   6m58s   v1.22.1+k3s-ad1a40a9

NAMESPACE     NAME                                         READY   STATUS              RESTARTS   AGE
kube-system   pod/helm-install-traefik-crd--1-5ltf8        0/1     ContainerCreating   0          6m47s
kube-system   pod/helm-install-traefik--1-flwvs            0/1     ContainerCreating   0          6m47s
kube-system   pod/coredns-85cb69466-9x4tb                  0/1     ContainerCreating   0          6m47s
kube-system   pod/metrics-server-9cf544f65-8swh7           0/1     ContainerCreating   0          6m47s
kube-system   pod/local-path-provisioner-64ffb68fd-p8c6n   0/1     ContainerCreating   0          6m47s

$ kubectl describe -n kube-system pod/coredns-85cb69466-9x4tb
Events:
  Type     Reason                  Age                    From               Message
  ----     ------                  ----                   ----               -------
  Normal   Scheduled               7m26s                  default-scheduler  Successfully assigned kube-system/coredns-85cb69466-9x4tb to ip-172-31-28-150
  Warning  FailedCreatePodSandBox  118s (x26 over 7m15s)  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to get sandbox image "rancher/mirrored-pause:3.1": failed to pull image "rancher/mirrored-pause:3.1": failed to pull and unpack image "docker.io/rancher/mirrored-pause:3.1": failed to resolve reference "docker.io/rancher/mirrored-pause:3.1": failed to do request: Head "https://registry-1.docker.io/v2/rancher/mirrored-pause/manifests/3.1": proxyconnect tcp: dial tcp: lookup this.is.my.fake.proxy: no such host

[nicolasbrechet]

Hi,

I'm testing k3s behind corporate proxy, and I couldn't make it work with environment variables "CONTAINERD_http_proxy" and "CONTAINERD_https_proxy".

There is something missing in the install script, cf line 730: you're retrieving the _proxy environment variables but not the CONTAINERD_ ones.

Thus the FILE_K3S_ENV is incomplete, and the k3s service starts without the environment variables prefixed with CONTAINERD.

I would add the following:
env | grep '^CONTAINERD_' | $SUDO tee -a ${FILE_K3S_ENV} >/dev/null

Let's hope it can make it to the 1.22.2 release 😉