Setting CONTAINERD_NO_PROXY does not work

[aussielunix]

Environmental Info:
K3s Version:

# k3s -v
k3s version v1.23.6+k3s1 (418c3fa8)
go version go1.17.5

Node(s) CPU architecture, OS, and Version:

# uname -a
Linux k3s-01 5.15.0-35-generic #36-Ubuntu SMP Sat May 21 02:24:07 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Cluster Configuration:

Tested with single node with embedded sqlite and 3 node HA with embedded etcd

Describe the bug:

Setting CONTAINERD_NO_PROXY does not work.

As mentioned in #2026, #3552 and #4009, a new feature has been added to configure HTTP proxy settings, for Containerd.

I can confirm that any variable prefixed with CONTAINERD_ is passed through with the exception of CONTAINERD_NO_PROXY

Steps To Reproduce:

• Installed K3s:

curl -sfL https://get.k3s.io | INSTALL_K3S_SKIP_START=true sh -

• create /etc/systemd/system/k3s.service.env

CONTAINERD_HTTP_PROXY=http://this.is.my.fake.proxy/
CONTAINERD_HTTPS_PROXY=https://this.is.my.fake.proxy/
CONTAINERD_NO_PROXY=".docker.io"

• start k3s

systemctl start k3s.service

Expected behavior:

containerd should have the following environment variables set which should allow it to pull containers from the Docker hub without going via the HTTP_PROXY.

HTTP_PROXY=http://this.is.my.fake.proxy/
HTTPS_PROXY=https://this.is.my.fake.proxy/
NO_PROXY=".docker.io"

Actual behavior:

Containers can not be pulled.

Failed to pull image "busybox": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/library/busybox:latest": failed to resolve reference "docker.io/library/busybox:latest": failed to do request: Head "https://registry-1.docker.io/v2/library/busybox/manifests/latest": proxyconnect tcp: dial tcp: lookup this.is.my.fake.proxy on 127.0.0.53:53: no such host

Additional context / logs:

This is where vars prefixed with CONTAINERD_ are passed through.

What I think is happening is https://github.com/k3s-io/k3s/blob/master/pkg/server/server.go#L440-L455 is always setting NO_PROXY which is overriding the CONTAINERD_NO_PROXY

An inspection of the environment for the containerd process shows NO_PROXY is always set regardless if CONTAINERD_NO_PROXY is set or not.

# cat /proc/29698/environ 
LANG=C.UTF-8PATH=/var/lib/rancher/k3s/data/8c2b0191f6e36ec6f3cb68e2302fcc4be850c6db31ec5f8a74e4b3be403101d8/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin:/var/lib/rancher/k3s/data/8c2b0191f6e36ec6f3cb68e2302fcc4be850c6db31ec5f8a74e4b3be403101d8/bin/auxINVOCATION_ID=a3956a06a8284ebc968c16ac89e3cb65JOURNAL_STREAM=8:298854SYSTEMD_EXEC_PID=29687HTTP_PROXY=http://this.is.my.fake.proxy/HTTPS_PROXY=https://this.is.my.fake.proxy/NO_PROXY=.svc,.cluster.local,10.42.0.0/16,10.43.0.0/16RES_OPTIONS= K3S_DATA_DIR=/var/lib/rancher/k3s/data/8c2b0191f6e36ec6f3cb68e2302fcc4be850c6db31ec5f8a74e4b3be403101d8NODE_NAME=k3s-01.example.com

• change /etc/systemd/system/k3s.service.env and it works (but not ideal).

CONTAINERD_HTTP_PROXY=http://this.is.my.fake.proxy/
CONTAINERD_HTTPS_PROXY=https://this.is.my.fake.proxy/
NO_PROXY=".docker.io"

[brandond]

The problem is that environment variables aren't ordered, so CONTAINERD_ variables may be shadowed by other variables, if they are processed later in the loop. This is easy enough to fix.

[mdrahman-suse]

Validated on master with 61b714b

Environment Details

Infrastructure

• Cloud
• Hosted

Node(s) CPU architecture, OS, and Version:

Linux  5.4.0-1009-aws #9-Ubuntu x86_64 GNU/Linux
"Ubuntu 20.04 LTS"

Cluster Configuration:

Single node

Config.yaml:

write-kubeconfig-mode: 644
token: summerheat
debug: true

Replication Steps

• Copy config.yaml

sudo mkdir -p /etc/rancher/k3s && sudo cp config.yaml /etc/rancher/k3s

• Install K3S (without starting):

$ curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.24.1+k3s1 INSTALL_K3S_SKIP_START=true sh -

• Modify /etc/systemd/system/k3s.service.env with

CONTAINERD_HTTP_PROXY=http://this.is.my.fake.proxy/
CONTAINERD_HTTPS_PROXY=https://this.is.my.fake.proxy/
CONTAINERD_NO_PROXY=".docker.io"

• Start k3s service

$ sudo systemctl start k3s.service

• The cluster is expected to be down as fake proxy is used

$ kubectl get nodes
NAME                   STATUS   ROLES                  AGE     VERSION      
node/ip-172-31-43-48   Ready    control-plane,master   2m38s   v1.24.1+k3s1

$ kubectl get pod -n kube-system
NAME                                      READY   STATUS              RESTARTS   AGE
helm-install-traefik-2mbjs                0/1     ContainerCreating   0          8m51s
helm-install-traefik-crd-7f7gw            0/1     ContainerCreating   0          8m51s
local-path-provisioner-7b7dc8d6f5-qk29b   0/1     ContainerCreating   0          8m50s
coredns-b96499967-86s2t                   0/1     ContainerCreating   0          8m50s
metrics-server-668d979685-5d6hn           0/1     ContainerCreating   0          8m50s

Results:

• Check containerd process ID

$ ps -aux | grep  containerd
root       13726  0.4  1.3 759200 52632 ?        Sl   22:55   0:01 containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd
ubuntu     14379  0.0  0.0   8160   736 pts/0    R+   23:01   0:00 grep --color=auto containerd

• Use the process ID to look up the environment variables

$ sudo cat /proc/13726/environ
LANG=C.UTF-8PATH=/var/lib/rancher/k3s/data/800a5d0e7fc03a5fbad1b1beb4ef598c103cb08b0c360588fbe8662d7d207f22/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin:/var/lib/rancher/k3s/data/800a5d0e7fc03a5fbad1b1beb4ef598c103cb08b0c360588fbe8662d7d207f22/bin/auxINVOCATION_ID=8e97e946464f4d30ab9197973af54c84JOURNAL_STREAM=9:149719HTTP_PROXY=http://this.is.my.fake.proxy/HTTPS_PROXY=https://this.is.my.fake.proxy/NO_PROXY=.svc,.cluster.local,10.42.0.0/16,10.43.0.0/16RES_OPTIONS= K3S_DATA_DIR=/var/lib/rancher/k3s/data/800a5d0e7fc03a5fbad1b1beb4ef598c103cb08b0c360588fbe8662d7d207f22NODE_NAME=ip-172-31-43-48

• Observe that the NO_PROXY env var is NOT set as expected

Validation Steps

• Copy config.yaml

sudo mkdir -p /etc/rancher/k3s && sudo cp config.yaml /etc/rancher/k3s

• Install K3S (without starting):

curl -sfL https://get.k3s.io | INSTALL_K3S_COMMIT=61b714b2dd2b70535bca1089cb160bbb6b886226 INSTALL_K3S_SKIP_START=true sh -

• Modify /etc/systemd/system/k3s.service.env with

CONTAINERD_HTTP_PROXY=http://this.is.my.fake.proxy/
CONTAINERD_HTTPS_PROXY=https://this.is.my.fake.proxy/
CONTAINERD_NO_PROXY=".docker.io"

• Start k3s service

$ sudo systemctl start k3s.service

• The cluster is expected to be down as fake proxy is used

$ kubectl get nodes
NAME                   STATUS   ROLES                  AGE   VERSION               
node/ip-172-31-43-48   Ready    control-plane,master   17s   v1.24.2+k3s-61b714b2

$ kubectl get pod -n kube-system
NAME                                      READY   STATUS              RESTARTS   AGE
helm-install-traefik-rd79s                0/1     ContainerCreating   0          14m
helm-install-traefik-crd-x9w6m            0/1     ContainerCreating   0          14m
metrics-server-668d979685-xj5t5           0/1     ContainerCreating   0          14m
coredns-b96499967-2bzc2                   0/1     ContainerCreating   0          14m
local-path-provisioner-7b7dc8d6f5-rk2qd   0/1     ContainerCreating   0          14m

Results:

• Check containerd process ID

$ ps -aux | grep  containerd
root       14869  1.2  1.3 759380 55064 ?        Sl   23:09   0:00 containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd
ubuntu     15367  0.0  0.0   8160   736 pts/0    R+   23:10   0:00 grep --color=auto containerd

• Use the process ID to look up the environment variables

$ sudo cat /proc/14869/environ
LANG=C.UTF-8PATH=/var/lib/rancher/k3s/data/862082577ea5e18ca377cc06aa1fddedbce10f8aaab2467266152ffb56c665fd/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin:/var/lib/rancher/k3s/data/862082577ea5e18ca377cc06aa1fddedbce10f8aaab2467266152ffb56c665fd/bin/auxINVOCATION_ID=8653a01a0292483da6a6b50c286c7064JOURNAL_STREAM=9:161245HTTP_PROXY=http://this.is.my.fake.proxy/HTTPS_PROXY=https://this.is.my.fake.proxy/NO_PROXY=.docker.io,.svc,.cluster.local,10.42.0.0/16,10.43.0.0/16RES_OPTIONS= K3S_DATA_DIR=/var/lib/rancher/k3s/data/862082577ea5e18ca377cc06aa1fddedbce10f8aaab2467266152ffb56c665fdNODE_NAME=ip-172-31-43-48

• Observe that the NO_PROXY env var is set as expected

[mdrahman-suse]

Validated in 1.24.2-rc1+k3s1

[aussielunix]

I sure do appreciate how fast this was resolved.
Thank you very much.