Tunnel authorizer sets port before Kubelet is ready

[brandond]

From @StrongMonkey:

When using k3s with karpenter, we have been seeing issues that we are not able to exec and log into pods that lands into karpenter nodes.

When digging further, we have found out that every time we made exec and log requests, k3s agent returns the following log

Tunnel authorizer checking dial request for 127.0.0.1:10250"
Mar 09 17:36:42 ip-10-0-0-125 k3s[15315]: time="2023-03-09T17:36:42Z" level=error msg="Remotedialer proxy error" error="connect not allowed"

When k3s-agent starts, we've seen that it was setting kubelet port to 0, which is not correct.

Tunnel authorizer set Kubelet Port 0

I think what's happening in here is that when using k3s agent with karpenter, the node resource was precreated by karpenter. So agent boots up, it already sees the node but at this point the kubelet port might not be populated back since k3s-agent is still starting. So it just sets the zero port.

[VestigeJ]

Reproduced using VERSION=1.26.2+k3s1
Validated using VERSION=v1.26.3-rc2+k3s1

Infrastructure

• Cloud

Node(s) CPU architecture, OS, and version:

Linux 5.14.21-150400.24.11-default x86_64 GNU/Linux 
PRETTY_NAME="SUSE Linux Enterprise Server 15 SP4"

Cluster Configuration:

NAME              STATUS   ROLES                       AGE     VERSION
ip-NODE           Ready    control-plane,etcd,master   4m53s   v1.26.3-rc2+k3s1 

Config.yaml:

write-kubeconfig-mode: 644
debug: true
token: gardenvarietydragons
protect-kernel-defaults: true
cluster-init: true
selinux: true
kubelet-arg:
  - port=11555

Reproduced using VERSION=v1.26.2+k3s1

$ curl https://get.k3s.io --output install-"k3s".sh
$ sudo chmod +x install-"k3s".sh
$ sudo groupadd --system etcd && sudo useradd -s /sbin/nologin --system -g etcd etcd
$ sudo modprobe ip_vs_rr
$ sudo modprobe ip_vs_wrr
$ sudo modprobe ip_vs_sh
$ sudo printf "on_oovm.panic_on_oom=0 \nvm.overcommit_memory=1 \nkernel.panic=10 \nkernel.panic_ps=1 \nkernel.panic_on_oops=1 \n" > ~/90-kubelet.conf
$ sudo cp 90-kubelet.conf /etc/sysctl.d/
$ sudo systemctl restart systemd-sysctl 
$ sudo INSTALL_K3S_VERSION=v1.26.2+k3s1 INSTALL_K3S_EXEC=server ./install-k3s.sh 
$ kg no,po -A //check for node status pod status
$ sudo journalctl -u k3s | grep "Kubelet Port" //view current logs from default install
$ conf //sudo vi /etc/rancher/k3s/config.yaml add kubelet-arg port
$ sudo systemctl restart k3s$ kg no,po -A //check for node status pod status
$ vi dnsutils.yaml
$ vi ginx.yaml
$ vi dnsutils.yaml
$ vi busy.yaml
$ k apply -f dnsutils.yaml -f ginx.yaml -f busy.yaml 
$ kgp -A //check for pods status'
$ sudo journalctl -u k3s | grep "Kubelet Port"

Results:

$ sudo journalctl -u k3s | grep "Kubelet Port"

Mar 23 17:57:22 ip-172-31-27-113 k3s[7811]: time="2023-03-23T17:57:22Z" level=info msg="Tunnel authorizer set Kubelet Port 10250"

$ sudo systemctl restart k3s
$ sudo journalctl -u k3s | grep "Kubelet Port" //output shows config.yaml value ignored

Mar 23 17:57:22 ip-172-31-27-113 k3s[7811]: time="2023-03-23T17:57:22Z" level=info msg="Tunnel authorizer set Kubelet Port 10250"
Mar 23 18:03:56 ip-172-31-27-113 k3s[10701]: time="2023-03-23T18:03:56Z" level=info msg="Tunnel authorizer set Kubelet Port 10250"

Validated using VERSION=v1.26.3-rc2+k3s1

Validation Steps

$ curl https://get.k3s.io --output install-"k3s".sh
$ sudo chmod +x install-"k3s".sh
$ sudo groupadd --system etcd && sudo useradd -s /sbin/nologin --system -g etcd etcd
$ sudo modprobe ip_vs_rr
$ sudo modprobe ip_vs_wrr
$ sudo modprobe ip_vs_sh
$ sudo printf "on_oovm.panic_on_oom=0 \nvm.overcommit_memory=1 \nkernel.panic=10 \nkernel.panic_ps=1 \nkernel.panic_on_oops=1 \n" > ~/90-kubelet.conf
$ sudo cp 90-kubelet.conf /etc/sysctl.d/
$ sudo systemctl restart systemd-sysctl 
$ sudo INSTALL_K3S_VERSION=v1.26.3-rc1+k3s1 INSTALL_K3S_EXEC=server ./install-k3s.sh 
$ kg no,po -A //check node pods status'
$ sudo journalctl -u k3s | grep "Kubelet Port"
$ sudo INSTALL_K3S_VERSION=v1.26.3-rc2+k3s1 INSTALL_K3S_EXEC=server ./install-k3s.sh 
$ kg no,po -A //check node pods status'
$ vi dnsutils.yaml
$ vi ginx.yaml
$ vi dnsutils.yaml
$ vi busy.yaml
$ k apply -f dnsutils.yaml -f ginx.yaml -f busy.yaml 
$ conf //sudo vi /etc/rancher/k3s/config.yaml add kubelet-arg port
$ sudo systemctl restart k3s
$ kgp -A //check the deployed node pods status is good
$ sudo journalctl -u k3s | grep "Kubelet Port"
$ get_report //output this template

Results:

$ sudo journalctl -u k3s | grep "Kubelet Port"

Mar 23 18:17:45 ip-172-31-27-43 k3s[6711]: time="2023-03-23T18:17:45Z" level=info msg="Tunnel authorizer set Kubelet Port 10250"
Mar 23 18:24:12 ip-172-31-27-43 k3s[8039]: time="2023-03-23T18:24:12Z" level=info msg="Tunnel authorizer set Kubelet Port 10250"
Mar 23 18:28:10 ip-172-31-27-43 k3s[12369]: time="2023-03-23T18:28:10Z" level=debug msg="Waiting for Ready condition to be updated for Kubelet Port assignment"
Mar 23 18:28:12 ip-172-31-27-43 k3s[12369]: time="2023-03-23T18:28:12Z" level=info msg="Tunnel authorizer set Kubelet Port 11555"

Additional context / logs:

$ cat busy.yaml

apiVersion: v1
kind: Service
metadata:
  name: busyb
spec:
  selector:
    app: busy
  clusterIP: None
  ports:
  - name: foo # Actually, no port is needed.
    port: 1234
    targetPort: 1234
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: busydep
spec:
  replicas: 3
  selector:
    matchLabels:
      app: busy
  template:
    metadata:
      labels:
        app: busy
    spec:
      containers:
        - name: busybox
          image: busybox:1.28
          command:
            - sleep
            - "3600"
          securityContext:
            runAsUser: 1000
            runAsNonRoot: true

$ cat ginx.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-nodeport-deployment
spec:
  selector:
    matchLabels:
      app: nginx-app-node
  replicas: 4
  template:
    metadata:
      labels:
        app: nginx-app-node
    spec:
      containers:
      - name: nginx
        image: ranchertest/mytestcontainer:unprivileged
        #image: maxross/mytestcontainer:unprivileged
        ports:
        - containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: nginx-app-node
  name: nginx-nodeport-svc
  namespace: default
spec:
  type: NodePort
  ports:
    - port: 8080
      nodePort: 30096
      name: http
  selector:
    app: nginx-app-node

$ cat dnsutils.yaml

apiVersion: v1
kind: Pod
metadata:
  name: dnsutils
  namespace: default
spec:
  securityContext:
     runAsUser: 1000
     runAsGroup: 3000
     fsGroup: 2000
  containers:
  - name: dnsutils
    image: registry.k8s.io/e2e-test-images/jessie-dnsutils:1.3
    command:
      - sleep
      - "3600"
    imagePullPolicy: IfNotPresent
    securityContext:
       allowPrivilegeEscalation: false
  restartPolicy: Always