[release-1.25] apiserver egress proxy attempts to use tunnel for connections to hostnetwork pods in agent mode

[brandond]

Backport:

• apiserver egress proxy attempts to use tunnel for connections to hostnetwork pods in agent mode #6830

[est-suse]

Validated on release-1.25 branch with commit 14f2226

k3s version v1.25.6+k3s-14f2226b (14f2226b)
go version go1.19.5

Linux ip-172-31-36-161 5.14.21-150400.22-default #1 SMP PREEMPT_DYNAMIC Wed May 11 06:57:18 UTC 2022 (49db222) x86_64 x86_64 x86_64 GNU/Linux
NAME="SLES"
VERSION="15-SP4"
VERSION_ID="15.4"
PRETTY_NAME="SUSE Linux Enterprise Server 15 SP4"
ID="sles"
ID_LIKE="suse"
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:suse:sles:15:sp4"
DOCUMENTATION_URL="https://documentation.suse.com/"

Cluster Configuration:

1 server

Config.yaml:

# config 1
write-kubeconfig-mode: 644
# config 2
write-kubeconfig-mode: 644
egress-selector-mode: cluster
# config 3
write-kubeconfig-mode: 644
egress-selector-mode: pod
# config 4
write-kubeconfig-mode: 644
egress-selector-mode: disabled

Additional files

# hp.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-hostport-deployment
spec:
  selector:
    matchLabels:
      app: test-hostport-app
  replicas: 6
  template:
    metadata:
      labels:
        app: test-hostport-app
    spec:
      containers:
      - name: test-hostport
        image: ranchertest/mytestcontainer:unprivileged
        ports:
        - containerPort: 8080
          hostPort: 8080

Testing Steps
Copy config.yaml
$ sudo mkdir -p /etc/rancher/k3s && sudo cp config.yaml /etc/rancher/k3s
Install k3s
Validate the presence (or not) of the egress.k3s.io/cluster: "true" node label: kubectl get node -o yaml | grep -i label -A 15
Validate that hostport resources deploy successfully and can be accessed:
$ kubectl apply -f hp.yaml
$ curl node ip:8080/name.html # should return pod name

For default case, egress-selector-mode: cluster, egress-selector-mode: pod and egress-selector-mode: disabled, the node label is present:

$ kubectl get nodes -o yaml | grep -i label -A 15
    labels:
      beta.kubernetes.io/arch: amd64
      beta.kubernetes.io/instance-type: k3s
      beta.kubernetes.io/os: linux
      egress.k3s.io/cluster: "true"
      kubernetes.io/arch: amd64
      kubernetes.io/hostname: ip-172-31-14-128
      kubernetes.io/os: linux
      node-role.kubernetes.io/control-plane: "true"
      node-role.kubernetes.io/master: "true"
      node.kubernetes.io/instance-type: k3s
...

Validation Results:
In the default case and egress-selector-mode: disabled, the node label is NOT present:

 kubectl get nodes -o yaml | grep -i label -A 15
    labels:
      beta.kubernetes.io/arch: amd64
      beta.kubernetes.io/instance-type: k3s
      beta.kubernetes.io/os: linux
      kubernetes.io/arch: amd64
      kubernetes.io/hostname: ip-172-31-10-98
      kubernetes.io/os: linux
      node-role.kubernetes.io/control-plane: "true"
      node-role.kubernetes.io/master: "true"
      node.kubernetes.io/instance-type: k3s

In the egress-selector-mode: pod and egress-selector-mode: cluster, it is present:

$ kubectl get nodes -o yaml | grep -i label -A 15
    labels:
      beta.kubernetes.io/arch: amd64
      beta.kubernetes.io/instance-type: k3s
      beta.kubernetes.io/os: linux
      egress.k3s.io/cluster: "true"
      kubernetes.io/arch: amd64
      kubernetes.io/hostname: ip-172-31-10-98
      kubernetes.io/os: linux
      node-role.kubernetes.io/control-plane: "true"
      node-role.kubernetes.io/master: "true"
      node.kubernetes.io/instance-type: k3s