apiserver egress proxy attempts to use tunnel for connections to hostnetwork pods in agent mode

[brandond]

From @PaulSD in #6829

With egress-selector-mode: agent (the default), and egress.X.io/cluster: true node labels (also a default), if any Pods are running with hostNetwork: true (and therefore use the Node IP), then the API Server will attempt to use agent tunnels to communicate with those Pods, but tunnel authorization will fail, resulting in connection failures.

[mdrahman-suse]

Validated on master branch with commit 2156015

Environment Details

Infrastructure

• Cloud
• Hosted

Node(s) CPU architecture, OS, and Version:

Linux ip-172-31-45-8 5.15.0-1026-aws #30-Ubuntu SMP Wed Nov 23 14:15:21 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
PRETTY_NAME="Ubuntu 22.04.1 LTS"

Cluster Configuration:

1 server

Config.yaml:

# config1.yaml - default
write-kubeconfig-mode: 644
token: summer
cluster-init: true

# config2.yaml - disabled
write-kubeconfig-mode: 644
token: summer
cluster-init: true
egress-selector-mode: disabled

# config3.yaml - cluster
write-kubeconfig-mode: 644
token: summer
cluster-init: true
egress-selector-mode: cluster

# config4.yaml - pod
write-kubeconfig-mode: 644
token: summer
cluster-init: true
egress-selector-mode: pod

Additional files

• hostport.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-hostport-deployment
spec:
  selector:
    matchLabels:
      app: test-hostport-app
  replicas: 6
  template:
    metadata:
      labels:
        app: test-hostport-app
    spec:
      containers:
      - name: test-hostport
        image: ranchertest/mytestcontainer:unprivileged
        ports:
        - containerPort: 8080
          hostPort: 8080

Testing Steps

1. Copy config.yaml

$ sudo mkdir -p /etc/rancher/k3s && sudo cp config.yaml /etc/rancher/k3s

2. Install k3s
3. Check node label: kubectl get nodes -o yaml | grep -i label -A 15

Validation Results:

• k3s version used for validation:

k3s version v1.26.1+k3s-21560155 (21560155)
go version go1.19.5

• Observed NO label with egress.k3s.io/cluster on default and disabled mode

$ kubectl get nodes -o yaml | grep -i label -A 15
    labels:
      beta.kubernetes.io/arch: amd64
      beta.kubernetes.io/instance-type: k3s
      beta.kubernetes.io/os: linux
      kubernetes.io/arch: amd64
      kubernetes.io/hostname: ip-172-31-45-8
      kubernetes.io/os: linux
      node-role.kubernetes.io/control-plane: "true"
      node-role.kubernetes.io/etcd: "true"
      node-role.kubernetes.io/master: "true"
      node.kubernetes.io/instance-type: k3s
    name: ip-172-31-45-8
    resourceVersion: "714"
    uid: ce995590-d5d7-4dd0-9a8c-f43350cd65a4
  spec:
    podCIDR: 10.42.0.0/24

• Observed label egress.k3s.io/cluster: "true" with cluster and pod mode

$ kubectl get nodes -o yaml | grep -i label -A 15
    labels:
      beta.kubernetes.io/arch: amd64
      beta.kubernetes.io/instance-type: k3s
      beta.kubernetes.io/os: linux
      egress.k3s.io/cluster: "true"
      kubernetes.io/arch: amd64
      kubernetes.io/hostname: ip-172-31-45-8
      kubernetes.io/os: linux
      node-role.kubernetes.io/control-plane: "true"
      node-role.kubernetes.io/etcd: "true"
      node-role.kubernetes.io/master: "true"
      node.kubernetes.io/instance-type: k3s
    name: ip-172-31-45-8
    resourceVersion: "1342"
    uid: bd6604b9-a78c-413e-a7db-6a1e75688fea
  spec:

• Deployed hostport.yaml and validated connectivity

$ curl <node-ip>:8080/name.html
nginx-hostport-deployment-c5b858567-56qjl