Problem enabling proxy protocol on traefik using HelmChartConfig

[garutilorenzo]

Environmental Info:

K3s version: v1.23.8+k3s2

Node(s) CPU architecture, OS, and Version:

Linux inst-tslxf-k3s-servers 5.15.0-1016-oracle #20-Ubuntu SMP Mon Aug 8 07:08:08 UTC 2022 aarch64 aarch64 aarch64 GNU/Linux

Cluster Configuration:

One server and three worker

root@inst-tslxf-k3s-servers:~# kubectl get nodes
NAME                     STATUS   ROLES                       AGE   VERSION
inst-qoktf-k3s-workers   Ready    <none>                      36m   v1.23.8+k3s2
inst-tslxf-k3s-servers   Ready    control-plane,etcd,master   38m   v1.23.8+k3s2
inst-um89e-k3s-workers   Ready    <none>                      36m   v1.23.8+k3s2
k3s-third-worker-node    Ready    <none>                      35m   v1.23.8+k3s2

Describe the bug:

When installing k3s with traefik and trying to configure traefik with HelmChartConfig to use proxy protocol the configuration seems to be ignored (or partially ignored).

This is the HelmChartConfig configutation:

apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: traefik
  namespace: kube-system
spec:
  valuesContent: |-
    service:
      enabled: true
      type: NodePort

    # Configure ports
    ports:
      # The name of this one can't be changed as it is used for the readiness and
      # liveness probes, but you can adjust its config to your liking
      traefik:
        port: 9000
        # Use hostPort if set.
        # hostPort: 9000
        #
        # Use hostIP if set. If not set, Kubernetes will default to 0.0.0.0, which
        # means it's listening on all your interfaces and all your IPs. You may want
        # to set this value if you need traefik to listen on specific interface
        # only.
        # hostIP: 192.168.100.10

        # Override the liveness/readiness port. This is useful to integrate traefik
        # with an external Load Balancer that performs healthchecks.
        # Default: ports.traefik.port
        # healthchecksPort: 9000

        # Override the liveness/readiness scheme. Useful for getting ping to
        # respond on websecure entryPoint.
        # healthchecksScheme: HTTPS

        # Defines whether the port is exposed if service.type is LoadBalancer or
        # NodePort.
        #
        # You SHOULD NOT expose the traefik port on production deployments.
        # If you want to access it from outside of your cluster,
        # use `kubectl port-forward` or create a secure ingress
        expose: false
        # The exposed port for this service
        exposedPort: 9000
        # The port protocol (TCP/UDP)
        protocol: TCP
      web:
        port: 8000
        # hostPort: 8000
        expose: true 
        exposedPort: 80
        # The port protocol (TCP/UDP)
        protocol: TCP
        # Use nodeport if set. This is useful if you have configured Traefik in a
        # LoadBalancer
        nodePort: ${ingress_controller_http_nodeport}
        # Port Redirections
        # Added in 2.2, you can make permanent redirects via entrypoints.
        # https://docs.traefik.io/routing/entrypoints/#redirection
        # redirectTo: websecure
        #
        # Trust forwarded  headers information (X-Forwarded-*).
        # forwardedHeaders:
        #   trustedIPs: []
        #   insecure: false
        #
        # Enable the Proxy Protocol header parsing for the entry point
        proxyProtocol:
          trustedIPs:
            - 0.0.0.0/0
            - 127.0.0.1/32
          insecure: false
      websecure:
        port: 8443
        # hostPort: 8443
        expose: true
        exposedPort: 443
        # The port protocol (TCP/UDP)
        protocol: TCP
        nodePort: ${ingress_controller_https_nodeport}
        # Enable HTTP/3.
        # Requires enabling experimental http3 feature and tls.
        # Note that you cannot have a UDP entrypoint with the same port.
        # http3: true
        # Set TLS at the entrypoint
        # https://doc.traefik.io/traefik/routing/entrypoints/#tls
        tls:
          enabled: true
          # this is the name of a TLSOption definition
          options: ""
          certResolver: ""
          domains: []
          # - main: example.com
          #   sans:
          #     - foo.example.com
          #     - bar.example.com
        #
        # Trust forwarded  headers information (X-Forwarded-*).
        # forwardedHeaders:
        #   trustedIPs: []
        #   insecure: false
        #
        # Enable the Proxy Protocol header parsing for the entry point
        proxyProtocol:
          trustedIPs:
            - 0.0.0.0/0
            - 127.0.0.1/32
          insecure: false
        #
        # One can apply Middlewares on an entrypoint
        # https://doc.traefik.io/traefik/middlewares/overview/
        # https://doc.traefik.io/traefik/routing/entrypoints/#middlewares
        # /!\ It introduces here a link between your static configuration and your dynamic configuration /!\
        # It follows the provider naming convention: https://doc.traefik.io/traefik/providers/overview/#provider-namespace
        # middlewares:
        #   - namespace-name1@kubernetescrd
        #   - namespace-name2@kubernetescrd
        middlewares: []
      metrics:
        # When using hostNetwork, use another port to avoid conflict with node exporter:
        # https://github.com/prometheus/prometheus/wiki/Default-port-allocations
        port: 9100
        # hostPort: 9100
        # Defines whether the port is exposed if service.type is LoadBalancer or
        # NodePort.
        #
        # You may not want to expose the metrics port on production deployments.
        # If you want to access it from outside of your cluster,
        # use `kubectl port-forward` or create a secure ingress
        expose: false
        # The exposed port for this service
        exposedPort: 9100
        # The port protocol (TCP/UDP)
        protocol: TCP

but when I describe the traefik pod i cannot see this entries on Args:

• --entrypoints.web.proxyProtocol.trustedIPs=0.0.0.0/0,127.0.0.1/32
• --entrypoints.websecure.proxyProtocol.trustedIPs=0.0.0.0/0,127.0.0.1/32

and I cannot reach traefik from my public LB (with proxy protocol enabled)

root@inst-xgoau-k3s-servers:~# kubectl describe pod traefik-df4ff85d6-fjx47 -n kube-system
Name:                 traefik-df4ff85d6-fjx47
Namespace:            kube-system
Priority:             2000000000
Priority Class Name:  system-cluster-critical
Node:                 k3s-third-worker-node/10.0.0.84
Start Time:           Tue, 25 Oct 2022 09:26:12 +0000
Labels:               app.kubernetes.io/instance=traefik
                      app.kubernetes.io/managed-by=Helm
                      app.kubernetes.io/name=traefik
                      helm.sh/chart=traefik-10.19.300
                      pod-template-hash=df4ff85d6
Annotations:          prometheus.io/path: /metrics
                      prometheus.io/port: 9100
                      prometheus.io/scrape: true
Status:               Running
IP:                   10.42.2.17
IPs:
  IP:           10.42.2.17
Controlled By:  ReplicaSet/traefik-df4ff85d6
Containers:
  traefik:
    Container ID:  containerd://f9bee155a6ee45b382602ec1e2b499c00d466508ef3231cdc91891d58e9e8249
    Image:         rancher/mirrored-library-traefik:2.6.2
    Image ID:      docker.io/rancher/mirrored-library-traefik@sha256:ad2226527eea71b7591d5e9dcc0bffd0e71b2235420c34f358de6db6d529561f
    Ports:         9100/TCP, 9000/TCP, 8000/TCP, 8443/TCP
    Host Ports:    0/TCP, 0/TCP, 0/TCP, 0/TCP
    Args:
      --global.checknewversion
      --global.sendanonymoususage
      --entrypoints.metrics.address=:9100/tcp
      --entrypoints.traefik.address=:9000/tcp
      --entrypoints.web.address=:8000/tcp
      --entrypoints.websecure.address=:8443/tcp
      --api.dashboard=true
      --ping=true
      --metrics.prometheus=true
      --metrics.prometheus.entrypoint=metrics
      --providers.kubernetescrd
      --providers.kubernetesingress
      --providers.kubernetesingress.ingressendpoint.publishedservice=kube-system/traefik
      --entrypoints.websecure.http.tls=true
    State:          Running
      Started:      Tue, 25 Oct 2022 09:26:20 +0000
    Ready:          True
    Restart Count:  0
    Liveness:       http-get http://:9000/ping delay=10s timeout=2s period=10s #success=1 #failure=3
    Readiness:      http-get http://:9000/ping delay=10s timeout=2s period=10s #success=1 #failure=1
    Environment:    <none>
    Mounts:
      /data from data (rw)
      /tmp from tmp (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-fwbmp (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  data:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  tmp:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  kube-api-access-fwbmp:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 CriticalAddonsOnly op=Exists
                             node-role.kubernetes.io/control-plane:NoSchedule op=Exists
                             node-role.kubernetes.io/master:NoSchedule op=Exists
                             node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:                      <none>

But if i try to install traefik2 manually with the same values.yaml file (same k3s version and same OS/arch), this is the result:

root@inst-tslxf-k3s-servers:~# kubectl describe pods -n traefik
Name:         traefik-5744bd9cd4-6w977
Namespace:    traefik
Priority:     0
Node:         inst-tslxf-k3s-servers/10.0.0.10
Start Time:   Tue, 25 Oct 2022 12:24:34 +0000
Labels:       app.kubernetes.io/instance=traefik-traefik
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=traefik
              helm.sh/chart=traefik-17.0.5
              pod-template-hash=5744bd9cd4
Annotations:  prometheus.io/path: /metrics
              prometheus.io/port: 9100
              prometheus.io/scrape: true
Status:       Running
IP:           10.42.0.12
IPs:
  IP:           10.42.0.12
Controlled By:  ReplicaSet/traefik-5744bd9cd4
Containers:
  traefik:
    Container ID:  containerd://71084c2dd45dbe6481fc4843e8671585c938750c70e2d64e8307fe09b00bc51b
    Image:         traefik:2.9.1
    Image ID:      docker.io/library/traefik@sha256:4ebf68cdb32ce65e8786ac83ece782ec0dbe583471c04dfd0af43f245b96c88f
    Ports:         9100/TCP, 9000/TCP, 8000/TCP, 8443/TCP
    Host Ports:    0/TCP, 0/TCP, 0/TCP, 0/TCP
    Args:
      --global.checknewversion
      --global.sendanonymoususage
      --entrypoints.metrics.address=:9100/tcp
      --entrypoints.traefik.address=:9000/tcp
      --entrypoints.web.address=:8000/tcp
      --entrypoints.websecure.address=:8443/tcp
      --api.dashboard=true
      --ping=true
      --metrics.prometheus=true
      --metrics.prometheus.entrypoint=metrics
      --providers.kubernetescrd
      --providers.kubernetesingress
      --entrypoints.web.proxyProtocol.trustedIPs=0.0.0.0/0,127.0.0.1/32
      --entrypoints.websecure.http.tls=true
      --entrypoints.websecure.proxyProtocol.trustedIPs=0.0.0.0/0,127.0.0.1/32
    State:          Running
      Started:      Tue, 25 Oct 2022 12:25:59 +0000
    Ready:          True
    Restart Count:  0
    Liveness:       http-get http://:9000/ping delay=2s timeout=2s period=10s #success=1 #failure=3
    Readiness:      http-get http://:9000/ping delay=2s timeout=2s period=10s #success=1 #failure=1
    Environment:    <none>
    Mounts:
      /data from data (rw)
      /tmp from tmp (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-k6h5s (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  data:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  tmp:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  kube-api-access-k6h5s:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason     Age    From               Message
  ----    ------     ----   ----               -------
  Normal  Scheduled  2m58s  default-scheduler  Successfully assigned traefik/traefik-5744bd9cd4-6w977 to inst-tslxf-k3s-servers
  Normal  Pulling    2m52s  kubelet            Pulling image "traefik:2.9.1"
  Normal  Pulled     93s    kubelet            Container image "traefik:2.9.1" already present on machine
  Normal  Created    93s    kubelet            Created container traefik
  Normal  Started    93s    kubelet            Started container traefik

and I can reach traefik from my public LB without any problem.

Steps To Reproduce:

Tested with terraform this terraform module.

[brandond]

Are you sure you're using values for the correct Traefik chart release? The version of K3s you're using includes v10.19.3 of the chart: https://github.com/traefik/traefik-helm-chart/blob/v10.19.3/traefik/values.yaml - this version does not support any of the proxyProtocol stuff that you're trying to use.

The most recent release of K3s includes a newer version of the chart: https://github.com/traefik/traefik-helm-chart/blob/v12.0.0/traefik/values.yaml

I'm not sure that will help you much though, as the proxyProtocol stuff was added literally a week ago today, and we haven't had a chance to pull it in to a K3s release yet: traefik/traefik-helm-chart#673

[garutilorenzo]

Hi @brandond,

I've also tried also to use this example:

apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: traefik
  namespace: kube-system
spec:
  valuesContent: |-
    image:
      name: traefik
      tag: v2.6.1
    proxyProtocol:
      enabled: true
      trustedIPs:
        - 10.0.0.0/8
    forwardedHeaders:
      enabled: true
      trustedIPs:
        - 10.0.0.0/8
    ssl:
      enabled: true
      permanentRedirect: false
      enabled: true
      trustedIPs:
        - 10.0.0.0/8
    ssl:
      enabled: true
      permanentRedirect: false

from the docs, but I have the same problem.
Maybe I'm missing something from the docs but where I can find the included Helm charts in k3s and the chart version?

[brandond]

As I mentioned, the version of the chart we're bundling doesn't support configuring proxy protocol support, so it doesn't really matter what example you tried.

You can find the chart version in the k3s repo: https://github.com/k3s-io/k3s/blob/v1.25.3+k3s1/manifests/traefik.yaml

We currently only list the traefik binary version in the release notes. We may improve that in future releases.

[garutilorenzo]

Ok thank you @brandond.
I think is better to update the docs on docs.k3s.io, and remove the examples that the bundled Helm chart does not support.

[brandond]

Yes, that's fair. I'm honestly not sure where those crept in - I think someone must have copy-pasted in some bad example values during our recent documentation restructuring.

I'll reopen this to track fixing the docs.

[shishkin]

Thanks for pointing the docs and versions mismatch. I ended up disabling Traefik in K3s entirely and installing it via a Helm chart so that I can rely on using the latest version and matching docs. I use Terraform automation on top of K3s installation, so one more Helm chart is not a big deal.

This is not a critique of K3s though. There are valid reasons to not include the latest dependency versions into a distribution.

[dereknola]

This was cleaned up in k3s-io/docs#38. At this point in v1.26.1+k3s1, we do have a traefik chart that supports proxy protocol, but we want to keep the docs simple with an example that supports all our K3s releases.