Updating Traefik Ingress Configuration

[mkenne11]

I have K3s (release v1.17.0+k3s.1) installed on my Raspberry Pi cluster, running on Raspbian Buster.

I used the standard configuration for installation, and Traefik is being used as the ingress controller.

I would like to disable TLS verification in Traefik by setting the "insecureSkipVerify" setting to "true".
Note. I am running Kubernetes Dashboard with a self-signed certificate. This is on my home network and I'm not too concerned about verifying the validity of the certificate.

K3s appears to install Traefik using a Helm Chart, and I can see the Traefik chart manifest is installed in:
/var/lib/rancher/k3s/server/manifests/traefik.yaml

I have updated the traefik.yaml in this folder to include the additional setting:

apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
  name: traefik
  namespace: kube-system
spec:
  chart: https://%{KUBERNETES_API}%/static/charts/traefik-1.81.0.tgz
  set:
    rbac.enabled: "true"
    ssl.enabled: "true"
    metrics.prometheus.enabled: "true"
    kubernetes.ingressEndpoint.useDefaultPublishedService: "true"
    # New TLS skip verify certificate setting
    ssl.insecureSkipVerify: "true"

How can I apply the updated Traefik settings from my chart manifest file?

[a01fe]

Make sure your change is still in traefik.yaml and re-run the installer adding --no-deploy=traefik like this:

curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="--no-deploy=traefik" sh -

[mkenne11]

Thanks very much for your assistance @a01fe :) Btw. I installed k3s using the k3sup tool.

I believe the equivalent command for k3sup is:

k3sup install <standard-install-args> --k3s-extra-args '--no-deploy traefik'

Also, I saw the following recent k3s post where the author changed the traefik manifest file to "traefik-custom.yaml" and also applied the '--no-deploy traefik' argument to achieve the same result:
#1254

I'll try both suggestions and let you know the results.

[mkenne11]

Just confirming that both these methods worked to changed the traefik.yaml settings for my k3s install.

[theAkito]

Is there no good way of upgrading the traefik.yaml? Re-installing the entire k3s application just to apply a new Traefik configuration is at best a workaround.
What is the proper way to update the Traefik configuration?

[a01fe]

You need to run the installer once with --disable traefik to disable the built-in deployment. After that, all you have to do is change the manifest and k3s will reapply it.

[zhangguanzhang]

This is too inhumane

[brandond]

k3s 1.19 (and current master builds) include a new HelmChartConfig CRD that can be used to customize the values of a HelmChart without modifying the resource itself. The core use case for this is things like the packaged Traefik deployment. See: https://rancher.com/docs/k3s/latest/en/helm/#customizing-packaged-components-with-helmchartconfig

[hwk42]

The default port 443 need to be used by other service and cannot not be changed, to assign another port for k3s, I had to reinstall the k3s server,that is unpractical.Any plan to resolve this?

[a01fe]

If you don't want to reinstall, you could edit the service and add --disable traefik to the k3s server command line.

[praveenmak]

@brandond one quick question.

When communicating from "traefik <==> backend " and backend has https on port 22000.
What is best approach to hit the 22000 (https) port?
I am using

serversTransport:
insecureSkipVerify = true

Using that I see an error in Traefik logs.

'500 Internal Server Error' caused by: x509: cannot validate certificate for "IP" because it doesn't contain any IP SANs"

Is this because of 1.81 (default Traefik version in K3s 1.24) ?
Should I bump up the version of traefik ?

[brandond]

I'm not sure what that is for or where you're putting it, but you appear to be mixing yaml and ini style configuration syntax? You don't normally see : and = mixed together.

[praveenmak]

Ignore the syntax please, I edited the way toml syntax expects it to be.
We can edit the configMap this way.

k edit cm -n kube-system traefik

Then restart the traefik controller by scaling down and up the pod.

That insecure... flag is to ignore tls verify , when Traefik talks to the backend.

[davesilva]

If anyone else is like me and stumbles upon this while trying to figure out how to set insecureSkipVerify now that k3s has moved to Traefik v2, here's the easiest way I found to do it:

apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: traefik
  namespace: kube-system
spec:
  valuesContent: |-
    globalArguments:
    - "--serversTransport.insecureSkipVerify=true"

[cawoodm]

If you don't want to reinstall, you could edit the service and add --disable traefik to the k3s server command line.

I'm confused - in order to update the traefik config we have to disable traefik??

[brandond]

Yes, you disable it (to uninstall the old traefik v1 chart) and then enable it again to get v2. The v1 chart can't be safely directly upgraded to v2 so you have to basically uninstall and reinstall it.

[cawoodm]

OK, the solution with HelmChartConfig works nicely. Thanks @davesilva !

I battled with this issue for days now but there was nothing logged to the kube-system/traefik pods. They only log some unhelpful deprecation warnings which can't be traced. When I shell into the traefik ingress controller /var/log is empty.

Where/how can I see proper traefik error and access logs?

[brandond]

It sounds like you want to enable logging via the helm chart values: https://github.com/traefik/traefik-helm-chart/blob/master/traefik/values.yaml#L170-L171

[cawoodm]

I am unclear as to how this should work. Since we (as users) did not install traefik, I'm not sure how we should manage it via helm.

The above discussion seems to imply re-installing the k3s server which is a bit extreme. The HelmChartConfig option is nice but I don't see the link to a helm chart or a values.yaml.

[brandond]

K3s installs Traefik via helm chart. The values you can set in the HelmChartConfig as shown above are passed through directly to the chart installation; any of the upstream chart values can be used. You don't need to reinstall to modify the configuration. This is covered in the docs at https://rancher.com/docs/k3s/latest/en/helm/#customizing-packaged-components-with-helmchartconfig

[cawoodm]

I've read those docs but the various version numbers (helm, traefik, k3s, kubernetes...) are confusing.

Is it:

accessLog:
  enabled: true

OR

logs:
  access:
    enabled: true

Where can I see which helm version is employed?

[cawoodm]

It seems to be the latter. I am now getting access logs on stdout of the pod.

[loganmarchione]

Sorry to bump a dead issue. I'm trying to use @davesilva's suggestion, but it doesn't seem to be working. I'm using K3s v1.25.2+k3s1 .

I'm trying to run the UniFi controller in K3s, and the UniFi controller uses a self-signed certificate.

[browser]----Let's Encrypt cert---->[traefik]----self-signed cert---->[UniFi controller]

I applied the HelmChartConfig as suggested, but am still getting this in the UniFi web UI.

Bad Request
This combination of host and port requires TLS.

When running this command kubectl describe pod traefik-7c76489d5-whqrp -n kube-system, I can see this (last line below). What am I missing?

Containers:
  traefik:
    Container ID:  containerd://60638954e5ff3e30d86f741385b1bc5d4196c769258ec261e20b6bed083b41a0
    Image:         rancher/mirrored-library-traefik:2.6.2
    Image ID:      docker.io/rancher/mirrored-library-traefik@sha256:ad2226527eea71b7591d5e9dcc0bffd0e71b2235420c34f358de6db6d529561f
    Ports:         9100/TCP, 9000/TCP, 8000/TCP, 8443/TCP
    Host Ports:    0/TCP, 0/TCP, 0/TCP, 0/TCP
    Args:
      --serversTransport.insecureSkipVerify=true

[brandond]

@loganmarchione All that shows is that you've disabled certificate validation on the back-end connection. The error from your unifi web server is that a HTTPS connection between traefik and the pod is required, but you're using HTTP instead. This is not an error about invalid certificates, as you're not even using them.

If you're using an Ingress resource to configure the routing to the pod, you're probably missing configuration to tell traefik to use https. See: https://doc.traefik.io/traefik/v2.2/routing/providers/kubernetes-ingress/#on-service

[loganmarchione]

@brandond thanks for the quick reply, it's working now!

---
apiVersion: v1
kind: Service
metadata:
  name: unifi-service
  namespace: unifi
  annotations:
    traefik.ingress.kubernetes.io/service.serversscheme: https
spec:
  type: ClusterIP
  ports:
    - name: http
      targetPort: 8443
      port: 8443
      protocol: TCP
  selector:
    app: unifi

[brandond]

Note that fixing the port on the service to be properly named https instead of incorrectly labeling it as http would also have worked, due to traefik being smart enough to guess at protocols based on the port name.

[loganmarchione]

Note that fixing the port on the service to be properly named https instead of incorrectly labeling it as http would also have worked, due to traefik being smart enough to guess at protocols based on the port name.

Yep you're right, thanks!