Traefik 2.0 integration

[Zikoel]

Is your feature request related to a problem? Please describe.
The feature tls-passthrough is missing.
Es. Installing argocd on the cluster is difficult due the missing of this feature.

Describe the solution you'd like
Substitute the actual version < 2.0 with the actual (that reach the GA with version 2.0)

Describe alternatives you've considered
Describe a reproducible way for remove the actual version in favor of the most updated version.

Additional context
The version of Traefik 2.0 seems most kubernetes friendly so, this seems to me a very natural step to do!

[sebastien-prudhomme]

It seems that Traefik 2.0 drops Ingress support and so drops compatibility with most of the community Helm charts.

So i don't think it's a good idea if it's really the case, even if Ingress has some limitations.

[Zikoel]

From the documentation Traefik & Kubernetes seems that the Ingress support is not dropped but fully supported, more of this there is another custom Ingress the with enhanced capabilities for complex routes and middlewares!

[sebastien-prudhomme]

Thanks for reading the full doc! It was not clear as also talk on migrating to their new CRD.

As k3s is using a chart to deploy Traefik, i guess we need to wait for the new version of the chart which is still not ready for production: traefik/traefik#5440

[denysvitali]

I'm using this and it works pretty well once you disable the original traefik:

write_files:
- encoding: ""
  content: |-
    apiVersion: helm.cattle.io/v1
    kind: HelmChart
    metadata:
      name: traefik
      namespace: kube-system
    spec:
      chart: traefik
      repo: https://nanosapp.github.io/chart-releases/stable
      targetNamespace: kube-system
      set:
        image.tag: 2.0.4
        ports.web.port: 80
        ports.websecure.port: 443
  owner: root
  path: /var/lib/rancher/k3s/server/manifests/traefik.yaml
  permissions: '0600'

k3os:
  dns_nameservers:
    - 1.1.1.1
    - 8.8.8.8
  labels:
    region: uk1
  k3s_args:
    - server
    - "--no-deploy=traefik"

[sotiris84]

Looking forward as well the update to Traefik 2.0

[brandond]

@denysvitali that repo appears to be gone; do you have a substitute that you're still using?

[denysvitali]

@brandond Nope, that repo should still work, look at https://github.com/nanosapp/chart-releases/tree/gh-pages/stable/traefik and you'll see the chart source. I've created that repo and the company I work for owns it 👍

[brandond]

I'll give it another try. I'm just getting started with Helm and don't quite grok the repo format; I'll admit to not actually loading the chart but just hit the repo URL from the CR with my browser and got a 404.

[denysvitali]

That's just because Chart repos do not need an index.html. If you visit the repo URL with /index.yaml appended, you'll see all of the charts available in that specific repository. Check it out! 😊

[mickkael]

The Helm chart is now production ready
traefik/traefik-helm-chart#104

https://github.com/containous/traefik-helm-chart/blob/master/traefik/Chart.yaml

[r1cebank]

Been trying to get the Helm chart to work, but had not a lot of luck, followed the documentation on Traefik's website and got it working with k3s. The ansible role is here https://github.com/r1cebank/geck/tree/traefik-v2/roles/traefik

[mamiu]

@r1cebank
Create a yaml file with following content:

apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
  name: traefik
  namespace: kube-system
spec:
  chart: traefik
  repo: https://containous.github.io/traefik-helm-chart
  set:
    image.tag: "2.2"

Then place this file in /var/lib/rancher/k3s/server/manifests and install or run the k3s server with --disable traefik.

[couillonnade]

I am not sure no-deploy is the right one, it is listed as deprecated in the documentation here

--disable value sounds the option you should use.

[mamiu]

@torpinouche You're absolutely right! Thanks, I've adapted my comment.

[SayakMukhopadhyay]

Hi,
I see that this issue has been added to the 1.20 backlog, does that mean that Traefik 2.0 integration is only coming with kubernetes 1.20 and wont be there in 1.18.x and 1.19.x?

[kvncrw]

o/ greetings from Containous. If you need any assistance with the transition to v2 of Traefik please don't hesitate to reach out to myself or anyone on the team. cc @SantoDE

[kbrowder]

@notsureifkevin fwict values content is pretty different in the contanious chart, specifically values that go into the toml in the traefik v1 chart aren't there (basically you need to bring your own toml). Am I understanding that right?

[kvncrw]

@kbrowder I believe your observation is correct, we are working to abstract some of these use cases into more helm-friendly values.yml approach, but for the purpose of k3s, either providing a toml or specifying the command line arguments through https://github.com/containous/traefik-helm-chart/blob/master/traefik/values.yaml#L118-L129 may be the best approach at the moment. cc @SantoDE

[YamacorePriv]

would love to see this coming sooner since traefik 2.0 is released since over a year now

[samirsss]

Bump on this one. We've run into an issue with the current installation of k3s (bundled traefik helm chart 1.81.0), only allows single external IP. The newer versions of Traefik charts support externalIPs as an array.

[SantoDE]

If you wanna tackle that @samirsss, I guess feel free.

We are happy to help :)

[mamiu]

We use Traefik 2.3 on a k3s multi node cluster in production for a few weeks now and it's working without any issues so far.

Fist I installed the k3s cluster with the --disable=traefik option. Here's a simplified version of our install command:

curl -sfL https://get.k3s.io | bash -s - \
  --write-kubeconfig-mode "0644" \
  --disable=traefik

Then I apply a file called traefik2.yaml (for whatever reason traefik.yaml didn't work - maybe because we disabled it in the first step and then k3s automatically removes traefik.yaml files or so). We use Traefik also for the automated certificate handling process but if you want to use other solutions like the cert-manager just delete all the certificate related lines. Since we have our domains on OVH the following example has ovh as a dns challange provider, but of course you can use one of all the other providers.

apiVersion: v1
kind: Namespace
metadata:
  name: traefik

---
apiVersion: v1
kind: Secret
metadata:
  name: ovh-credentials
  namespace: traefik
type: Opaque
data:
  OVH_APPLICATION_KEY: PoBhNa...     # the base64 encoded application key
  OVH_APPLICATION_SECRET: ReFGac...  # the base64 encoded application secret
  OVH_CONSUMER_KEY: YlaEW...         # the base64 encoded consumer key
  OVH_ENDPOINT: b3ZoLWV1

---
apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
  name: traefik
  namespace: kube-system
spec:
  repo: https://containous.github.io/traefik-helm-chart
  chart: traefik
  bootstrap: true
  targetNamespace: traefik
  valuesContent: |-

    additionalArguments:
      - "--certificatesresolvers.letsencrypt.acme.email=your.email@mail.com"
      - "--certificatesresolvers.letsencrypt.acme.storage=/data/acme.json"
      # Let's Encrypt staging API:
      - "--certificatesresolvers.letsencrypt.acme.caserver=https://acme-staging-v02.api.letsencrypt.org/directory"
      # Let's Encrypt production API:
      # - "--certificatesresolvers.letsencrypt.acme.caserver=https://acme-v02.api.letsencrypt.org/directory"
      - "--certificatesResolvers.letsencrypt.acme.dnschallenge=true"
      - "--certificatesResolvers.letsencrypt.acme.dnschallenge.provider=ovh"

      # If you want to use jaeger tracing for traefik uncomment these lines and adjust them to your jaeger installation
      # - "--tracing=true"
      # - "--tracing.serviceName=traefik"
      # - "--tracing.jaeger=true"
      # - "--tracing.jaeger.propagation=jaeger"
      # - "--tracing.jaeger.traceContextHeaderName=uber-trace-id"
      # - "--tracing.jaeger.samplingServerURL=http://jaeger-agent.jaeger.svc.cluster.local:5778/sampling"
      # - "--tracing.jaeger.localAgentHostPort=jaeger-agent.jaeger.svc.cluster.local:6831"
      # - "--tracing.jaeger.collector.endpoint=http://jaeger-collector.jaeger.svc.cluster.local:14268/api/traces?format=jaeger.thrift"

      # If you want to use prometheus monitoring for traefik uncomment these lines
      # - "--metrics=true"
      # - "--metrics.prometheus=true"
      # - "--metrics.prometheus.addServicesLabels=true"

      - "--accesslog=true"
      - "--log.level=INFO"

    persistence:
      enabled: true
      path: /data

    podAnnotations: 
      prometheus.io/port: '8082'
      prometheus.io/scrape: 'true'

    # We will apply our own ingress rule for the dashboard in the next step
    ingressRoute:
      dashboard:
        enabled: false

    envFrom:
      - secretRef:
          name: ovh-credentials

If you don't want to use Traefiks dashboard just ignore the next step.
Otherwise make sure that the Traefik installation is done and was successful. As soon as that's the case apply a file with the following content (adjusted to your needs):

apiVersion: v1
kind: Secret
metadata:
  name: admin-dashboard-auth
  namespace: traefik
data:
  # This is the base64 and openssl encoded version of "user" as user and "password" as password
  users: dXNlcjokYXByMSRxWVFuaS9FZSRqRHNNN3pnb1RoL01QeHJZTjlxVlEvCg==

---
apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  name: admin-auth
  namespace: traefik
spec:
  basicAuth:
    secret: admin-dashboard-auth

---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: traefik-dashboard
  namespace: traefik
spec:
  entryPoints:
    - websecure
  routes:
    - kind: Rule
      match: Host(`traefik.example.com`)
      services:
        - name: api@internal
          kind: TraefikService
      middlewares:
        - name: admin-auth
  tls:
    certResolver: letsencrypt
    domains:
      - main: "*.example.com"

Now Traefik 2.3 is installed in a k3s cluster with a dashboard reachable on traefik.example.com. If it doesn't work as expected please let me know because this is a highly simplified version of the one we're using and maybe I deleted something essential.

[ams0]

FWIW, I also used Traefik 2 with k3s/k3d (deployed with Helm with this script) as I wrote this article and it works just fine.

[Tautcius]

We use Traefik 2.3 on a k3s multi node cluster in production for a few weeks now and it's working without any issues so far.

Fist I installed the k3s cluster with the --disable=traefik option. Here's a simplified version of our install command:

curl -sfL https://get.k3s.io | bash -s - \
  --write-kubeconfig-mode "0644" \
  --disable=traefik

Then I apply a file called traefik2.yaml (for whatever reason traefik.yaml didn't work - maybe because we disabled it in the first step and then k3s automatically removes traefik.yaml files or so). We use Traefik also for the automated certificate handling process but if you want to use other solutions like the cert-manager just delete all the certificate related lines. Since we have our domains on OVH the following example has ovh as a dns challange provider, but of course you can use one of all the other providers.

apiVersion: v1
kind: Namespace
metadata:
  name: traefik

---
apiVersion: v1
kind: Secret
metadata:
  name: ovh-credentials
  namespace: traefik
type: Opaque
data:
  OVH_APPLICATION_KEY: PoBhNa...     # the base64 encoded application key
  OVH_APPLICATION_SECRET: ReFGac...  # the base64 encoded application secret
  OVH_CONSUMER_KEY: YlaEW...         # the base64 encoded consumer key
  OVH_ENDPOINT: b3ZoLWV1

---
apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
  name: traefik
  namespace: kube-system
spec:
  repo: https://containous.github.io/traefik-helm-chart
  chart: traefik
  bootstrap: true
  targetNamespace: traefik
  valuesContent: |-

    additionalArguments:
      - "--certificatesresolvers.letsencrypt.acme.email=your.email@mail.com"
      - "--certificatesresolvers.letsencrypt.acme.storage=/data/acme.json"
      # Let's Encrypt staging API:
      - "--certificatesresolvers.letsencrypt.acme.caserver=https://acme-staging-v02.api.letsencrypt.org/directory"
      # Let's Encrypt production API:
      # - "--certificatesresolvers.letsencrypt.acme.caserver=https://acme-v02.api.letsencrypt.org/directory"
      - "--certificatesResolvers.letsencrypt.acme.dnschallenge=true"
      - "--certificatesResolvers.letsencrypt.acme.dnschallenge.provider=ovh"

      # If you want to use jaeger tracing for traefik uncomment these lines and adjust them to your jaeger installation
      # - "--tracing=true"
      # - "--tracing.serviceName=traefik"
      # - "--tracing.jaeger=true"
      # - "--tracing.jaeger.propagation=jaeger"
      # - "--tracing.jaeger.traceContextHeaderName=uber-trace-id"
      # - "--tracing.jaeger.samplingServerURL=http://jaeger-agent.jaeger.svc.cluster.local:5778/sampling"
      # - "--tracing.jaeger.localAgentHostPort=jaeger-agent.jaeger.svc.cluster.local:6831"
      # - "--tracing.jaeger.collector.endpoint=http://jaeger-collector.jaeger.svc.cluster.local:14268/api/traces?format=jaeger.thrift"

      # If you want to use prometheus monitoring for traefik uncomment these lines
      # - "--metrics=true"
      # - "--metrics.prometheus=true"
      # - "--metrics.prometheus.addServicesLabels=true"

      - "--accesslog=true"
      - "--log.level=INFO"

    persistence:
      enabled: true
      path: /data

    podAnnotations: 
      prometheus.io/port: '8082'
      prometheus.io/scrape: 'true'

    # We will apply our own ingress rule for the dashboard in the next step
    ingressRoute:
      dashboard:
        enabled: false

    envFrom:
      - secretRef:
          name: ovh-credentials

If you don't want to use Traefiks dashboard just ignore the next step.
Otherwise make sure that the Traefik installation is done and was successful. As soon as that's the case apply a file with the following content (adjusted to your needs):

apiVersion: v1
kind: Secret
metadata:
  name: admin-dashboard-auth
  namespace: traefik
data:
  # This is the base64 and openssl encoded version of "user" as user and "password" as password
  users: dXNlcjokYXByMSRxWVFuaS9FZSRqRHNNN3pnb1RoL01QeHJZTjlxVlEvCg==

---
apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  name: admin-auth
  namespace: traefik
spec:
  basicAuth:
    secret: admin-dashboard-auth

---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: traefik-dashboard
  namespace: traefik
spec:
  entryPoints:
    - websecure
  routes:
    - kind: Rule
      match: Host(`traefik.example.com`)
      services:
        - name: api@internal
          kind: TraefikService
      middlewares:
        - name: admin-auth
  tls:
    certResolver: letsencrypt
    domains:
      - main: "*.example.com"

Now Traefik 2.3 is installed in a k3s cluster with a dashboard reachable on traefik.example.com. If it doesn't work as expected please let me know because this is a highly simplified version of the one we're using and maybe I deleted something essential.

not working for me...

[zdzichu]

@Thalooka Traefik 2 was merged in 669d0c0 this week.

[rancher-max]

Validated in v1.21.0-rc1+k3s1

• Ingress related checks still continue to pass
• In an upgraded setup, traefik has NOT been upgraded as expected
• disabling still works as expected, and it remains disabled after an upgrade
• Validated some of the v2 functionality w.r.t Middlewares and IngressRoutes

[boindil]

Validated in v1.21.0-rc1+k3s1

• Ingress related checks still continue to pass
• In an upgraded setup, traefik has NOT been upgraded as expected
• disabling still works as expected, and it remains disabled after an upgrade
• Validated some of the v2 functionality w.r.t Middlewares and IngressRoutes

Hi rancher-max, is there any documentation / way on how to upgrade to traefik 2.0?

I have Rancher running and a frew other ingresses. I dont care to recreate these (it would be nice to have rancher running however).

[rancher-max]

Hi @boindil, our docs should be getting merged for this soon, but you can take a look at rancher/docs#3021 which should have the steps linked there.

[tbcdns]

Hi @rancher-max, thanks for the pointer, however, if I understood correctly, the docs mention mainly how to migrate the manifests from Ingress to IngressRoute.
But I have k3s v1.21.5+k3s2 running still with Traefik v1, what are the steps to upgrade the ingress controller to Traefik v2.
Thanks for the help!

[rancher-max]

If I understand correctly, you're looking for how to upgrade traefik itself from v1 to v2, and then will handle all of the breaking changes?

K3s doesn't provide a built-in method of doing that for you since there are so many breaking changes with this, so we would recommend you use the Traefik Migration Tool and follow Traefik's migration docs. K3s uses helm to install traefik, so you can potentially use helm to upgrade it, although the Traefik docs mention that the helm chart for traefik v2 is not yet stable, so proceed with caution.

[camaeel]

According to https://rancher.com/docs/k3s/latest/en/networking/#traefik-ingress-controller k3s installer will install traefik v2 when v1 is not installed and your cluster version is >= 1.21.

This is how (more or less) I upgraded traefik

1. Remove old traefik

   1. Helm chart from master nodes: /var/lib/rancher/k3s/server/static/charts/traefik*
   2. Helm chart CRD from master nodes: /var/lib/rancher/k3s/server/manifests/traefik*
   3. Uninstall traefik helm chart helm uninstall traefik -n kube-system

2. Reinstall k3s with the same settings. It should detect that trafik is missing but not disabled and it should upgrade it to version 2.

3. Optionaly create HelmChartConfig file under /var/lib/rancher/k3s/server/manifests/traefik-config.yaml with your custom config overrides. An example (I enabled ingress class, made it default, disabled anonymous reporting and setup Let's Encrpt with cloudflare's DNS challenge) jinja2 template (used by ansible):

   apiVersion: helm.cattle.io/v1
   kind: HelmChartConfig
   metadata:
     name: traefik
     namespace: kube-system
   spec:
     valuesContent: |-
       ingressClass:
         enabled: true
         isDefaultClass: true
       ingressRoute:
         dashboard:
           enabled: false    
       globalArguments:
         - "--global.checknewversion"
         - "--global.sendanonymoususage=false"          
       logs:
         general:
           level: INFO
       ports:
         websecure:
           tls:
             enabled: true
   {% if traefik_letsencrypt_cloud_domain is defined %}
             certResolver: le
             domains:
             - main: "*.{{ traefik_letsencrypt_cloud_domain }}"
               sans: "{{ traefik_letsencrypt_cloud_domain }}"
       envFrom:
       - secretRef:
           name: {{ cloudflare_api_token_secret_name }}
       persistence:
         enabled: true
         storageClass: {{ traefik_certs_storageclass | default('local-path') }}
       additionalArguments:
       - "--certificatesresolvers.le.acme.storage=/data/acme.json"
       - "--certificatesresolvers.le.acme.email={{ traefik_letsencrypt_email }}"
       - "--certificatesresolvers.le.acme.storage=/data/acme.json"
       - "--certificatesresolvers.le.acme.caserver=https://acme-v02.api.letsencrypt.org/directory"
       - "--certificatesResolvers.le.acme.dnschallenge=true"
       - "--certificatesResolvers.le.acme.dnschallenge.provider=cloudflare"
     
   

[tbcdns]

Thanks!
Yes that's what I wanted to do, upgrade Traefik itself to v2.

I followed the steps proposed by @camaeel, for removing the old Traefik, I simply deleted the helmchart:
kubectl -n kube-system delete helmchart traefik
I also removed the charts from the master nodes and the current Traefik manifest.

After relauching the install script, Traefik v2 with the CRD was properly installed.
I then used the Traefik migration tool to migrate my ingresses, everything work as expected!

Thanks!

[brandond]

On a single node cluster where you haven't customized the Traefik configuration at all you can just:

1. restart k3s with --disable=traefik
2. restart k3s again without --disable=traefik

The same will obviously work on multi-node clusters, you just have to do each step on all the servers before moving on to the next.

[zoulux]

@r1cebank Create a yaml file with following content:

apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
  name: traefik
  namespace: kube-system
spec:
  chart: traefik
  repo: https://containous.github.io/traefik-helm-chart
  set:
    image.tag: "2.2"

Then place this file in /var/lib/rancher/k3s/server/manifests and install or run the k3s server with --disable traefik.

systemctl restart k3s is work!