[INFRA-2743] add traefik as a secondary ingress controller #867
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
So I have the same question the last time an attempt was made. Why switch away from nginx that is working, to something that might work? Is there a problem that the nginx ingress isn't solving that traefik is? Is there a discussion somwhere? |
jetersen
commented
Feb 19, 2021
config/default/traefik.yaml
Outdated
Comment on lines
18
to
25
| minVersion: VersionTLS12 | ||
| cipherSuites: | ||
| - "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256" | ||
| - "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256" | ||
| - "TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384" | ||
| - "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384" | ||
| - "TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305" | ||
| - "TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305" |
There was a problem hiding this comment.
|
@halkeye based on @olblak comment in https://issues.jenkins.io/browse/INFRA-2743 There are some benefits to traefik such as being able to support gRPC and web on the same https port. Another benefit would be the ability to route TCP and UDP over the same loadbalancer without workarounds once kubernetes/kubernetes#94028 lands. @olblak could hopefully attest to some of the reasons, I suspect ingressroute is one reason. Nothing is preventing us from keeping nginx, we could choose to only enable the ingressroute for traefik if that is the feature that @olblak is after :) Another thing that is superior in traefik is the ability to add endless middleware to both k8s ingress and ingressroute. For my prod cluster another reason is the addition of https://github.com/traefik/mesh-helm-chart Another plan that I have for my prod cluster is to setup VPN using wireguard have the http frontend and UDP port using the same traefik ingressroute on the same hostname 👏 Here is a working example with ingressroute for argocd and using cert-manager and external-dns. where traefik is hostname is set using external dns: traefik:
service:
annotations:
external-dns.alpha.kubernetes.io/hostname: private-traefik.company.iokind: Service
apiVersion: v1
metadata:
name: argocd-external-name
annotations:
external-dns.alpha.kubernetes.io/hostname: argocd.company.io
spec:
type: ExternalName
externalName: private-traefik.company.ioapiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: argocd.company.io-tls
spec:
dnsNames:
- argocd.company.io
issuerRef:
group: cert-manager.io
kind: ClusterIssuer
name: letsencrypt-prod
secretName: argocd.company.io-tlsapiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
name: argocd-server
namespace: argocd
annotations:
kubernetes.io/ingress.class: traefik
spec:
entryPoints:
- websecure
routes:
- kind: Rule
match: Host(`argocd.company.io`)
priority: 10
services:
- name: argocd-server
port: 80
- kind: Rule
match: Host(`argocd.company.io`) && Headers(`Content-Type`, `application/grpc`)
priority: 11
services:
- name: argocd-server
port: 80
scheme: h2c
tls:
secretName: argocd.company.io-tls |
|
if anything the current use of stable/nginx-ingress should be switched to new chart: https://github.com/kubernetes/ingress-nginx/tree/master/charts/ingress-nginx |
halkeye
changed the title
config/default/traefik.yaml
Is that done? does any existing annotations need to be updated? do you have any checklists or anything? |
|
Not a checklist per say, one annoyance I found was traefik/traefik#7912 https://doc.traefik.io/traefik/master/routing/providers/kubernetes-ingress/#annotations |
|
better traefik support in oauth2-proxy was recently added: oauth2-proxy/oauth2-proxy#957 |
|
While I agree that Nginx does the work, Traefik offers more advanced configuration and more analytics. So I am definitely interested |
oh wow, this was a pain for us at work, great to know it's solved |
dduportal
requested changes
Feb 19, 2021
If we have specific nginx annotation, we have to be careful to understand what it is doing: even without switching to Traefik, it would be a valuable self-documentation task. Also, note that we should ensure that the Traefik's CRD provider is enabled to ensure that the advanced/native features can be used (as the default Ingress Controller provider is very limited in term of features in Traefik). It should be the case by default |
timja
reviewed
Feb 19, 2021
|
=> Please note that the We might need to do a separate PR to be applied first, with only the CRDs to install initially to bootstrap (or do it manually 1 time). |
Co-authored-by: Damien Duportal <damien.duportal@gmail.com>
|
We could use the presync event to load the crds, see https://github.com/zakkg3/cert-manager-installer for a helmfile example of how to load the crds. These guys used python to load the versioned CRD: https://github.com/cloudposse/helmfiles/tree/master/releases/cert-manager |
|
Thanks for the changes @jetersen!
So the problem is not about installing CRDs, but bootstraping the initial env, as the helmfile hooks are not applied when the Thanks for the idea though, it could have been a great use! If it is OK for you, I'll check with @olblak and the rest of the team to preinstall the CRDs if it is OK, and we'll ping you here (+ relaunch the CI job that should pass the |
jetersen
commented
Feb 19, 2021
dduportal
added a commit
to dduportal/azure
that referenced
this pull request
Feb 23, 2021
References: - https://issues.jenkins.io/browse/INFRA-2743 - jenkins-infra/kubernetes-management#867 Signed-off-by: Damien Duportal <damien.duportal@gmail.com>
Signed-off-by: Damien Duportal <damien.duportal@gmail.com>
|
Worklog:
|
|
|
markjacksonfishing
approved these changes
Feb 25, 2021
|
For information, the master branch is failing with this PR merged with the following error message: WiP on fixing this, upcoming PR |
|
@dduportal sorry my bad 😓 |
|
Oh don't be sorry for this, we all reviewed and decided to merge. You did the heavy lifting and it really is nice! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This adds traefik currently there is no way to scrape data with prometheus.
I tried to map everything that the nginx controllers had.
For default backend, the option to add another container and add error frontend see the comments here: traefik/traefik#4218
Perhaps this should be subcharted to allow for a way to add custom middleware and ingressroute at the global level or better yet create a separate chart for those global things.