-
-
Notifications
You must be signed in to change notification settings - Fork 45
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[INFRA-2743] add traefik as a secondary ingress controller #867
Conversation
So I have the same question the last time an attempt was made. Why switch away from nginx that is working, to something that might work? Is there a problem that the nginx ingress isn't solving that traefik is? Is there a discussion somwhere? |
config/default/traefik.yaml
Outdated
minVersion: VersionTLS12 | ||
cipherSuites: | ||
- "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256" | ||
- "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256" | ||
- "TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384" | ||
- "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384" | ||
- "TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305" | ||
- "TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@halkeye based on @olblak comment in https://issues.jenkins.io/browse/INFRA-2743 There are some benefits to traefik such as being able to support gRPC and web on the same https port. Another benefit would be the ability to route TCP and UDP over the same loadbalancer without workarounds once kubernetes/kubernetes#94028 lands. @olblak could hopefully attest to some of the reasons, I suspect ingressroute is one reason. Nothing is preventing us from keeping nginx, we could choose to only enable the ingressroute for traefik if that is the feature that @olblak is after :) Another thing that is superior in traefik is the ability to add endless middleware to both k8s ingress and ingressroute. For my prod cluster another reason is the addition of https://github.com/traefik/mesh-helm-chart Another plan that I have for my prod cluster is to setup VPN using wireguard have the http frontend and UDP port using the same traefik ingressroute on the same hostname 👏 Here is a working example with ingressroute for argocd and using cert-manager and external-dns. where traefik is hostname is set using external dns: traefik:
service:
annotations:
external-dns.alpha.kubernetes.io/hostname: private-traefik.company.io kind: Service
apiVersion: v1
metadata:
name: argocd-external-name
annotations:
external-dns.alpha.kubernetes.io/hostname: argocd.company.io
spec:
type: ExternalName
externalName: private-traefik.company.io apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: argocd.company.io-tls
spec:
dnsNames:
- argocd.company.io
issuerRef:
group: cert-manager.io
kind: ClusterIssuer
name: letsencrypt-prod
secretName: argocd.company.io-tls apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
name: argocd-server
namespace: argocd
annotations:
kubernetes.io/ingress.class: traefik
spec:
entryPoints:
- websecure
routes:
- kind: Rule
match: Host(`argocd.company.io`)
priority: 10
services:
- name: argocd-server
port: 80
- kind: Rule
match: Host(`argocd.company.io`) && Headers(`Content-Type`, `application/grpc`)
priority: 11
services:
- name: argocd-server
port: 80
scheme: h2c
tls:
secretName: argocd.company.io-tls |
if anything the current use of stable/nginx-ingress should be switched to new chart: https://github.com/kubernetes/ingress-nginx/tree/master/charts/ingress-nginx |
config/default/traefik.yaml
Is that done? does any existing annotations need to be updated? do you have any checklists or anything? |
Not a checklist per say, one annoyance I found was traefik/traefik#7912 https://doc.traefik.io/traefik/master/routing/providers/kubernetes-ingress/#annotations |
better traefik support in oauth2-proxy was recently added: oauth2-proxy/oauth2-proxy#957 |
While I agree that Nginx does the work, Traefik offers more advanced configuration and more analytics. So I am definitely interested |
oh wow, this was a pain for us at work, great to know it's solved |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for your contribution 👍
Could you check the feedback from this first pass of review?
If we have specific nginx annotation, we have to be careful to understand what it is doing: even without switching to Traefik, it would be a valuable self-documentation task. Also, note that we should ensure that the Traefik's CRD provider is enabled to ensure that the advanced/native features can be used (as the default Ingress Controller provider is very limited in term of features in Traefik). It should be the case by default |
=> Please note that the We might need to do a separate PR to be applied first, with only the CRDs to install initially to bootstrap (or do it manually 1 time). |
Co-authored-by: Damien Duportal <damien.duportal@gmail.com>
We could use the presync event to load the crds, see https://github.com/zakkg3/cert-manager-installer for a helmfile example of how to load the crds. These guys used python to load the versioned CRD: https://github.com/cloudposse/helmfiles/tree/master/releases/cert-manager |
Thanks for the changes @jetersen!
So the problem is not about installing CRDs, but bootstraping the initial env, as the helmfile hooks are not applied when the Thanks for the idea though, it could have been a great use! If it is OK for you, I'll check with @olblak and the rest of the team to preinstall the CRDs if it is OK, and we'll ping you here (+ relaunch the CI job that should pass the |
References: - https://issues.jenkins.io/browse/INFRA-2743 - jenkins-infra/kubernetes-management#867 Signed-off-by: Damien Duportal <damien.duportal@gmail.com>
Signed-off-by: Damien Duportal <damien.duportal@gmail.com>
Worklog:
|
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM 👍
|
For information, the master branch is failing with this PR merged with the following error message:
WiP on fixing this, upcoming PR |
@dduportal sorry my bad 😓 |
Oh don't be sorry for this, we all reviewed and decided to merge. You did the heavy lifting and it really is nice! |
This adds traefik currently there is no way to scrape data with prometheus.
I tried to map everything that the nginx controllers had.
For default backend, the option to add another container and add error frontend see the comments here: traefik/traefik#4218
Perhaps this should be subcharted to allow for a way to add custom middleware and ingressroute at the global level or better yet create a separate chart for those global things.