Where to terminate TLS connections?

[MartiniMoe]

Hi,

I really appreciate your work and have successfully created my own k8s cluster on the Hetzner cloud :)

Now I wanted to add TLS / HTTPS support and wanted to let the TLS connection terminate on the loadbalancer. Automatically retrieved certificates from letsencrypt seem fine. However the loadbalancer does not seem to work when I change its service

from
"[tcp] 443 -> 31028"
to
"[https] 443 -> 30468"

I have completely removed the tcp service for port 80, because I think I will not need it.
The loadbalancer shows 'unhealthy' for this service and I cannot access any ingress anymore.

Can someone please advise me on how to achieve TLS support with the hetzner loadbalancer and traefik ingress? :) Thanks!

[mysticaltech]

What I do personally is "outsource" SSL to Cloudflare. Will explain exactly how.

[mysticaltech]

1- Go to Cloudflare, login to your account, under SSL, create "origin certificates". Those are static certificates that are valid for 15 years.

[mysticaltech]

2- Save the certificates to two files:

origin-ca.crt

-----BEGIN CERTIFICATE-----
-----BEGIN CERTIFICATE-----

origin-ca.pk

-----BEGIN PRIVATE KEY-----
-----END PRIVATE KEY-----

[mysticaltech]

3- Add these certificates to Kube with:

kubectl create secret tls cloudflare-tls --key origin-ca.pk --cert origin-ca.crt

[mysticaltech]

4- In your ingress manifest and add the following spec:

spec:
  tls:
  - hosts:
    - test.yourdomain.dev
    secretName: cloudflare-tls

[mysticaltech]

5- By now when you hit test.yourdomain.dev, you'll be served with Cloudflare static SSL that you have saved to your cluster. But that is not what we want, that will just serve to secure the communication to Cloudflare itself.

Go to Cloudflare SSL again, and choose the following setup SSL/TLS strict:

[mysticaltech]

6- Then push the orange cloud icon, on your domain, in the Cloudflare DNS zone. That way you'll have amazing SSL and even more Cloudflare goodies.

[mysticaltech]

7- (Optional) Couple that with external-dns, and you'll be deploying like a boss, without even touching Cloudflare ever again.

Do deploy it, use:

helm install external-dns external-dns/external-dns -f external-dns.values.yaml -n kube-system

With this external-dns.values.yaml file:

image:
  registry: docker.io
  repository: bitnami/external-dns
  tag: latest
  pullPolicy: Always
env:
  - name: CF_API_KEY
    value: "YOU-CF-API-KEY-HERE"
  - name: CF_API_EMAIL
    value: "me@email.com"
sources:
  - ingress
  - service
policy: sync
provider: cloudflare
logLevel: debug% 

Then you can have your ingresses in the following format, and it will create the DNS records automatically:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: example-ingress
  annotations:
    "external-dns.alpha.kubernetes.io/hostname": "test.yourdomain.dev"
    "external-dns.alpha.kubernetes.io/cloudflare-proxied": "true"
spec:
  tls:
  - hosts:
    - test.mystical.dev
    secretName: cloudflare-tls
  rules:
    - host: test.yourdomain.dev
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: apple-service
                port:
                  number: 5678%       

[mysticaltech]

Basically, @MartiniMoe, when you have the above setup once, that's it, the rest just flows automatically, and you do not ever need to configure anything on Hetzner's side, it's all done in the ingress manifest like in the above example. Good luck!

[MartiniMoe]

Hi, thank you very much for your detailed explanation!
How comes the certificate is available as a secret in kubernetes at step 4?
I have added my certificate to hetzner cloud via the web ui and I do not have it as a secret in kubernetes.

[mysticaltech]

@MartiniMoe You never need to touch the load balancer manually. You let Traefik do its own thing. It configures the LB with the proxy protocol for maximum flexibility. The SSL is added in Kubernetes and shows up automatically on the LB when port 443 is hit.

Again, everything just works out of the box, you just need to think about feeding the Ingress with a certificate of your choice, either from Let's Encrypt, Traefik can do that, or Cloudflare like I did above.

[mysticaltech]

Closing this for now @MartiniMoe, do not hesitate to ask questions if you still have some about SSL, but it really is as simple as described above. No manual intervention is needed on the LB. The only LB boss in town is Traefik.

[MartiniMoe]

OK, now I understand!
Awesome, thanks! Then I will look into Traefik configuration for Lets Encryp :)

[mysticaltech]

Exactly @MartiniMoe, and don't forget Traefik on k3s can be configured via this HelmChartConfig, you can recreate this file, add the acme resolver info for instance like shown here and here, then apply it, and it should get you well on your way of making this work!

Just make sure to keep the name of the HelmChartConfig the same so it overwrites the previous config that was applied during install:

metadata:
  name: traefik
  namespace: kube-system

[MartiniMoe]

Thanks!

At first I wondered why terraform apply does not recreate the traefik deployment with the added certificate resolver, but then I figured out, I can just kubectl apply the rendered HelmChartConfig after running terraform apply.

I already got it to work with my main domain, but not with subdomains 🤔 But maybe thats a DNS issue on my side, I will have to look into it 😄

[mysticaltech]

Great to hear, please don't hesitate to post your HelmChartConfig just to help others that would want to do the same in the future.

About subdomains, you have to make sure you request for them with wildcard SSL in the form of *.yourdomain.dev, if already done, then indeed, must be a DNS thing.

[MartiniMoe]

Sure, my HelmChartConfig with certificateresolver looks like this:

apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: traefik
  namespace: kube-system
spec:
  valuesContent: |-
    service:
      enabled: true
      type: LoadBalancer
      annotations:
        "load-balancer.hetzner.cloud/name": "traefik"
        "load-balancer.hetzner.cloud/use-private-ip": "true"
        "load-balancer.hetzner.cloud/location": "${location}"
        "load-balancer.hetzner.cloud/type": "${lb_server_type}"
        "load-balancer.hetzner.cloud/uses-proxyprotocol": "true"
    additionalArguments:
      - "--entryPoints.web.proxyProtocol.trustedIPs=127.0.0.1/32,10.0.0.0/8"
      - "--entryPoints.websecure.proxyProtocol.trustedIPs=127.0.0.1/32,10.0.0.0/8"
      - "--entryPoints.web.forwardedHeaders.trustedIPs=127.0.0.1/32,10.0.0.0/8"
      - "--entryPoints.websecure.forwardedHeaders.trustedIPs=127.0.0.1/32,10.0.0.0/8"
      - "--certificatesresolvers.letsencrypt.acme.email=<MY-EMAIL-ADDRESS>"
      - "--certificatesresolvers.letsencrypt.acme.storage=/data/acme.json"
      - "--certificatesresolvers.letsencrypt.acme.caserver=https://acme-staging-v02.api.letsencrypt.org/directory"
      - "--certificatesresolvers.letsencrypt.acme.tlschallenge=true"

Note that this is using LetsEncrypts staging certificates to not run into rate limiting while testing. For production one should remove the caserver line.

For subdomains I was thinking about letting Traefik get a seperate certificate for each subdomain. Why do you think this wont work and I would need a wildcard certificate? Or did you mention the wildcard certificate because it is easier?

[mysticaltech]

Great, thanks for sharing @MartiniMoe!

About the wildcard, I just thought it's easier, but maybe I'm mistaken (never used it on Kube) because I can read it needs a DNS challenge to work.

[phaer]

@mysticaltech I tried to replicate your setup with external-dns, but for some reason, my ingress seems to inherit the private ip from my load-balancer as well and so its end up in DNS.

kubectl get ing
NAME           CLASS    HOSTS                ADDRESS                                      PORTS     AGE
test-ingress   <none>   test.my-domain.dev   10.0.0.6,2a01:4f8:c011:501::1,49.12.16.189   80, 443   13m

I would expect it to only expose the public ips.

Is that the same for you? It could be that it works for you because you enable cloudflare proxying and i think that filters private ips by default, but it might be buggy if one is using cloudflares dns only?

EDIT: That seems to be the case. With cloudflare-proxied set to true, it works fine. Without it, it's using both the public and the private ipv4 address and ignores the ipv6.

[phaer]

I've got it to work without cloudflare proxying & ipv4 only, by adding the following two annotations to ./templates/traefik_config.yaml.tpl

        "load-balancer.hetzner.cloud/ipv6-disabled": "true"
        "load-balancer.hetzner.cloud/disable-private-ingress": "true"

I disable ipv6 because hetzner-ccm recommends doing so while using external-dns, but there's an open issue in external-dns to solve that.

[mysticaltech]

Thanks for sharing, great info @phaer!

[mysticaltech]

@MartiniMoe FYI, not sure it's even relevant, but just wanted you to know that I had made a mistake at step 3/ of the tutorial above, I had pasted the wrong command with hcloud certificate create now rectified here with kubectl create secret tls.

[MartiniMoe]

@MartiniMoe FYI, not sure it's even relevant, but just wanted you to know that I had made a mistake at step 3/ of the tutorial above, I had pasted the wrong command with hcloud certificate create now rectified here with kubectl create secret tls.

Thanks for letting me know! I was wondering how your certificate ended up as a secret in kubernetes :D

[kiloOhm]

This was super helpful! @mysticaltech Maybe link to this in the Readme unter the Examples section