Support non-GKE clusters in Helm

[djmitche]

TC's Helm configuration currently assumes it is running in GKE, and in particular that it is using a Google ingress controller, the configuration for which is different from that for nginx. Let's make it possible to use either Google or non-Google ingress controllers.

[ricky26]

I'd very much like to use the Traefik for our ingress too so this is interesting to me. 😄

fwiw it should just be a case of allowing people to customise the annotations that are put on the ingress (and perhaps deferring the GCP specific ones to be in your deployments' values.yml). We don't use pre-provisioned HTTPS certificates, so we already have to override some of these in https://github.com/wellplayedgames/taskcluster-operator.

[djmitche]

It seems like we have a large-scale choice: either continue to ship Helm charts, or replace the helm chart with a controller.

Helm Chart

• Requires some external coordination -- creating DB accounts and Pulse accounts, among other things (stuff handled by yarn dev:init)
• Currently makes some assumptions about deployment -- using the GLB Ingress controller, and using pre-provisioned certs. I think these assumptions could be handled with some more Helm values and conditional expressions.
• It might make sense to actually publish these to Artifact Hub -- maybe with a bit more polish first

Controller

• A controller could handle much of the necessary external coordination -- holding DB superuser credentials, pulse admin credentials, and so on, and taking care of the things currently handled by yarn dev:init.
• A controller could also handle DB migrations, although this is probably a bit more complex

---

So I think the question to answer here is, should we try to continue to improve the chart approach, or switch to controller? @oremj / @sciurus your input would be useful too :)

[sciurus]

@djmitche did you mean to say "operator" instead of controller?

My hot take is that it's much more realistic to have the helm chart support bringing your own ingress than it is to switch to an operator. In fact, we are already using a hack to skip the ingress you ship and inject our own for Mozilla's taskcluster deployment.

[djmitche]

@djmitche did you mean to say "operator" instead of controller?

I did, sorry! I'll leave it as-is for history :)

I agree that's more realistic. But it's getting increasingly tempting to implement the "external coordination" in a more general way, so maybe setting aside the realism of resources available within Mozilla (which basically means "don't change anything"), an operator is a good thing to consider.

[lotas]

I've addressed some of those concerns in #5613.

As ingress.yml is autogenerated, it is almost impossible to introduce helm templates features like flow control, to conditionally include or exclude parts of it. Instead, I've added few annotations and couple of new config values.

This allows to use ingress-nginx and certbot through annotations. And also generate proper routes for nginx