Use nginx-ingress as entrypoint for all hubs

[yuvipanda]

For some hubs that don't have the support chart deployed (#388), we use individual LoadBalancer IPs to expose the hubs. This prevents easily exposing something like grafana, and also adds complexity - some hubs use LoadBalancer, and some use nginx-ingress.

We should move everything to nginx-ingress, and new hubs should all use the nginx-ingress entrypoint.

We need to move the following hubs:

• Farallon
• Openscapes
• Meom-IGE
• Pangeo-181919 (COESSING hub I think)

We can do so by:

1. Changing TTL of current DNS entry for a hub to 1minute, so changes can propagate faster. This will take some time to take effect - as long as the current TTL (often 5 minutes) of the record.
2. Change the record, pointing it to the IP / CNAME of the nginx-ingress load balancer service in the support/ namespace
3. Set proxy.https.enabled to false for the hub, and unset proxy.service.type. Do a deploy immediately. The autohttps deployment should go away, and proxy-public should change type. You might have to delete the proxy-public service before doing a deploy, or the deploy might fail, since it'll try to change the service from a LoadBalancer to a ClusterIP (and this is often not fully supported)
4. Make this change in a PR and merge it

This might lead to a minute or two of downtime, so has to be done at a time that'll hopefully affect nobody.

[damianavila]

We should move everything to nginx-ingress, and new hubs should all use the nginx-ingress entrypoint.

So this means we are deploying support in all our clusters, right? I would agree with that.

[choldgraf]

I'm setting this as medium priority and adding to our backlog now...these kinds of "standardize and simplify our infrastructure" issues seem particularly important to make sure that we can iterate quickly and without confusing ourselves in the future

[damianavila]

AFAIK, we have support deployed in all of our clusters.
In fact, recently #1091 was merged... which is using nginx-ingress to expose Prometheus with basic auth.
@yuvipanda, can you confirm somehow this was already done for openscapes (I guess when it was transitioned to use eksctl) and the only remaining ones are meom-ige and farallon? (Pangeo-181919 does not exist anymore, IIRC).

[yuvipanda]

This is done now, thanks to @GeorgianaElena!

[damianavila]

For future readers, Farallon is being decommissioned here: #1454.