Deploy the support chart to meom-ige

[GeorgianaElena]

• This adds config for deploying the support chart for meom-ige
• Enables prometheus authentication
• Removes the LoadBalancer + autohttps config since we'll be using nginx-ingress

The following manual steps are required:

• deploy the support chart with:
  python3 deployer deploy-support meom-ige
• retrieve the external IP address for the nginx-ingress load balancer.
  kubectl --namespace support get svc support-ingress-nginx-controller
• point the dns record to this ip instead of proxy-public
• delete the proxy-public service
• do a deploy of the cluster
  python3 deployer deploy meom-ige
• merge this PR

@2i2c-org/tech-team, can you please confirm if the manual steps mentioned above ⬆️ are ok please?

Ref: #1278 and #594

[sgibson91]

Manual steps also sound reasonable to me! 🙌🏻

[GeorgianaElena]

@2i2c-org/tech-team, note that I plan to run the manual steps today, as soon as there is no user on the hub.

@yuvipanda, I would really appreciate it if you could confirm whether the resource limits I chose for prometheus are sensible enough for this cluster 👀

[GeorgianaElena]

@2i2c-org/tech-team, I believe I need some help debugging an issue during the support chart deploy.

The support-ingress-nginx-admission-create is hanging and I see this events:

 Normal   Scheduled               11m    gke.io/optimize-utilization-scheduler  Successfully assigned support/support-ingress-nginx-admission-create-xx5r7 to gke-meom-ige-cluster-core-pool-d987f2c6-jw1c
  Warning  FailedCreatePodSandBox  11m    kubelet                                Failed to create pod sandbox: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial unix /run/containerd/containerd.sock: connect: no such file or directory"
  Normal   Pulled                  8m17s  kubelet                                Container image "docker.io/jettech/kube-webhook-certgen:v1.5.1" already present on machine

[yuvipanda]

@GeorgianaElena can you try deleting the pod and trying again? I've seen similar issues caused by race conditions go away on restart.

[GeorgianaElena]

Deleted that pod then ran the deploy again and now I'm getting a x509: certificate signed by unknown authority
from validate.nginx.ingress.kubernetes.io

[yuvipanda]

@GeorgianaElena delete all pods in the support namespace? 😁

[choldgraf]

Cattle not pets!

[GeorgianaElena]

Deleted all, then ran the deploy-support again and same error. I'm pasting its full text here:

Error: UPGRADE FAILED: cannot patch "support-prometheus-server" with kind Ingress: Internal error occurred: failed calling webhook "validate.nginx.ingress.kubernetes.io": Post "https://support-ingress-nginx-controller-admission.support.svc:443/networking/v1beta1/ingresses?timeout=10s": x509: certificate signed by unknown authority
Traceback (most recent call last):

I'm keep seeing solutions online to delete the support-ingress-nginx-admission webhook. But there has to be a better way?

[yuvipanda]

@GeorgianaElena if you delete the hook, helm should just recreate it next time, right? It's a job so might work ok.

[GeorgianaElena]

Deleting the hook did the trick! I feel like I still need to wrap my head around how first approach in solving something in k8s world, most of the times seem to be to delete things until they come up healthy and until a certain dependency order is achieved 🤯

Thanks a lot for the help @yuvipanda <3

[choldgraf]

For what it's worth @GeorgianaElena my strategy for debugging literally any issue in binder when we were first operating it was "Something wrong? Delete everything in kubernetes except for the helm installation" 😂

[GeorgianaElena]

This is now done, so I'll merge it and then add this clustrer to the central grafana.

[GeorgianaElena]

🚀 Thanks everybody

[github-actions]

🎉🎉🎉🎉

Monitor the deployment of the hubs here 👉 https://github.com/2i2c-org/infrastructure/actions/workflows/deploy-hubs.yaml?query=branch%3Amaster