[Feature] replace helm-rollout-restarter method with helm best practice to restart pods on configmap changes

[jkleinlercher]

Problem Statement

pods should get restarted during deployment when configmaps change. However, the current approach with "helm-rollout-restarter" always restarts the pods when "helm template" is running which results in a restart whenever argocd recognizes git commits in the gitops repo. There is a feature to disable helm-rollout-restarter introduced with #3324. however, then pods never ever get restarted when configmaps change.

Solution Description

Use the helm best practice https://helm.sh/docs/howto/charts_tips_and_tricks/#automatically-roll-deployments which is also used in loki: https://github.com/grafana/loki/blob/5ada92b190c671055bb09ca2dd234b6bac49289e/production/helm/loki/templates/admin-api/deployment-admin-api.yaml#L35

Alternatives

No response

Additional Context

No response

Troubleshooting

• I have read and followed the issue guidelines and this is a feature request only for the Helm chart.
• I have searched other issues in this repository and mine is not recorded.

[mmclane]

I turned on debug logging on the cluster-controller. Here is the log from my most recent attempt to get this to work.
This is after the cluster-controller has created the singleton nodepool and moved itself onto that node.

2024-05-17T19:48:22Z INF Determined to be running in a cluster. Using in-cluster K8s config.
2024-05-17T19:48:24Z DBG Backing off startup IsAvailable() check for recommendation service. error="failed to execute request: Get \"http://kubecost-cost-analyzer.kubecost:9090/model/savings/requestSizingV2\": context deadline exceeded" dur=398.625271ms
2024-05-17T19:48:26Z DBG Backing off startup IsAvailable() check for recommendation service. error="failed to execute request: Get \"http://kubecost-cost-analyzer.kubecost:9090/model/savings/requestSizingV2\": context deadline exceeded" dur=848.536205ms
2024-05-17T19:48:29Z DBG Backing off startup IsAvailable() check for recommendation service. error="failed to execute request: Get \"http://kubecost-cost-analyzer.kubecost:9090/model/savings/requestSizingV2\": context deadline exceeded" dur=1.155696389s
2024-05-17T19:48:32Z DBG Backing off startup IsAvailable() check for recommendation service. error="failed to execute request: Get \"http://kubecost-cost-analyzer.kubecost:9090/model/savings/requestSizingV2\": context deadline exceeded" dur=2.068513865s
2024-05-17T19:48:37Z DBG Backing off startup IsAvailable() check for recommendation service. error="failed to execute request: Get \"http://kubecost-cost-analyzer.kubecost:9090/model/savings/requestSizingV2\": context deadline exceeded" dur=2.828424303s
2024-05-17T19:48:41Z DBG Backing off startup IsAvailable() check for recommendation service. error="failed to execute request: Get \"http://kubecost-cost-analyzer.kubecost:9090/model/savings/requestSizingV2\": context deadline exceeded" dur=4.950030515s
2024-05-17T19:48:48Z DBG Backing off startup IsAvailable() check for recommendation service. error="failed to execute request: Get \"http://kubecost-cost-analyzer.kubecost:9090/model/savings/requestSizingV2\": context deadline exceeded" dur=7.047671072s
2024-05-17T19:48:57Z DBG Backing off startup IsAvailable() check for recommendation service. error="failed to execute request: Get \"http://kubecost-cost-analyzer.kubecost:9090/model/savings/requestSizingV2\": context deadline exceeded" dur=10.404012694s
2024-05-17T19:49:10Z DBG Backing off startup IsAvailable() check for recommendation service. error="failed to execute request: Get \"http://kubecost-cost-analyzer.kubecost:9090/model/savings/requestSizingV2\": context deadline exceeded" dur=10.401772356s
2024-05-17T19:49:22Z DBG Backing off startup IsAvailable() check for recommendation service. error="failed to execute request: Get \"http://kubecost-cost-analyzer.kubecost:9090/model/savings/requestSizingV2\": context deadline exceeded" dur=21.567909358s
2024-05-17T19:49:46Z DBG Backing off startup IsAvailable() check for recommendation service. error="failed to execute request: Get \"http://kubecost-cost-analyzer.kubecost:9090/model/savings/requestSizingV2\": context deadline exceeded" dur=28.477893192s
2024-05-17T19:50:16Z DBG Backing off startup IsAvailable() check for recommendation service. error="failed to execute request: Get \"http://kubecost-cost-analyzer.kubecost:9090/model/savings/requestSizingV2\": context deadline exceeded" dur=33.660883186s
2024-05-17T19:50:52Z DBG Backing off startup IsAvailable() check for recommendation service. error="failed to execute request: Get \"http://kubecost-cost-analyzer.kubecost:9090/model/savings/requestSizingV2\": context deadline exceeded" dur=1m0.854438656s
2024-05-17T19:51:55Z DBG Backing off startup IsAvailable() check for recommendation service. error="failed to execute request: Get \"http://kubecost-cost-analyzer.kubecost:9090/model/savings/requestSizingV2\": context deadline exceeded" dur=56.07696287s
2024-05-17T19:52:53Z ERR Kubescaler setup failed error="creating a Kubescaler: recommendation service unavailable: failed to execute request: Get \"http://kubecost-cost-analyzer.kubecost:9090/model/savings/requestSizingV2\": context deadline exceeded"
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x23f31fe]

goroutine 1 [running]:
main.main()
	/app/cmd/clustercontroller/main.go:237 +0x53e

[mmclane]

I do see this in the logs when its not trying to implement cluster turndown:

 DBG Recommendation service IsAvailable() GET finished endpoint=http://kubecost-cost-analyzer.kubecost:9090/model/savings/requestSizingV2 status=500