HPA objects can always be OutOfSync #1079
Comments
Can you elaborate on this? Can you share redacted output of |
|
Sure, if you look at the log, you can see the last one being |
|
@bappr this payload is from Argo CD v0.10. There were a bunch of improvements to the sync logic in v0.11 (among other things). I think we'll need to see if this is still happening in v0.11 to go further. Can you upgrade to v0.11.1 and report back? Please be aware of the breaking changes in v0.11 before you upgrade, most notably the change in resource names and the CLI incompatibility. Release notes are described here: |
|
It s planned for our next sprint. |
|
I get the same problem with v0.11.1. It seems related to having multiple However this one always shows Out of Sync: And the Diff from the CLI: |
jessesuen
changed the title
|
We understand the problem. It appears that the HPA controller can re-order the metrics list. Argo CD is correctly detecting this as a difference because everywhere else, an order of a list is significant. I think the solution may be one or more of:
|
|
IMO, the fact that kubernetes is re-ordering the metrics list is a kubernetes bug, and should be fixed upstream. In the mean-time, option 1 appears to be working internally, without any Argo CD code changes. |
|
Thank you - can confirm that option 1, reordering the metrics (placing memory before cpu in this case), seems to resolve the issue I was seeing. |
|
Filed upstream issue kubernetes/kubernetes#74099. |
|
Thanks guys, gonna do that as well! |
rarguelloF
added a commit
to rarguelloF/argo-helm
that referenced
this issue
Feb 20, 2020
As described in this issue: argoproj/argo-cd#1079 Argocd is displaying a diff as k8s is internally reordering the metrics.
5 tasks
8 tasks
4 tasks
danielgblanco
added a commit
to danielgblanco/opentelemetry-helm-charts
that referenced
this issue
Nov 29, 2021
A current issue between ArgoCD (argoproj/argo-cd#1079) and Kubernetes (kubernetes/kubernetes#74099) makes the HPA stay constantly out of sync. This affects Gitops pipelines when both memory and CPU metrics are used to autoscale the deployment. This change reorders metrics to make sure that they remain in the order that the Kubernetes HPA controller currently expects them to be.
danielgblanco
added a commit
to danielgblanco/opentelemetry-helm-charts
that referenced
this issue
Nov 29, 2021
A current issue between ArgoCD (argoproj/argo-cd#1079) and Kubernetes (kubernetes/kubernetes#74099) makes the HPA stay constantly out of sync. This affects Gitops pipelines when both memory and CPU metrics are used to autoscale the deployment. This change reorders the HPA metrics to make sure that they remain in the order that the Kubernetes HPA controller currently expects them to be.
dmitryax
pushed a commit
to open-telemetry/opentelemetry-helm-charts
that referenced
this issue
Dec 3, 2021
A current issue between ArgoCD (argoproj/argo-cd#1079) and Kubernetes (kubernetes/kubernetes#74099) makes the HPA stay constantly out of sync. This affects Gitops pipelines when both memory and CPU metrics are used to autoscale the deployment. This change reorders the HPA metrics to make sure that they remain in the order that the Kubernetes HPA controller currently expects them to be.
chrisdotm
added a commit
to chrisdotm/helm-charts-2
that referenced
this issue
Aug 10, 2022
Really this is for argoproj/argo-cd#1079 which points blame at upstream k8s.
JamieDanielson
pushed a commit
to honeycombio/helm-charts
that referenced
this issue
Aug 18, 2022
* Swap order of hpa spec.metrics Really this is for argoproj/argo-cd#1079 which points blame at upstream k8s. Swapping the CPU and memory metric for the HPA. Nothing should change other than the resulting order of the rendered template.
5 tasks
|
To any future users you can use spec:
ignoreDifferences:
- group: autoscaling
kind: HorizontalPodAutoscaler
jqPathExpressions:
- .spec.metrics[].resource.name | select((. == "cpu") or (. == "memory")) |
|
Or you can add this to your argocd config map: Following the system wide configuration: https://argo-cd.readthedocs.io/en/stable/user-guide/diffing/#system-level-configuration |
docwhat
added a commit
to holtje/hive-config
that referenced
this issue
Jan 23, 2023
|
This issue can be resolved just by chainging the order of resources in the HPA. i was able to reoslve it by placing memory reousrces first and then cpu in spec.metrics.type. Try it |
8 tasks
tiithansen
added a commit
to tiithansen/agent
that referenced
this issue
May 30, 2023
By reordering metrics argocd will no longer be stuck in endless loop. Related issue in ArgoCD project argoproj/argo-cd#1079
rfratto
pushed a commit
to grafana/agent
that referenced
this issue
May 30, 2023
By reordering metrics argocd will no longer be stuck in endless loop. Related issue in ArgoCD project argoproj/argo-cd#1079
Merged
4 tasks
poblahblahblah
added a commit
to poblahblahblah/sumologic-kubernetes-collection
that referenced
this issue
Jun 7, 2023
This addresses an issue when using ArgoCD (and maybe other GitOps operators) where Kubernetes reorders the objects under the spec.metrics key thus causing Sync issues with ArgoCD. Originally reported to the ArgoCD project here: argoproj/argo-cd#1079 Originally reported to the Kubernetes project here: kubernetes/kubernetes#74099 Other projects and companies have also addressed this by simply reordering the metrics section: * kubernetes/ingress-nginx#10043 * nginxinc/kubernetes-ingress#3773 * grafana/helm-charts#758 * open-telemetry/opentelemetry-helm-charts#103 * Nextdoor/k8s-charts#102 Signed-off-by: Patrick O’Brien <patrick.obrien@thetradedesk.com>
poblahblahblah
added a commit
to poblahblahblah/sumologic-kubernetes-collection
that referenced
this issue
Jun 12, 2023
This addresses an issue when using ArgoCD (and maybe other GitOps operators) where Kubernetes reorders the objects under the spec.metrics key thus causing Sync issues with ArgoCD. Originally reported to the ArgoCD project here: argoproj/argo-cd#1079 Originally reported to the Kubernetes project here: kubernetes/kubernetes#74099 Other projects and companies have also addressed this by simply reordering the metrics section: * kubernetes/ingress-nginx#10043 * nginxinc/kubernetes-ingress#3773 * grafana/helm-charts#758 * open-telemetry/opentelemetry-helm-charts#103 * Nextdoor/k8s-charts#102 Signed-off-by: Patrick O’Brien <patrick.obrien@thetradedesk.com>
swiatekm
pushed a commit
to SumoLogic/sumologic-kubernetes-collection
that referenced
this issue
Jun 13, 2023
* Reordering HPA metrics to match HPA ordering This addresses an issue when using ArgoCD (and maybe other GitOps operators) where Kubernetes reorders the objects under the spec.metrics key thus causing Sync issues with ArgoCD. Originally reported to the ArgoCD project here: argoproj/argo-cd#1079 Originally reported to the Kubernetes project here: kubernetes/kubernetes#74099 Other projects and companies have also addressed this by simply reordering the metrics section: * kubernetes/ingress-nginx#10043 * nginxinc/kubernetes-ingress#3773 * grafana/helm-charts#758 * open-telemetry/opentelemetry-helm-charts#103 * Nextdoor/k8s-charts#102 Signed-off-by: Patrick O’Brien <patrick.obrien@thetradedesk.com> * add CHANGELOG entry --------- Signed-off-by: Patrick O’Brien <patrick.obrien@thetradedesk.com>
clayton-cornell
pushed a commit
to grafana/agent
that referenced
this issue
Aug 14, 2023
By reordering metrics argocd will no longer be stuck in endless loop. Related issue in ArgoCD project argoproj/argo-cd#1079
clayton-cornell
pushed a commit
to grafana/agent
that referenced
this issue
Aug 14, 2023
By reordering metrics argocd will no longer be stuck in endless loop. Related issue in ArgoCD project argoproj/argo-cd#1079
2 tasks
6 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
HPA not being refreshed even if update is successful.
The source is a chart description.
Logs from application controller.
The text was updated successfully, but these errors were encountered: