-
Notifications
You must be signed in to change notification settings - Fork 5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ArgoCD Application Stuck In Syncing/Terminating State, when status field update exceeds Application CR resource size #8113
Comments
time="2022-01-07T18:08:00Z" level=warning msg="finished unary call with code FailedPrecondition" error="rpc error: code = FailedPrecondition desc = another operation is already in progress" grpc.code=FailedPrecondition grpc.method=Sync grpc.service=application.ApplicationService grpc.start_time="2022-01-07T18:08:00Z" grpc.time_ms=484.012 span.kind=server system=grpc |
@ravi-cldcvr can you provide more information on the Application you are attempting to synchronize? For example, can you provide the YAML definition for it? It appears that Argo CD is unable to update the status field of the CR, due to it being too large (for example, the Application has too many child resources). |
@jgwest Thanks for your response. |
I think every-time when argocd sync with git in the backend it creates a revision and append in same application yaml file. |
@jgwest Can you suggest something on this to make it work in our current production env bcz all our application got stuck in syncing state which is very painful. |
I'm not aware of any Argo CD settings that you can tweak here to reduce the size/verbosity of the status field (you could ask on Argo CD slack to see if anyone else has hit this or similar issue), my only suggestion would be to refactor your Argo CD application to use fewer resources. |
@jgwest Thanks for your response. |
@gouravjoshicldcvr Argo CD planning is not that formally organized, each bug and feature is evaluated by individual teams/companies/contributors for contributions, such I can't give you such an estimate. |
@jgwest Can we restrict argocd to store only certain number of revision like 3 or 5, in this limit will never breach. |
This issue was discussed in the contributing meeting today.
The history of syncs normally contributes very little to the size. However, it's possible it may get quite large if you are leveraging inlined helm values. Is that the case? For example, here is example of items in history:
Yes, if history is contributing to your CR size, you can reduce this using: kind: Application
spec:
revisionHistoryLimit: 1 Outside of that, the other largest contributors to CRD size are:
However, there is no way to reduce those two in size |
|
@jgwest by limiting the reversion this issue got solved. |
In case anyone else ends up here with a similar issue, we ran into this, but we couldn't get the application controller to "terminate" the sync, even after setting the revision limit. The solution ended up being to delete the "status" field via a "kubectl edit" and set the status to an empty object. |
ArgoCD Application deployment is stuck in an infinite loop of Syncing the Application. Initially it works but after sometimes it get stuck in Syncing state.
We have tried Terminating the sync but after terminating it got stuck in Terminating State.
Can Anyone Help to sort this out. What would be the possible issue!!
Screenshots
Version
Logs
The text was updated successfully, but these errors were encountered: