sync is not working with ignoreDifferences and RespectIgnoreDifferences=true #9678
Comments
|
I am also experiencing this issue |
|
We are also seeing the issue, we have two ignore differences on Deployments, one for Through experimentation I've found that if you change So my assumption is that using an array, potentially only a 'wildcard' array, within JQ breaks the Sync logic |
|
Did a further experiment and found that it doesn't appear to be just the JQ expression but in fact anything with an array. |
|
Facing also the same issue. Here is a simple PoC of the error to reproduce it: If you change the image tag from This occurs on Argo 2.4.8 and 2.4.14, also on the newest version 2.5.2. |
|
Same behavior on arrays if syncPolicy:
syncOptions:
- RespectIgnoreDifferences=true
ignoreDifferences:
- group: apps
kind: Deployment
managedFieldsManagers:
- openshift-controller-managerOpenShift allows triggering image stream updates directly on the Deployment and updates the container image. Was easier to use the manager name, so the difference is only ignored, if the trigger is used in our apps. |
|
Confirmed this is also not working with 2.5.2. We are have an ignore like this ignoreDifferences:
- group: certmanager.io
kind: Certificate
jsonPointers:
- /spec/dnsNames/2We see the ignore works correctly, but the 3rd dns name is removed when syncing with spec:
dnsNames:
- in git
- also in git
- manually added and needs to be ignoredAnd desired state spec:
dnsNames:
- in git
- also in git |
|
any updates on this issue? |
|
I'm facing the same issue with the list of environment variables for a container. Any updates on this issue? Or even a work around? 🙁 |
|
We are using some operator that is changing the env vars of the pod in the deployment. So now, because of this issue, argo rolling back operator changes to its defaults on every sync (and then operator changes it again). |
|
@Len4i did you forget to configure |
|
@QuingKhaos, I'll elaborate at this stage behavior is following: After adding |
unfortunately this is the only viable workaround currently, to not end in a sync loop. |
In my use case, if we update some value in Problem still exists in ArgoCD v2.4.40 |
|
what are the chances that this issue is related? |
|
same issue on latest version |
|
same issue on latest version |
|
same here with the latest version :( I have With this configuration if I add a new container in my deployment, application stay OutOfSync and never update the Deployment with the new container. Same thing occurs with jsonPointers: Interesting things occurs with With this managedFieldsManagers configuration, on a fresh deployment, if I add a container in my spec, it is added and synced correctly, then I make a change in Syncing from the UI do not work until I disable the |
|
Is there any update on this? Having this issue on the latest version {
"Version": "v2.7.3+e7891b8.dirty",
"BuildDate": "2023-05-24T15:05:34Z",
"GitCommit": "e7891b899a35dca06ae94965ea5ae2a86b344848",
"GitTreeState": "dirty",
"GoVersion": "go1.19.6",
"Compiler": "gc",
"Platform": "linux/amd64",
"KustomizeVersion": "v5.0.1 2023-03-14T01:32:48Z",
"HelmVersion": "v3.11.2+g912ebc1",
"KubectlVersion": "v0.24.2",
"JsonnetVersion": "v0.19.1"
} |
|
I had a similar issue where I couldn't sync because of a difference in an immutable field which I wanted to ignore. However, setting |
|
Same here with 2.7.4 Still showing differences on the caBundle. I see the gojq library in question was recently updated in main but doesnt appear to have made it into the 2.7.4 release. So hopefully it will go into the 2.7.5 release. |
|
Same here with 2.7.6: Upon changing container command arguments, ArgoCD sync succeeds with
indeed works, and the resource is |
|
There may be more than one problem, but I think most of them can be traced back to this function: Line 446 in ef8dae8 Imagine I want to modify a CronJob to add a new container. To construct the patch, Argo CD takes the git state ( When That's the answer for the specific case of "adding a new container." The general answer is: this patch function is custom and is not Kubernetes-aware. A Kubernetes-aware patch function would have merged the array on a merge key (like container name) if present. |
|
That's the boring explanation of "what seems to be wrong." In order to fix everyone's use cases, I need details about what is failing:
We have those for a few use cases above. Those can inform the unit tests we write for the fix. But the more cases we can cover, the better. |
|
@blakepettersson @crenshaw-dev I'm trying to add/update the liveness/readiness probes on a container, and have the following ignoreDifferences:
- group: argoproj.io
kind: "Rollout"
jqPathExpressions:
- ".spec.template.spec.containers[] | select(.name == \"init\") | .image"This is set on an
In this example, I have changed the
The application is also marked as "OutOfSync", which is correct, but when I sync the application with "Respect Ignore Differences" it loads for a split second and then does nothing. Hope that was a sufficient example. |
|
@FredrikAugust could you post the full (or somewhat redacted) live state of the |
|
@blakepettersson Sure! I've posted the live state to a gist here: https://gist.github.com/FredrikAugust/8ba73d9665b60b168ab96d3f5ec69949 The desired state, if I understand the UI correctly should be that, but with the |
|
@FredrikAugust indeed it was easy to reproduce, and it also seems like it's currently failing locally. I'll look into that and fix it, and push the update to #14602. |
|
hi folks, update on my comment above: #9678 (comment) for those seeing this issue with |
|
Sounds like a good case to unit test and add to #14602 :-) |
There's an issue with `intersectMap` not appending items for array properties where the live array is smaller than the desired array. Hopefully fixes argoproj#9678. Signed-off-by: Blake Pettersson <blake.pettersson@gmail.com>
|
I think I've hit the same bug. Is there any progress or estimation on this? |
|
Hello, we are experiencing this issue on ArgoCD v2.8.1 |
|
ArgoCD v2.9.2 - same issue |
|
would this also apply when ignoring map keys (like in labels/annotations)? resource.customizations.ignoreDifferences.all: |
managedFieldsManagers:
- kube-controller-manager
jsonPointers:
- /spec/replicas
- /metadata/annotations/reloader.stakater.com~1last-reloaded-from
- /spec/template/metadata/annotations/reloader.stakater.com~1last-reloaded-fromAnd then a live manifest: spec:
template:
metadata:
annotations:
reloader.stakater.com/last-reloaded-from: >-
{"type":"CONFIGMAP","name":"REDACTED","namespace":"REDACTED","hash":"4a665ab5539934f5d6ccd39b90976ab28786bdcc","containerRefs":["app"],"observedAt":1703227429}And a desired manifest: spec:
template:
metadata:
annotations: {}This creates the following diff template:
metadata:
- annotations: {}The application gets stuck in |
|
I like how I'm following this issue, Thanks for making me laugh once again @cirano-eusebi ** I'll be ruthlessly truthful, The joke is this 'tool' and that more and more people are being affected by this issue.. (I'm on your team) #9678 (comment) |
Hi @Len4i what do you mean by removing |
|
I confirm that the original issue still persists, however, this workaround seemed to do the trick for me - kind: Secret
name: aws-load-balancer-tls
jsonPointers:
- /data/ca.crt
- /data/tls.crt
- /data/tls.key
- group: admissionregistration.k8s.io
kind: MutatingWebhookConfiguration
name: aws-load-balancer-webhook
jsonPointers:
- /webhooks/0/clientConfig/caBundle
- /webhooks/1/clientConfig/caBundle
- /webhooks/2/clientConfig/caBundle
- group: admissionregistration.k8s.io
kind: ValidatingWebhookConfiguration
name: aws-load-balancer-webhook
jsonPointers:
- /webhooks/0/clientConfig/caBundle
- /webhooks/1/clientConfig/caBundle
- /webhooks/2/clientConfig/caBundle |
|
I am working on testing this out but want to get some clarification on my usage. if I have a jq expression for ignoring an array in a resource spec. If I was to delete the entire argo application without cascading and have it recreated, would having Just trying to better understand the documentation at https://argo-cd.readthedocs.io/en/stable/user-guide/sync-options/#respect-ignore-difference-configs |
|
Hi. I'm wondering if there are any updates on this issue? We're still affected by this, and have to use a workaround to get our CD pipelines to function as expected. We would like to have the image of one of our containers be ignored so we can update that from the pipeline, and then manage everything else with ArgoCD. This would work very well, but due to this issue it doesn't, and we ended up managing the image tag in the manifests in git and running sync from the CD pipeline. This feels a bit hacky, and as a result all other scheduled changes on the apps will be applied as well. If anyone has any suggestions as to other ways you could do CD with argocd that would be greatly appreciated. @crenshaw-dev You mentioned in the linked PR that this should be solved by improving the diffing/patch generation functionality by using the strategic patch merge. Is this on the roadmap, or is there any estimate of when it will be addressed? |
|
There's no more updates than what is above. I believe that Blake would welcome contributions to the MR. As for your 'workaround' I'm not sure what the issue is. ArgoCD is a GitOps CD tool. It feels like what you are trying to achieve is something different to GitOps and why not just do a What you describe is more or less exactly what our pipelines do by design, except we have auto sync on so anything that goes to the main branch gets synced immediately. We still use sync from the pipelines to report status back to the merger and manage tests/promotions. Two of the major goals we had were removing privileged access from pipelines and DR, this pipeline design solves both. The pipeline only needs read access to the cluster and if we ever lose the cluster everything is in Git. |
|
My issue is as follows.
Along with The resourc ports arrays looks like below to begin (initial sync and creation) and a k8s operator appends and removes ports based off some business logic we instrument, We need argo to not label it as out of sync and also to not remove those ports added during a sync (which its currently doing even with the ignoreDifference config provided above). My other test case is also deleting the entirety of the argo application |
|
The last 'update' is effectively that the PR from blake isn't the best approach to take and it has be re-thought. |
|
With #14602, trying to fix this quickly got hairy and had a bunch of edge cases which didn't actually work. I took another look at this issue and think that a more viable approach would be to rethink Here's an example using apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: client
name: client
spec:
replicas: 1
selector:
matchLabels:
app: client
strategy: {}
template:
metadata:
labels:
app: client
spec:
containers:
- image: alpine:3
name: alpine
resources: {}kubectl apply -f minimal-image-replicas-deployment.yaml --server-side
deployment.apps/client serverside-appliedThe way to make apiVersion: apps/v1
kind: Deployment
metadata:
name: client
spec:
template:
spec:
containers:
- image: alpine:3
name: alpineThen we apply this subset with another field manager than the one used in the previous SSA. The naming is arbitrary but we call it Then we try to update the deployment with a bunch of changed fields, as well as attempting to change the image tag (from apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: client
appProcess: web
name: client
spec:
replicas: 2
selector:
matchLabels:
app: client
strategy: {}
template:
metadata:
labels:
app: client
appProcess: web
spec:
containers:
- image: alpine:2
name: alpine
resources:
requests:
cpu: 400m
env:
- name: EV
value: hereThis will fail. Sadly apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: client
appProcess: web
name: client
spec:
replicas: 2
selector:
matchLabels:
app: client
strategy: {}
template:
metadata:
labels:
app: client
appProcess: web
spec:
containers:
# This time without the image tag
- name: alpine
resources:
requests:
cpu: 400m
env:
- name: EV
value: herekubectl apply -f additional-image-replicas-deployment.yaml --server-side
deployment.apps/client serverside-appliedThe reason I bring up server-side diff is that in many cases server-side diff will show the desired changes without the ignored fields, so could potentially save a round-trip to the API server (preventing the need to retry with removed fields). This example didn't work while having the image tag changed using server-side diff though, but it did work for other examples I tried. |
|
@blakepetterson Thanks for the great writeup. I'm sadly not too familiar with the diffing ecosystem, but I am able to contribute if there is any development that needs to be done if I'm shown the general direction. The problem caused by this issue is blocking us from running the pipelines how we'd prefer so I'd be able to dedicate quite some time to resolving this if you want some assistance. Am I also reading this correctly in that this would require changes to Kubernetes (or the underlying library for kubectl)? Or would it be possible to implement this in ArgoCD/GitOpsEngine? After reading the Field management documentation from Kubernetes, would it be possible to use a field manager for the ignored fields, and filter out the conflicting fields during applies:
|
I think the way to go would be to create a formal proposal for this. Since Argo CD is going in the direction of only having SSA in the future, I think this would fit in neatly. Before writing up something more formal it would be good to get some feedback from either @crenshaw-dev or @leoluz to see if this is something that sounds agreeable (perhaps a topic for tomorrow's contributors' meeting?).
No, I think this can be resolved in Argo CD / gitops-engine, having this done natively in k8s would have made things a bit more convenient though.
I was thinking something along the lines of fetching the fields from the ignore field manager and filtering out those fields before an apply could be an option, but haven't given this too much thought. I think in most cases doing a server-side diff and applying only that patch during a sync should work though (and in the cases it doesn't it shouldn't be too difficult to remove the offending field from the diff). |
and others
Checklist:
argocd version.Describe the bug
I have an application where I want to ignore the image of the deployment and ignore it from being synced, I can see that the image is ignored and there is new env variables to be added in app diff but once I sync it it succeeds without applying the env variables like it's ignored with the image itself and I still see them in app diff!. This is the application:
To Reproduce
Apply an application with ignoreDifferences like the application above and add new changes inside the same container specs
The new specs will not be applied after sync.
Expected behavior
I expect the image to be ignored and the new env variables to be applied/synced
Version
The text was updated successfully, but these errors were encountered: