Updates to CronWorkflows from UI not persisting

[yevhen-harmonizehr]

Pre-requisites

• I have double-checked my configuration
• I can confirm the issues exists when I tested with :latest
• I'd like to contribute the fix myself (see contributing guide)

What happened/what you expected to happen?

I am updating cron expression from the UI to any new valid expression. Pressing "UPDATE". Can see "Updated" notification and successful PUT request to backend. But after refreshing the page, i can see old value again.
In logs i can not see anything abnormal near accepted method=PUT request.
Maybe something missing in my configuration? But anyway, why then i see "Updated" notification if it not works.

Version

v3.5.0

Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

Any

Logs from the workflow controller

time="2023-10-20T15:27:55.460Z" level=info msg="Processing argo-workflows/example" cronWorkflow=argo-workflows/example
time="2023-10-20T15:27:55.463Z" level=info msg="CronWorkflow argo-workflows/example added" cronWorkflow=argo-workflows/example
time="2023-10-20T15:27:56.055Z" level=info msg="Processing argo-workflows/example" cronWorkflow=argo-workflows/example
time="2023-10-20T15:27:56.057Z" level=info msg="CronWorkflow argo-workflows/example added" cronWorkflow=argo-workflows/example

Logs from in your workflow's wait container

time="2023-10-20T15:12:26.252Z" level=info duration=52.405881ms method=PUT path=/api/v1/cron-workflows/argo-workflows/example size=2405 status=0

[terrytangyuan]

@agilgur5 This seems related to front-end refactoring since backend works as expected

[agilgur5]

@agilgur5 This seems related to front-end refactoring since backend works as expected

There were actually no recent refactors to the CronWorkflowEditor nor CronWorkflowSpecEditor. One hasn't changed in years and the other only received a small stylistic upgrade in #11686. There was a recent feature to the CronWorkflowDetails page in #11811, but no refactor.

So I'm not sure if that's related.

But after refreshing the page, i can see old value again.

A refresh of the page should also get fresh data (we don't store Workflows in localStorage, only UI preferences), so that actually might suggest that the resource did not get correctly modified on the back-end

Let me see if I can reproduce, may need more details

[agilgur5]

Let me see if I can reproduce, may need more details

I couldn't reproduce this locally. It's updating and persisting fine for me.

so that actually might suggest that the resource did not get correctly modified on the back-end

I might hazard a guess that the modification didn't reach the k8s API correctly. But if you got a successful response, that usually means the whole chain was successful.

Is your actual CronWorkflow resource in k8s updated?

[yevhen-harmonizehr]

Is your actual CronWorkflow resource in k8s updated?

No

[yevhen-harmonizehr]

More logs when i am pressing "update":
Server:

time="2023-10-23T09:56:10.403Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=GetCronWorkflow grpc.service=cronworkflow.CronWorkflowService grpc.start_time="2023-10-23T09:56:10Z" grpc.time_ms=87.833 span.kind=server system=grpc
time="2023-10-23T09:56:10.403Z" level=info duration=88.810145ms method=GET path=/api/v1/cron-workflows/argo-workflows/example size=2403 status=0
time="2023-10-23T09:56:10.534Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=UpdateCronWorkflow grpc.service=cronworkflow.CronWorkflowService grpc.start_time="2023-10-23T09:56:10Z" grpc.time_ms=59.808 span.kind=server system=grpc
time="2023-10-23T09:56:10.536Z" level=info duration=62.9128ms method=PUT path=/api/v1/cron-workflows/argo-workflows/example size=2403 status=0

Controller:

time="2023-10-23T09:56:10.537Z" level=info msg="Processing argo-workflows/example" cronWorkflow=argo-workflows/example
time="2023-10-23T09:56:10.540Z" level=info msg="CronWorkflow argo-workflows/example added" cronWorkflow=argo-workflows/example
time="2023-10-23T09:56:11.080Z" level=info msg="Processing argo-workflows/example" cronWorkflow=argo-workflows/example
time="2023-10-23T09:56:11.087Z" level=info msg="CronWorkflow argo-workflows/example added" cronWorkflow=argo-workflows/example

I am wondering if old CRDs might cause this behaviour?

[agilgur5]

Is your actual CronWorkflow resource in k8s updated?

No

Ok, so this isn't a UI issue then. I'm not really sure why your Server is responding with a 200 if it didn't actually succeed in k8s.

Is there any error in your browser console? Might want to try turning on debug logging on the Server as well.

I am wondering if old CRDs might cause this behaviour?

Usually the CRDs only impact validation of new specs, but I'm not sure how old they are and, more generally speaking, there can be undefined behavior when versions are mixed

[agilgur5]

Another culprit to check for, although this would normally throw an error, is your RBAC properly set to allow updating of Cron Workflows?

[yevhen-harmonizehr]

Nothing in browser console and nothing in network tab.

Server logs with debug:

time="2023-10-24T14:59:51.538Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=GetCronWorkflow grpc.service=cronworkflow.CronWorkflowService grpc.start_time="2023-10-24T14:59:51Z" grpc.time_ms=5.511 span.kind=server system=grpc
time="2023-10-24T14:59:51.540Z" level=info duration=7.028256ms method=GET path=/api/v1/cron-workflows/argo-workflows/example size=2403 status=0
time="2023-10-24T14:59:51.601Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=,name=)" tmpl="*v1alpha1.WorkflowStep (example/example-execute#false)"
time="2023-10-24T14:59:51.602Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=,name=)" tmpl="*v1alpha1.WorkflowStep (example/example-execute#false)"
time="2023-10-24T14:59:51.602Z" level=debug msg="Getting the template from ref" base="*v1alpha1.Workflow (namespace=,name=)" tmpl="*v1alpha1.WorkflowStep (example/example-execute#false)"
time="2023-10-24T14:59:51.609Z" level=debug msg="Stored the template" base="*v1alpha1.Workflow (namespace=,name=)" tmpl="*v1alpha1.WorkflowStep (example/example-execute#false)"
time="2023-10-24T14:59:51.614Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=,name=)" tmpl="*v1alpha1.WorkflowStep (example/example-execute#false)"
time="2023-10-24T14:59:51.614Z" level=debug msg="Found stored template" base="*v1alpha1.Workflow (namespace=,name=)" tmpl="*v1alpha1.WorkflowStep (example/example-execute#false)"
time="2023-10-24T14:59:51.638Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=UpdateCronWorkflow grpc.service=cronworkflow.CronWorkflowService grpc.start_time="2023-10-24T14:59:51Z" grpc.time_ms=47.901 span.kind=server system=grpc
time="2023-10-24T14:59:51.639Z" level=info duration=49.55758ms method=PUT path=/api/v1/cron-workflows/argo-workflows/example size=2403 status=0
time="2023-10-24T14:59:55.085Z" level=info duration="228.296µs" method=GET path=index.html size=487 status=0
time="2023-10-24T14:59:55.573Z" level=info duration="142.273µs" method=GET path=index.html size=487 status=0

Controller logs with debug:

time="2023-10-24T14:59:47.429Z" level=debug msg="Get leases 200"
time="2023-10-24T14:59:47.435Z" level=debug msg="Update leases 200"
time="2023-10-24T14:59:47.661Z" level=debug msg="received config map kube-system/aws-load-balancer-controller-leader update"
time="2023-10-24T14:59:49.689Z" level=debug msg="received config map kube-system/aws-load-balancer-controller-leader update"
time="2023-10-24T14:59:51.078Z" level=debug msg="Syncing all CronWorkflows"
time="2023-10-24T14:59:51.086Z" level=debug msg="Enforcing history limit for 'example'" namespace=argo-workflows workflow=example
time="2023-10-24T14:59:51.477Z" level=debug msg="Watch configmaps 200"
time="2023-10-24T14:59:51.647Z" level=info msg="Processing argo-workflows/example" cronWorkflow=argo-workflows/example
time="2023-10-24T14:59:51.648Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=,name=)" tmpl="*v1alpha1.WorkflowStep (example/example-execute#false)"
time="2023-10-24T14:59:51.648Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=,name=)" tmpl="*v1alpha1.WorkflowStep (example/example-execute#false)"
time="2023-10-24T14:59:51.648Z" level=debug msg="Getting the template from ref" base="*v1alpha1.Workflow (namespace=,name=)" tmpl="*v1alpha1.WorkflowStep (example/example-execute#false)"
time="2023-10-24T14:59:51.649Z" level=debug msg="Stored the template" base="*v1alpha1.Workflow (namespace=,name=)" tmpl="*v1alpha1.WorkflowStep (example/example-execute#false)"
time="2023-10-24T14:59:51.651Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=,name=)" tmpl="*v1alpha1.WorkflowStep (example/example-execute#false)"
time="2023-10-24T14:59:51.651Z" level=debug msg="Found stored template" base="*v1alpha1.Workflow (namespace=,name=)" tmpl="*v1alpha1.WorkflowStep (example/example-execute#false)"
time="2023-10-24T14:59:51.651Z" level=info msg="CronWorkflow argo-workflows/example added" cronWorkflow=argo-workflows/example
time="2023-10-24T14:59:51.730Z" level=debug msg="received config map kube-system/aws-load-balancer-controller-leader update"
time="2023-10-24T14:59:52.278Z" level=debug msg="Get leases 200"
time="2023-10-24T14:59:52.269Z" level=info msg="Processing argo-workflows/example" cronWorkflow=argo-workflows/example
time="2023-10-24T14:59:52.270Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=,name=)" tmpl="*v1alpha1.WorkflowStep (example/example-execute#false)"
time="2023-10-24T14:59:52.270Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=,name=)" tmpl="*v1alpha1.WorkflowStep (example/example-execute#false)"
time="2023-10-24T14:59:52.270Z" level=debug msg="Getting the template from ref" base="*v1alpha1.Workflow (namespace=,name=)" tmpl="*v1alpha1.WorkflowStep (example/example-execute#false)"
time="2023-10-24T14:59:52.271Z" level=debug msg="Stored the template" base="*v1alpha1.Workflow (namespace=,name=)" tmpl="*v1alpha1.WorkflowStep (example/example-execute#false)"
time="2023-10-24T14:59:52.271Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=,name=)" tmpl="*v1alpha1.WorkflowStep (example/example-execute#false)"
time="2023-10-24T14:59:52.271Z" level=debug msg="Found stored template" base="*v1alpha1.Workflow (namespace=,name=)" tmpl="*v1alpha1.WorkflowStep (example/example-execute#false)"
time="2023-10-24T14:59:52.272Z" level=info msg="CronWorkflow argo-workflows/example added" cronWorkflow=argo-workflows/example
time="2023-10-24T14:59:52.445Z" level=debug msg="Get leases 200"
time="2023-10-24T14:59:52.451Z" level=debug msg="Update leases 200"
time="2023-10-24T14:59:53.753Z" level=debug msg="received config map kube-system/aws-load-balancer-controller-leader update"
time="2023-10-24T14:59:55.779Z" level=debug msg="received config map kube-system/aws-load-balancer-controller-leader update"
time="2023-10-24T14:59:57.464Z" level=debug msg="Get leases 200"
time="2023-10-24T14:59:57.487Z" level=debug msg="Update leases 200"
time="2023-10-24T14:59:57.819Z" level=debug msg="received config map kube-system/aws-load-balancer-controller-leader update"

Btw why is it looking into configmap of aws LB controller?)

Another culprit to check for, although this would normally throw an error, is your RBAC properly set to allow updating of Cron Workflows?

I personally didn't adjust any RBAC resources on my own, but i can get clusterrole to check, just say which one i should check.

[yevhen-harmonizehr]

I found the issue. Miss-configured ignoreDifferences on the Argo CD side + selfHeal: true in application params. Argo CD was changing k8s resource back to default state very fast, this is why i was not able to see any changes in k8s manifest. Sorry guys, this is really stupid from me.
Issue resolved, will close it. Thanks for your help.

off topic: do you know why controller updating aws LB configmap?

[agilgur5]

off topic: do you know why controller updating aws LB configmap?

It's probably not updating it, but the Controller does watch for ConfigMap updates as certain features use ConfigMaps (e.g. synchronization, passing arguments from a ConfigMap, auto-updating when the workflow-controller-configmap changes, etc). So it probably received an update event on your AWS LB ConfigMap, debug logged it, and otherwise ignored it as it's unrelated to Argo. It comes from this line in the codebase, for reference