Workflow Retry Does Not Work for Plugin #11489
Comments
|
To Have Failed state for the plugin node I have on purpose added exit -1 in the plugin script. |
Looking forward to your PR! |
agilgur5
added
area/plugins
area/retry-manual
|
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. |
I have a doubt that why the controller does not clean up the WorkflowTaskSet when the workflow is completed. The patch action of failed node status when manual retrying may be omitted if so. |
|
argo-workflows/workflow/controller/controller.go Lines 870 to 875 in 13444e6 argo-workflows/workflow/controller/taskset.go Lines 34 to 52 in 13444e6 Controller will try to delete completed nodes of HTTP/Plugin type. Howerver, the node delete patch did not affect to the status field.I've tried json patch like '[{"op": "remove", "path": "/status/nodes/plugin-demo-g5qfg"}]', it also not worked. |
|
As the WorkflowTaskSet CRD shown below, it has set the apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"apiextensions.k8s.io/v1","kind":"CustomResourceDefinition","metadata":{"annotations":{},"labels":{"app.kubernetes.io/part-of":"argo"},"name":"workflowtasksets.argoproj.io"},"spec":{"group":"argoproj.io","names":{"kind":"WorkflowTaskSet","listKind":"WorkflowTaskSetList","plural":"workflowtasksets","shortNames":["wfts"],"singular":"workflowtaskset"},"scope":"Namespaced","versions":[{"name":"v1alpha1","schema":{"openAPIV3Schema":{"properties":{"apiVersion":{"type":"string"},"kind":{"type":"string"},"metadata":{"type":"object"},"spec":{"type":"object","x-kubernetes-map-type":"atomic","x-kubernetes-preserve-unknown-fields":true},"status":{"type":"object","x-kubernetes-map-type":"atomic","x-kubernetes-preserve-unknown-fields":true}},"required":["metadata","spec"],"type":"object"}},"served":true,"storage":true,"subresources":{"status":{}}}]}}
creationTimestamp: "2024-01-26T03:22:45Z"
generation: 1
labels:
app.kubernetes.io/part-of: argo
name: workflowtasksets.argoproj.io
resourceVersion: "1011"
uid: 7fc3e411-435d-4b29-a0ea-75fc25510304
spec:
conversion:
strategy: None
group: argoproj.io
names:
kind: WorkflowTaskSet
listKind: WorkflowTaskSetList
plural: workflowtasksets
shortNames:
- wfts
singular: workflowtaskset
scope: Namespaced
versions:
- name: v1alpha1
schema:
openAPIV3Schema:
properties:
apiVersion:
type: string
kind:
type: string
metadata:
type: object
spec:
type: object
x-kubernetes-map-type: atomic
x-kubernetes-preserve-unknown-fields: true
status:
type: object
x-kubernetes-map-type: atomic
x-kubernetes-preserve-unknown-fields: true
required:
- metadata
- spec
type: object
served: true
storage: true
subresources:
status: {}
status:
acceptedNames:
kind: WorkflowTaskSet
listKind: WorkflowTaskSetList
plural: workflowtasksets
shortNames:
- wfts
singular: workflowtaskset
conditions:
- lastTransitionTime: "2024-01-26T03:22:45Z"
message: no conflicts found
reason: NoConflicts
status: "True"
type: NamesAccepted
- lastTransitionTime: "2024-01-26T03:22:45Z"
message: the initial names have been accepted
reason: InitialNamesAccepted
status: "True"
type: Established
storedVersions:
- v1alpha1
So, the `status` field needs to be modified with command like `kubectl patch wfts ${workflowTaskSetName} --subresource='status'` now.
|
|
argo-workflows/workflow/controller/taskset.go Lines 34 to 52 in 13444e6 So, spec and status can not be modified in one patch json, status should be patched with subresources.
|
Instead of fixing the patch bug, it would be much simpler to delete the entire WorkflowTaskSet. |
…ixes argoproj#11489 Signed-off-by: oninowang <oninowang@tencent.com>
…ixes argoproj#11489 Signed-off-by: oninowang <oninowang@tencent.com>
…ixes argoproj#11489 Signed-off-by: oninowang <oninowang@tencent.com>
…ixes argoproj#11489 Signed-off-by: oninowang <oninowang@tencent.com>
…ixes argoproj#11489 Signed-off-by: oninowang <oninowang@tencent.com>
…ixes argoproj#11489 Signed-off-by: oninowang <oninowang@tencent.com>
…ixes argoproj#11489 Signed-off-by: oninowang <oninowang@tencent.com>
…ixes argoproj#11489 Signed-off-by: oninowang <oninowang@tencent.com>
agilgur5
added
P2
Pre-requisites
:latestWhat happened/what you expected to happen?
What happened:
When using Plugins , If a plugin nodes goes into Failed Phase, Workflow Retry Does Not Work. Upon Retry the Plugin Node is not retried and remains in Failed Phase.
what you expected to happen:
Plugin Node should be retried.
Self Diagnosis:
Based on exploring the codebase, I see that workflow controller loads Plugin Node state from WorkflowTaskSet Status. Ref.
On Retry Argo Workflow Server Indeed removes the Failed Plugin Node from Workflow Node Status , But does not patch the WorkflowTaskSet . B/c of which workflow controller loads the plugin node status from WorkflowTaskSet.
IMO Upon Retry Argo Workflow Server should patch the WorkflowTaskSet (Remove the status for the failed Plugin Node).
Version
latest
Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.
Logs from the workflow controller
Logs from in your workflow's wait container
The text was updated successfully, but these errors were encountered: