add customizations for argoproj.io/v1alpha1/Workflow #3438

yike21 · 2023-04-20T12:46:56Z

What type of PR is this?
/kind cleanup

What this PR does / why we need it:
Add third-party resources argoproj.io/v1alpha1/Workflow into Resource Interpreter framework.

Which issue(s) this PR fixes:
Part of #3331

Special notes for your reviewer:
Only workflow-level resources and requirements are collected. The template-level resources which can be generated by workflow are ignored.

Does this PR introduce a user-facing change?:
NONE

`Resource Interpreter`: Support `argoproj.io/v1alpha1/Workflow`

codecov-commenter · 2023-04-20T13:05:45Z

Codecov Report

Merging #3438 (9471d16) into master (451a21c) will increase coverage by 0.01%.
The diff coverage is n/a.

❗ Your organization is not using the GitHub App Integration. As a result you may experience degraded service beginning May 15th. Please install the Github App Integration for your organization. Read more.

@@            Coverage Diff             @@
##           master    #3438      +/-   ##
==========================================
+ Coverage   54.81%   54.83%   +0.01%     
==========================================
  Files         216      216              
  Lines       20121    20121              
==========================================
+ Hits        11030    11033       +3     
+ Misses       8494     8491       -3     
  Partials      597      597

Flag	Coverage Δ
unittests	`54.83% <ø> (+0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

see 1 file with indirect coverage changes

chaunceyjiang · 2023-04-21T01:53:14Z

/assign

chaunceyjiang · 2023-04-24T08:58:16Z

Second, sometimes if I distribute a workflow to subclusters and check its pod logs, the pod has run well and succeed, however if I check the status with the command line kubectl describe workflow sample, the status field is not updated for the workflow instance in the subcluster. This further results in the workflow collected by resourcebinding also not having a status field.

The workflow sample in the member cluster has a status field.

Because the generation of the workflow sample in the member cluster is not equal to that of the workflow sample in Karmada, So, Karmada updated the workflow sample in the member cluster.

This update operation did not trigger the workflow controller in the member cluster.

yike21 · 2023-04-24T09:02:48Z

Second, sometimes if I distribute a workflow to subclusters and check its pod logs, the pod has run well and succeed, however if I check the status with the command line kubectl describe workflow sample, the status field is not updated for the workflow instance in the subcluster. This further results in the workflow collected by resourcebinding also not having a status field.

The workflow sample in the member cluster has a status field.

Because the generation of the workflow sample in the member cluster is not equal to that of the workflow sample in Karmada, So, Karmada updated the workflow sample in the member cluster.

This update operation did not trigger the workflow controller in the member cluster.

Thanks very much!

chaunceyjiang · 2023-04-24T09:23:45Z

This seems to be an issue with the Workflow itself. I suggest we put this PR on hold for now.

/hold

chaunceyjiang · 2023-04-25T01:39:50Z

...hirdparty/resourcecustomizations/apps.kruise.io/v1alpha1/AdvancedCronJob/customizations.yaml

This file should be submitted as a separate pr.

Ok, I will fix it soon!

yike21 · 2023-05-10T15:57:43Z

journal

...
I0510 14:50:45.880354       1 configurable.go:125] Aggregate status of object: argoproj.io/v1alpha1, Kind=Workflow test-workflow/sample with configurable interpreter.
I0510 14:50:45.880395       1 recorder.go:103] "events: Update resourceBinding(test-workflow/sample-workflow) with AggregatedStatus successfully." type="Normal" object={Kind:ResourceBinding Namespace:test-workflow Name:sample-workflow UID:489b085f-0aa6-43be-adfe-a229832d9896 APIVersion:work.karmada.io/v1alpha2 ResourceVersion:11982712 FieldPath:} reason="AggregateStatusSucceed"
I0510 14:50:45.880463       1 recorder.go:103] "events: Update resourceBinding(test-workflow/sample-workflow) with AggregatedStatus successfully." type="Normal" object={Kind:Workflow Namespace:test-workflow Name:sample UID:bed73f1e-5178-4a2e-8c79-ef94a573dc53 APIVersion:argoproj.io/v1alpha1 ResourceVersion:11982492 FieldPath:} reason="AggregateStatusSucceed"
I0510 14:50:45.881242       1 dependencies_distributor.go:257] Dropping resource binding(test-workflow/sample-workflow) as the Generation is not changed.
E0510 14:50:45.884087       1 common.go:78] Failed to update resource(argoproj.io/v1alpha1, Resource=workflows/test-workflow/sample), Error: workflows.argoproj.io "sample" not found
E0510 14:50:45.884135       1 controller.go:329] "Reconciler error" err="workflows.argoproj.io \"sample\" not found" controller="resourceBinding_status_controller" namespace="test-workflow" name="sample-workflow" reconcileID=bd8a901c-8f65-4b7e-8441-f6f79a6f4707
I0510 14:50:45.884161       1 rb_status_controller.go:46] Reconciling ResourceBinding test-workflow/sample-workflow.
I0510 14:50:45.893159       1 default.go:72] Default interpreter is not enabled for kind "argoproj.io/v1alpha1, Kind=Workflow" with operation "AggregateStatus".
I0510 14:50:45.893178       1 configurable.go:125] Aggregate status of object: argoproj.io/v1alpha1, Kind=Workflow test-workflow/sample with configurable interpreter.
...

Logs show that:
E0510 14:50:45.884087 1 common.go:78] Failed to update resource(argoproj.io/v1alpha1, Resource=workflows/test-workflow/sample), Error: workflows.argoproj.io "sample" not found,
but the resource workflows/test-workflow/sample is already in Karmada whose .status is not updated. related

yike21 · 2023-05-11T08:54:06Z

.../default/thirdparty/resourcecustomizations/argoproj.io/v1alpha1/Workflow/customizations.yaml

+            if statusItems[i].status ~= nil and statusItems[i].status.progress ~= nil then
+              if progress == '' then
+                progress = progress..statusItems[i].clusterName..'='..statusItems[i].status.progress
+              else
+                progress = progress..','..statusItems[i].clusterName..'='..statusItems[i].status.progress
+              end
+            end


Progress in N/M format. N is number of task complete. M is number of tasks.ref

yike21 · 2023-05-11T08:56:52Z

.../default/thirdparty/resourcecustomizations/argoproj.io/v1alpha1/Workflow/customizations.yaml

+            if statusItems[i].status ~= nil and statusItems[i].status.phase ~= nil then
+              if phase == '' then
+                phase = phase..statusItems[i].clusterName..'='..statusItems[i].status.phase
+              else
+                phase = phase..','..statusItems[i].clusterName..'='..statusItems[i].status.phase
+              end
+            end


Phase a simple, high-level summary of where the workflow is in its lifecycle. It will be "" (Unknown), "Pending", or "Running" before the workflow is completed, and "Succeeded", "Failed" or "Error" once the workflow has completed.ref

yike21 · 2023-05-13T15:37:59Z

journal
ref1 --> ref2
It seems that desiredObj is converted from workObject.Spec.Workload.Manifests which is designed to store .spec of resourceTemplate. Thus .status shouldn't be retain. ref3 shows that .status is updated into workCopy.status.manifestStatuses.

The retention rule may be:

        function Retain(desiredObj, observedObj)
          if observedObj.spec ~= nil then
            desiredObj.spec = observedObj.spec
          end
          return desiredObj
        end

I changed it and test, then I found that the scene go back to the beginning. There are someting unexpected in argo/workflow-controller of member cluster.

yike21 · 2023-05-14T07:32:11Z

The .status of workflow is not a subresources ref, so karmada-control-plane can't update .status of workflow by dynamicClient.Resource(gvr).Namespace(resource.GetNamespace()).UpdateStatus(context.TODO(), newObj, metav1.UpdateOptions{}) : syncBindingStatus -> updateResourceStatus

To make workflow available in karmada, users needs to manually change the CRD of workflow in the karmada-control-plane cluster to make the status subresource available.

The statusReflection operation should be prohibited, and the retention operation is enhanced to retain .status of workflow into work.spec.workload.manifests. @Poor12 @chaunceyjiang

# in member clusters which run workflow controller
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: workflows.argoproj.io
spec:
  group: argoproj.io
  names:
    kind: Workflow
    listKind: WorkflowList
    plural: workflows
    shortNames:
    - wf
    singular: workflow
  scope: Namespaced
  versions:
  - additionalPrinterColumns:
    - description: Status of the workflow
      jsonPath: .status.phase
      name: Status
      type: string
    - description: When the workflow was started
      format: date-time
      jsonPath: .status.startedAt
      name: Age
      type: date
    - description: Human readable message indicating details about why the workflow
        is in this condition.
      jsonPath: .status.message
      name: Message
      type: string
    name: v1alpha1
    schema:
      openAPIV3Schema:
        properties:
          apiVersion:
            type: string
          kind:
            type: string
          metadata:
            type: object
          spec:
            type: object
            x-kubernetes-map-type: atomic
            x-kubernetes-preserve-unknown-fields: true
          status:
            type: object
            x-kubernetes-map-type: atomic
            x-kubernetes-preserve-unknown-fields: true
        required:
        - metadata
        - spec
        type: object
    served: true
    storage: true
    subresources: {}

# in karmada-control-plane
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: workflows.argoproj.io
spec:
  group: argoproj.io
  names:
    kind: Workflow
    listKind: WorkflowList
    plural: workflows
    shortNames:
    - wf
    singular: workflow
  scope: Namespaced
  versions:
  - additionalPrinterColumns:
    - description: Status of the workflow
      jsonPath: .status.phase
      name: Status
      type: string
    - description: When the workflow was started
      format: date-time
      jsonPath: .status.startedAt
      name: Age
      type: date
    - description: Human readable message indicating details about why the workflow
        is in this condition.
      jsonPath: .status.message
      name: Message
      type: string
    name: v1alpha1
    schema:
      openAPIV3Schema:
        properties:
          apiVersion:
            type: string
          kind:
            type: string
          metadata:
            type: object
          spec:
            type: object
            x-kubernetes-map-type: atomic
            x-kubernetes-preserve-unknown-fields: true
          status:
            type: object
            x-kubernetes-map-type: atomic
            x-kubernetes-preserve-unknown-fields: true
        required:
        - metadata
        - spec
        type: object
    served: true
    storage: true
    subresources:
      status: {}

chaunceyjiang · 2023-05-14T16:01:28Z

The .status of workflow is not a subresources ref

https://github.com/argoproj/argo-workflows/blob/master/pkg/apis/workflow/v1alpha1/workflow_types.go#L136

chaunceyjiang · 2023-05-14T16:05:22Z

The statusReflection operation should be prohibited, and the retention operation is enhanced to retain .status of workflow into work.spec.workload.manifests

+1

yike21 · 2023-05-16T11:56:06Z

I reran the test and found new information:

After I change UpdateStatus to Update and build karmada-controller-manager image, then there is no need to add workflow crd's .status as a subresource, the workflow's status can be updated normally.
whether to use reflectStatus or not does not affect the successful operation of workflow.
To make workflow available, users must retain workflow's .status.
After retaining workflow's .status, comparing workflow propagated from karmada to member clusters with running workflow directly in member clusters, I found that the workflow-controller logs in member clusters are not consistent, and there is a difference between nil and {} in some fields of workflow.

# logs of workflow-controller when running workflow directly in member cluster
# kubectl logs workflow-controller-575d78d78f-6krv6 -n argo
time="2023-05-16T05:39:43.597Z" level=info msg="Processing workflow" namespace=test-workflow workflow=sample
time="2023-05-16T05:39:43.599Z" level=info msg="Get configmaps 404"
time="2023-05-16T05:39:43.599Z" level=warning msg="Non-transient error: configmaps \"artifact-repositories\" not found"
time="2023-05-16T05:39:43.599Z" level=info msg="resolved artifact repository" artifactRepositoryRef=default-artifact-repository
time="2023-05-16T05:39:43.599Z" level=info msg="Updated phase  -> Running" namespace=test-workflow workflow=sample
time="2023-05-16T05:39:43.599Z" level=info msg="Steps node sample initialized Running" namespace=test-workflow workflow=sample
time="2023-05-16T05:39:43.599Z" level=info msg="StepGroup node sample-2754495299 initialized Running" namespace=test-workflow workflow=sample
time="2023-05-16T05:39:43.600Z" level=info msg="Pod node sample-410485805 initialized Pending" namespace=test-workflow workflow=sample
time="2023-05-16T05:39:43.613Z" level=info msg="Create events 201"
time="2023-05-16T05:39:43.628Z" level=info msg="Create pods 201"
time="2023-05-16T05:39:43.629Z" level=info msg="Created pod: sample[0].hello (sample-whalesay-410485805)" namespace=test-workflow workflow=sample
time="2023-05-16T05:39:43.629Z" level=info msg="Workflow step group node sample-2754495299 not yet completed" namespace=test-workflow workflow=sample
time="2023-05-16T05:39:43.629Z" level=info msg="TaskSet Reconciliation" namespace=test-workflow workflow=sample
time="2023-05-16T05:39:43.629Z" level=info msg=reconcileAgentPod namespace=test-workflow workflow=sample
time="2023-05-16T05:39:43.629Z" level=info msg="Workflow to be dehydrated" Workflow Size=1664
time="2023-05-16T05:39:43.663Z" level=info msg="Update workflows 200"
time="2023-05-16T05:39:43.664Z" level=info msg="Workflow update successful" namespace=test-workflow phase=Running resourceVersion=11706616 workflow=sample
time="2023-05-16T05:39:43.677Z" level=info msg="Create events 201"
time="2023-05-16T05:39:43.715Z" level=info msg="Create events 201"
time="2023-05-16T05:39:48.320Z" level=info msg="Get leases 200"
time="2023-05-16T05:39:48.330Z" level=info msg="Update leases 200"
time="2023-05-16T05:39:53.336Z" level=info msg="Get leases 200"
time="2023-05-16T05:39:53.346Z" level=info msg="Update leases 200"
time="2023-05-16T05:39:53.631Z" level=info msg="Processing workflow" namespace=test-workflow workflow=sample
time="2023-05-16T05:39:53.631Z" level=info msg="Task-result reconciliation" namespace=test-workflow numObjs=0 workflow=sample
time="2023-05-16T05:39:53.632Z" level=info msg="node changed" namespace=test-workflow new.message=PodInitializing new.phase=Pending new.progress=0/1 nodeID=sample-410485805 old.message= old.phase=Pending old.progress=0/1 workflow=sample
time="2023-05-16T05:39:53.632Z" level=info msg="workflow active pod spec parallelism reached 1/1" namespace=test-workflow workflow=sample
time="2023-05-16T05:39:53.632Z" level=error msg="error in entry template execution" error="Max parallelism reached" namespace=test-workflow workflow=sample
time="2023-05-16T05:39:53.632Z" level=info msg="Workflow to be dehydrated" Workflow Size=1948
time="2023-05-16T05:39:53.651Z" level=info msg="Update workflows 200"
time="2023-05-16T05:39:53.652Z" level=info msg="Workflow update successful" namespace=test-workflow phase=Running resourceVersion=11706669 workflow=sample
time="2023-05-16T05:39:58.352Z" level=info msg="Get leases 200"
time="2023-05-16T05:39:58.361Z" level=info msg="Update leases 200"


time="2023-05-16T05:40:03.367Z" level=info msg="Get leases 200"
time="2023-05-16T05:40:03.377Z" level=info msg="Update leases 200"

time="2023-05-16T05:40:03.653Z" level=info msg="Processing workflow" namespace=test-workflow workflow=sample
time="2023-05-16T05:40:03.654Z" level=info msg="Task-result reconciliation" namespace=test-workflow numObjs=0 workflow=sample
time="2023-05-16T05:40:03.654Z" level=info msg="node changed" namespace=test-workflow new.message= new.phase=Running new.progress=0/1 nodeID=sample-410485805 old.message=PodInitializing old.phase=Pending old.progress=0/1 workflow=sample
time="2023-05-16T05:40:03.654Z" level=info msg="workflow active pod spec parallelism reached 1/1" namespace=test-workflow workflow=sample
time="2023-05-16T05:40:03.655Z" level=error msg="error in entry template execution" error="Max parallelism reached" namespace=test-workflow workflow=sample
time="2023-05-16T05:40:03.655Z" level=info msg="Workflow to be dehydrated" Workflow Size=1937
time="2023-05-16T05:40:03.660Z" level=info msg="cleaning up pod" action=terminateContainers key=test-workflow/sample-whalesay-410485805/terminateContainers
time="2023-05-16T05:40:03.668Z" level=info msg="Update workflows 200"
time="2023-05-16T05:40:03.669Z" level=info msg="Workflow update successful" namespace=test-workflow phase=Running resourceVersion=11706714 workflow=sample
time="2023-05-16T05:40:03.679Z" level=info msg="Create events 201"
time="2023-05-16T05:40:03.684Z" level=info msg="https://10.96.0.1:443/api/v1/namespaces/test-workflow/pods/sample-whalesay-410485805/exec?command=%2Fvar%2Frun%2Fargo%2Fargoexec&command=kill&command=15&command=1&container=wait&stderr=true&stdout=true&tty=false"
time="2023-05-16T05:40:03.730Z" level=info msg="Create pods/exec 500"
time="2023-05-16T05:40:03.737Z" level=info msg="signaled container" container=wait error="unable to upgrade connection: container not found (\"wait\")" namespace=test-workflow pod=sample-whalesay-410485805 stderr="<nil>" stdout="<nil>"
time="2023-05-16T05:40:08.382Z" level=info msg="Get leases 200"
time="2023-05-16T05:40:08.392Z" level=info msg="Update leases 200"
time="2023-05-16T05:40:13.398Z" level=info msg="Get leases 200"
time="2023-05-16T05:40:13.408Z" level=info msg="Update leases 200"

time="2023-05-16T05:40:14.548Z" level=info msg="Processing workflow" namespace=test-workflow workflow=sample
time="2023-05-16T05:40:14.549Z" level=info msg="Task-result reconciliation" namespace=test-workflow numObjs=0 workflow=sample
time="2023-05-16T05:40:14.549Z" level=info msg="node changed" namespace=test-workflow new.message= new.phase=Succeeded new.progress=0/1 nodeID=sample-410485805 old.message= old.phase=Running old.progress=0/1 workflow=sample
time="2023-05-16T05:40:14.550Z" level=info msg="Step group node sample-2754495299 successful" namespace=test-workflow workflow=sample
time="2023-05-16T05:40:14.550Z" level=info msg="node sample-2754495299 phase Running -> Succeeded" namespace=test-workflow workflow=sample
time="2023-05-16T05:40:14.550Z" level=info msg="node sample-2754495299 finished: 2023-05-16 05:40:14.550124491 +0000 UTC" namespace=test-workflow workflow=sample
time="2023-05-16T05:40:14.550Z" level=info msg="Outbound nodes of sample-410485805 is [sample-410485805]" namespace=test-workflow workflow=sample
time="2023-05-16T05:40:14.550Z" level=info msg="Outbound nodes of sample is [sample-410485805]" namespace=test-workflow workflow=sample
time="2023-05-16T05:40:14.550Z" level=info msg="node sample phase Running -> Succeeded" namespace=test-workflow workflow=sample
time="2023-05-16T05:40:14.550Z" level=info msg="node sample finished: 2023-05-16 05:40:14.55029534 +0000 UTC" namespace=test-workflow workflow=sample
time="2023-05-16T05:40:14.550Z" level=info msg="Checking daemoned children of sample" namespace=test-workflow workflow=sample
time="2023-05-16T05:40:14.550Z" level=info msg="TaskSet Reconciliation" namespace=test-workflow workflow=sample
time="2023-05-16T05:40:14.550Z" level=info msg=reconcileAgentPod namespace=test-workflow workflow=sample
time="2023-05-16T05:40:14.550Z" level=info msg="Updated phase Running -> Succeeded" namespace=test-workflow workflow=sample
time="2023-05-16T05:40:14.550Z" level=info msg="Marking workflow completed" namespace=test-workflow workflow=sample
time="2023-05-16T05:40:14.550Z" level=info msg="Checking daemoned children of " namespace=test-workflow workflow=sample
time="2023-05-16T05:40:14.550Z" level=info msg="Workflow to be dehydrated" Workflow Size=2154
time="2023-05-16T05:40:14.556Z" level=info msg="cleaning up pod" action=deletePod key=test-workflow/sample-1340600742-agent/deletePod
time="2023-05-16T05:40:14.561Z" level=info msg="Create events 201"
time="2023-05-16T05:40:14.569Z" level=info msg="Delete pods 404"
time="2023-05-16T05:40:14.571Z" level=info msg="Update workflows 200"
time="2023-05-16T05:40:14.573Z" level=info msg="Workflow update successful" namespace=test-workflow phase=Succeeded resourceVersion=11706761 workflow=sample
time="2023-05-16T05:40:14.578Z" level=info msg="DeleteCollection workflowtaskresults 200"
time="2023-05-16T05:40:14.584Z" level=info msg="cleaning up pod" action=labelPodCompleted key=test-workflow/sample-whalesay-410485805/labelPodCompleted

time="2023-05-16T05:40:14.585Z" level=info msg="Create events 201"
time="2023-05-16T05:40:14.594Z" level=info msg="Create events 201"
time="2023-05-16T05:40:14.601Z" level=info msg="Create events 201"
time="2023-05-16T05:40:14.615Z" level=info msg="Patch pods 200"
time="2023-05-16T05:40:14.839Z" level=info msg="Watch workflows 200"
time="2023-05-16T05:40:18.419Z" level=info msg="Get leases 200"
time="2023-05-16T05:40:18.428Z" level=info msg="Update leases 200"
time="2023-05-16T05:40:23.435Z" level=info msg="Get leases 200"
time="2023-05-16T05:40:23.444Z" level=info msg="Update leases 200"
time="2023-05-16T05:40:28.449Z" level=info msg="Get leases 200"
time="2023-05-16T05:40:28.461Z" level=info msg="Update leases 200"
time="2023-05-16T05:40:28.530Z" level=info msg="Watch pods 200"
time="2023-05-16T05:40:33.470Z" level=info msg="Get leases 200"
time="2023-05-16T05:40:33.479Z" level=info msg="Update leases 200"
time="2023-05-16T05:40:33.737Z" level=info msg="cleaning up pod" action=killContainers key=test-workflow/sample-whalesay-410485805/killContainers
time="2023-05-16T05:40:37.679Z" level=info msg="List workflows 200"
time="2023-05-16T05:40:37.679Z" level=info msg=healthz age=5m0s err="<nil>" instanceID= labelSelector="!workflows.argoproj.io/phase,!workflows.argoproj.io/controller-instanceid" managedNamespace=
time="2023-05-16T05:40:38.485Z" level=info msg="Get leases 200"
time="2023-05-16T05:40:38.495Z" level=info msg="Update leases 200"

# kubectl get workflow sample -n test-workflow -oyaml
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"argoproj.io/v1alpha1","kind":"Workflow","metadata":{"annotations":{},"labels":{"app":"sample"},"name":"sample","namespace":"test-workflow"},"spec":{"entrypoint":"example","parallelism":1,"templates":[{"name":"example","steps":[[{"name":"hello","template":"whalesay"}]]},{"container":{"args":["hello world"],"command":["cowsay"],"image":"docker/whalesay:latest"},"name":"whalesay"}],"tolerations":[{"effect":"NoSchedule","key":"key","operation":"Equal","value":"value"}]}}
    workflows.argoproj.io/pod-name-format: v2
  creationTimestamp: "2023-05-16T05:39:43Z"
  generation: 5
  labels:
    app: sample
    workflows.argoproj.io/completed: "true"
    workflows.argoproj.io/phase: Succeeded
  name: sample
  namespace: test-workflow
  resourceVersion: "11706761"
  uid: 4ad4d984-39ca-4112-86ce-b753c3ce5edf
spec:
  arguments: {}
  entrypoint: example
  parallelism: 1
  templates:
  - inputs: {}
    metadata: {}
    name: example
    outputs: {}
    steps:
    - - arguments: {}
        name: hello
        template: whalesay
  - container:
      args:
      - hello world
      command:
      - cowsay
      image: docker/whalesay:latest
      name: ""
      resources: {}
    inputs: {}
    metadata: {}
    name: whalesay
    outputs: {}
  tolerations:
  - effect: NoSchedule
    key: key
    value: value
status:
  artifactGCStatus:
    notSpecified: true
  artifactRepositoryRef:
    artifactRepository: {}
    default: true
  conditions:
  - status: "False"
    type: PodRunning
  - status: "True"
    type: Completed
  finishedAt: "2023-05-16T05:40:14Z"
  nodes:
    sample:
      children:
      - sample-2754495299
      displayName: sample
      finishedAt: "2023-05-16T05:40:14Z"
      id: sample
      name: sample
      outboundNodes:
      - sample-410485805
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 19
        memory: 19
      startedAt: "2023-05-16T05:39:43Z"
      templateName: example
      templateScope: local/sample
      type: Steps
    sample-410485805:
      boundaryID: sample
      displayName: hello
      finishedAt: "2023-05-16T05:40:03Z"
      hostNodeName: member3-control-plane
      id: sample-410485805
      name: sample[0].hello
      outputs:
        exitCode: "0"
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 19
        memory: 19
      startedAt: "2023-05-16T05:39:43Z"
      templateName: whalesay
      templateScope: local/sample
      type: Pod
    sample-2754495299:
      boundaryID: sample
      children:
      - sample-410485805
      displayName: '[0]'
      finishedAt: "2023-05-16T05:40:14Z"
      id: sample-2754495299
      name: sample[0]
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 19
        memory: 19
      startedAt: "2023-05-16T05:39:43Z"
      templateScope: local/sample
      type: StepGroup
  phase: Succeeded
  progress: 1/1
  resourcesDuration:
    cpu: 19
    memory: 19
  startedAt: "2023-05-16T05:39:43Z"

# logs of workflow-controller when propagating workflow from karmada
# kubectl logs workflow-controller-575d78d78f-6krv6 -n argo
time="2023-05-16T05:31:54.206Z" level=info msg="Processing workflow" namespace=test-workflow workflow=sample
time="2023-05-16T05:31:54.208Z" level=info msg="Get configmaps 404"
time="2023-05-16T05:31:54.208Z" level=warning msg="Non-transient error: configmaps \"artifact-repositories\" not found"
time="2023-05-16T05:31:54.208Z" level=info msg="resolved artifact repository" artifactRepositoryRef=default-artifact-repository
time="2023-05-16T05:31:54.208Z" level=info msg="Updated phase  -> Running" namespace=test-workflow workflow=sample
time="2023-05-16T05:31:54.208Z" level=info msg="Steps node sample initialized Running" namespace=test-workflow workflow=sample
time="2023-05-16T05:31:54.208Z" level=info msg="StepGroup node sample-2754495299 initialized Running" namespace=test-workflow workflow=sample
time="2023-05-16T05:31:54.208Z" level=info msg="Pod node sample-410485805 initialized Pending" namespace=test-workflow workflow=sample
time="2023-05-16T05:31:54.218Z" level=info msg="Create events 201"
time="2023-05-16T05:31:54.232Z" level=info msg="Create pods 201"
time="2023-05-16T05:31:54.233Z" level=info msg="Created pod: sample[0].hello (sample-whalesay-410485805)" namespace=test-workflow workflow=sample
time="2023-05-16T05:31:54.233Z" level=info msg="Workflow step group node sample-2754495299 not yet completed" namespace=test-workflow workflow=sample
time="2023-05-16T05:31:54.233Z" level=info msg="TaskSet Reconciliation" namespace=test-workflow workflow=sample
time="2023-05-16T05:31:54.233Z" level=info msg=reconcileAgentPod namespace=test-workflow workflow=sample
time="2023-05-16T05:31:54.233Z" level=info msg="Workflow to be dehydrated" Workflow Size=2982
time="2023-05-16T05:31:54.257Z" level=info msg="Update workflows 200"
time="2023-05-16T05:31:54.257Z" level=info msg="Workflow update successful" namespace=test-workflow phase=Running resourceVersion=11704729 workflow=sample
time="2023-05-16T05:31:54.275Z" level=info msg="Create events 201"
time="2023-05-16T05:31:54.288Z" level=info msg="Create events 201"
time="2023-05-16T05:31:56.658Z" level=info msg="Get leases 200"
time="2023-05-16T05:31:56.810Z" level=info msg="Update leases 200"
time="2023-05-16T05:32:01.816Z" level=info msg="Get leases 200"
time="2023-05-16T05:32:01.825Z" level=info msg="Update leases 200"
time="2023-05-16T05:32:04.235Z" level=info msg="Processing workflow" namespace=test-workflow workflow=sample
time="2023-05-16T05:32:04.236Z" level=info msg="Task-result reconciliation" namespace=test-workflow numObjs=0 workflow=sample
time="2023-05-16T05:32:04.236Z" level=info msg="node changed" namespace=test-workflow new.message=PodInitializing new.phase=Pending new.progress=0/1 nodeID=sample-410485805 old.message= old.phase=Pending old.progress=0/1 workflow=sample
time="2023-05-16T05:32:04.236Z" level=info msg="workflow active pod spec parallelism reached 1/1" namespace=test-workflow workflow=sample
time="2023-05-16T05:32:04.236Z" level=error msg="error in entry template execution" error="Max parallelism reached" namespace=test-workflow workflow=sample
time="2023-05-16T05:32:04.237Z" level=info msg="Workflow to be dehydrated" Workflow Size=3263
time="2023-05-16T05:32:04.269Z" level=info msg="Update workflows 200"
time="2023-05-16T05:32:04.271Z" level=info msg="Workflow update successful" namespace=test-workflow phase=Running resourceVersion=11704783 workflow=sample
time="2023-05-16T05:32:06.831Z" level=info msg="Get leases 200"
time="2023-05-16T05:32:06.839Z" level=info msg="Update leases 200"


time="2023-05-16T05:32:08.222Z" level=info msg="Watch configmaps 200"
time="2023-05-16T05:32:11.845Z" level=info msg="Get leases 200"
time="2023-05-16T05:32:11.859Z" level=info msg="Update leases 200"

time="2023-05-16T05:32:14.272Z" level=info msg="Processing workflow" namespace=test-workflow workflow=sample
time="2023-05-16T05:32:14.272Z" level=info msg="Task-result reconciliation" namespace=test-workflow numObjs=0 workflow=sample
time="2023-05-16T05:32:14.273Z" level=info msg="node unchanged" namespace=test-workflow nodeID=sample-410485805 workflow=sample
time="2023-05-16T05:32:14.273Z" level=info msg="workflow active pod spec parallelism reached 1/1" namespace=test-workflow workflow=sample
time="2023-05-16T05:32:14.273Z" level=error msg="error in entry template execution" error="Max parallelism reached" namespace=test-workflow workflow=sample
time="2023-05-16T05:32:16.864Z" level=info msg="Get leases 200"
time="2023-05-16T05:32:16.876Z" level=info msg="Update leases 200"
time="2023-05-16T05:32:19.963Z" level=info msg="Watch cronworkflows 200"
time="2023-05-16T05:32:21.882Z" level=info msg="Get leases 200"
time="2023-05-16T05:32:21.892Z" level=info msg="Update leases 200"

time="2023-05-16T05:32:25.619Z" level=info msg="Processing workflow" namespace=test-workflow workflow=sample
time="2023-05-16T05:32:25.619Z" level=info msg="Task-result reconciliation" namespace=test-workflow numObjs=0 workflow=sample
time="2023-05-16T05:32:25.619Z" level=info msg="node changed" namespace=test-workflow new.message= new.phase=Succeeded new.progress=0/1 nodeID=sample-410485805 old.message=PodInitializing old.phase=Pending old.progress=0/1 workflow=sample
time="2023-05-16T05:32:25.620Z" level=info msg="Step group node sample-2754495299 successful" namespace=test-workflow workflow=sample
time="2023-05-16T05:32:25.620Z" level=info msg="node sample-2754495299 phase Running -> Succeeded" namespace=test-workflow workflow=sample
time="2023-05-16T05:32:25.620Z" level=info msg="node sample-2754495299 finished: 2023-05-16 05:32:25.620027219 +0000 UTC" namespace=test-workflow workflow=sample
time="2023-05-16T05:32:25.620Z" level=info msg="Outbound nodes of sample-410485805 is [sample-410485805]" namespace=test-workflow workflow=sample
time="2023-05-16T05:32:25.620Z" level=info msg="Outbound nodes of sample is [sample-410485805]" namespace=test-workflow workflow=sample
time="2023-05-16T05:32:25.620Z" level=info msg="node sample phase Running -> Succeeded" namespace=test-workflow workflow=sample
time="2023-05-16T05:32:25.620Z" level=info msg="node sample finished: 2023-05-16 05:32:25.620070022 +0000 UTC" namespace=test-workflow workflow=sample
time="2023-05-16T05:32:25.620Z" level=info msg="Checking daemoned children of sample" namespace=test-workflow workflow=sample
time="2023-05-16T05:32:25.620Z" level=info msg="TaskSet Reconciliation" namespace=test-workflow workflow=sample
time="2023-05-16T05:32:25.620Z" level=info msg=reconcileAgentPod namespace=test-workflow workflow=sample
time="2023-05-16T05:32:25.620Z" level=info msg="Updated phase Running -> Succeeded" namespace=test-workflow workflow=sample
time="2023-05-16T05:32:25.620Z" level=info msg="Marking workflow completed" namespace=test-workflow workflow=sample
time="2023-05-16T05:32:25.620Z" level=info msg="Checking daemoned children of " namespace=test-workflow workflow=sample
time="2023-05-16T05:32:25.620Z" level=info msg="Workflow to be dehydrated" Workflow Size=3469
time="2023-05-16T05:32:25.624Z" level=info msg="Create events 201"
time="2023-05-16T05:32:25.625Z" level=info msg="cleaning up pod" action=deletePod key=test-workflow/sample-1340600742-agent/deletePod
time="2023-05-16T05:32:25.629Z" level=info msg="Delete pods 404"
time="2023-05-16T05:32:25.630Z" level=info msg="Update workflows 200"
time="2023-05-16T05:32:25.630Z" level=info msg="Workflow update successful" namespace=test-workflow phase=Succeeded resourceVersion=11704871 workflow=sample
time="2023-05-16T05:32:25.632Z" level=info msg="DeleteCollection workflowtaskresults 200"
time="2023-05-16T05:32:25.637Z" level=info msg="cleaning up pod" action=labelPodCompleted key=test-workflow/sample-whalesay-410485805/labelPodCompleted

time="2023-05-16T05:32:25.643Z" level=info msg="Create events 201"
time="2023-05-16T05:32:25.651Z" level=info msg="Patch pods 200"
time="2023-05-16T05:32:25.661Z" level=info msg="Create events 201"
time="2023-05-16T05:32:25.666Z" level=info msg="Create events 201"
time="2023-05-16T05:32:25.672Z" level=info msg="Create events 201"
time="2023-05-16T05:32:26.897Z" level=info msg="Get leases 200"
time="2023-05-16T05:32:26.908Z" level=info msg="Update leases 200"
time="2023-05-16T05:32:31.915Z" level=info msg="Get leases 200"
time="2023-05-16T05:32:31.925Z" level=info msg="Update leases 200"
time="2023-05-16T05:32:36.930Z" level=info msg="Get leases 200"
time="2023-05-16T05:32:36.940Z" level=info msg="Update leases 200"
time="2023-05-16T05:32:37.680Z" level=info msg="List workflows 200"
time="2023-05-16T05:32:37.680Z" level=info msg=healthz age=5m0s err="<nil>" instanceID= labelSelector="!workflows.argoproj.io/phase,!workflows.argoproj.io/controller-instanceid" managedNamespace=
time="2023-05-16T05:32:41.952Z" level=info msg="Get leases 200"
time="2023-05-16T05:32:41.962Z" level=info msg="Update leases 200"

# in member cluster
# kubectl get workflow sample -n test-workflow -oyaml
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"argoproj.io/v1alpha1","kind":"Workflow","metadata":{"annotations":{},"labels":{"app":"sample"},"name":"sample","namespace":"test-workflow"},"spec":{"entrypoint":"example","parallelism":1,"templates":[{"name":"example","steps":[[{"name":"hello","template":"whalesay"}]]},{"container":{"args":["hello world"],"command":["cowsay"],"image":"docker/whalesay:latest"},"name":"whalesay"}],"tolerations":[{"effect":"NoSchedule","key":"key","operation":"Equal","value":"value"}]}}
    resourcebinding.karmada.io/name: sample-workflow
    resourcebinding.karmada.io/namespace: test-workflow
    resourcetemplate.karmada.io/managed-annotations: kubectl.kubernetes.io/last-applied-configuration,resourcebinding.karmada.io/name,resourcebinding.karmada.io/namespace,resourcetemplate.karmada.io/managed-annotations,resourcetemplate.karmada.io/managed-labels,resourcetemplate.karmada.io/uid
    resourcetemplate.karmada.io/managed-labels: app,propagationpolicy.karmada.io/name,propagationpolicy.karmada.io/namespace,resourcebinding.karmada.io/key,work.karmada.io/name,work.karmada.io/namespace
    resourcetemplate.karmada.io/uid: 0696fd78-2559-4980-9b97-673070407e63
    workflows.argoproj.io/pod-name-format: v2
  creationTimestamp: "2023-05-16T05:53:28Z"
  generation: 5
  labels:
    app: sample
    propagationpolicy.karmada.io/name: sample
    propagationpolicy.karmada.io/namespace: test-workflow
    resourcebinding.karmada.io/key: 578768c846
    work.karmada.io/name: sample-dd88569b
    work.karmada.io/namespace: karmada-es-member3
    workflows.argoproj.io/completed: "true"
    workflows.argoproj.io/phase: Succeeded
  name: sample
  namespace: test-workflow
  resourceVersion: "11709940"
  uid: 9819b523-d313-4fa3-86cb-c1c2076b5b5b
spec:
  entrypoint: example
  parallelism: 1
  templates:
  - name: example
    steps:
    - - name: hello
        template: whalesay
  - container:
      args:
      - hello world
      command:
      - cowsay
      image: docker/whalesay:latest
      name: ""
    name: whalesay
  tolerations:
  - effect: NoSchedule
    key: key
    value: value
status:
  artifactGCStatus:
    notSpecified: true
  artifactRepositoryRef:
    default: true
  conditions:
  - status: "False"
    type: PodRunning
  - status: "True"
    type: Completed
  finishedAt: "2023-05-16T05:53:38Z"
  nodes:
    sample:
      children:
      - sample-2754495299
      displayName: sample
      finishedAt: "2023-05-16T05:53:38Z"
      id: sample
      name: sample
      outboundNodes:
      - sample-410485805
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 4
        memory: 4
      startedAt: "2023-05-16T05:53:28Z"
      templateName: example
      templateScope: local/sample
      type: Steps
    sample-410485805:
      boundaryID: sample
      displayName: hello
      finishedAt: "2023-05-16T05:53:34Z"
      hostNodeName: member3-control-plane
      id: sample-410485805
      name: sample[0].hello
      outputs:
        exitCode: "0"
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 4
        memory: 4
      startedAt: "2023-05-16T05:53:28Z"
      templateName: whalesay
      templateScope: local/sample
      type: Pod
    sample-2754495299:
      boundaryID: sample
      children:
      - sample-410485805
      displayName: '[0]'
      finishedAt: "2023-05-16T05:53:38Z"
      id: sample-2754495299
      name: sample[0]
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 4
        memory: 4
      startedAt: "2023-05-16T05:53:28Z"
      templateScope: local/sample
      type: StepGroup
  phase: Succeeded
  progress: 1/1
  resourcesDuration:
    cpu: 4
    memory: 4
  startedAt: "2023-05-16T05:53:28Z"

Signed-off-by: yike21 <yike21@qq.com>

yike21 · 2023-05-23T09:02:02Z

The test-report based on karmadactl interpret is below:

### Check the customizations in file
$ karmadactl interpret -f customizations.yaml --check
I0523 16:54:55.298761  883282 deploy.go:75] No default release version found. build version: version.Info{GitVersion:"", GitCommit:"c8720e81c7f4badb2a2e07ed7218391b3ea7d912", GitTreeState:"dirty", BuildDate:"2023-04-12T09:43:27Z", GoVersion:"go1.19.7", Compiler:"gc", Platform:"linux/amd64"}
I0523 16:54:55.299172  883282 enable_option.go:63] No default release version found. build version: version.Info{GitVersion:"", GitCommit:"c8720e81c7f4badb2a2e07ed7218391b3ea7d912", GitTreeState:"dirty", BuildDate:"2023-04-12T09:43:27Z", GoVersion:"go1.19.7", Compiler:"gc", Platform:"linux/amd64"}
I0523 16:54:55.338668  883282 cmdinit.go:149] No default release version found. build version: version.Info{GitVersion:"", GitCommit:"c8720e81c7f4badb2a2e07ed7218391b3ea7d912", GitTreeState:"dirty", BuildDate:"2023-04-12T09:43:27Z", GoVersion:"go1.19.7", Compiler:"gc", Platform:"linux/amd64"}
I0523 16:54:55.338968  883282 register.go:160] No default release version found. build version: version.Info{GitVersion:"", GitCommit:"c8720e81c7f4badb2a2e07ed7218391b3ea7d912", GitTreeState:"dirty", BuildDate:"2023-04-12T09:43:27Z", GoVersion:"go1.19.7", Compiler:"gc", Platform:"linux/amd64"}
-----------------------------------
SOURCE: declarative-configuration-workflow
TARGET: argoproj.io/v1alpha1 Workflow   
RULERS:
    Retain:                             PASS
    InterpretReplica:                   PASS
    ReviseReplica:                      PASS
    InterpretStatus:                    UNSET
    AggregateStatus:                    UNSET
    InterpretHealth:                    PASS
    InterpretDependency:                PASS

### Execute the retention rule
$ karmadactl interpret -f customizations.yaml --operation retain --desired-file testdata/desired-workflow.yaml --observed-file testdata/observed-workflow.yaml 
I0523 16:55:22.380282  883664 deploy.go:75] No default release version found. build version: version.Info{GitVersion:"", GitCommit:"c8720e81c7f4badb2a2e07ed7218391b3ea7d912", GitTreeState:"dirty", BuildDate:"2023-04-12T09:43:27Z", GoVersion:"go1.19.7", Compiler:"gc", Platform:"linux/amd64"}
I0523 16:55:22.380443  883664 enable_option.go:63] No default release version found. build version: version.Info{GitVersion:"", GitCommit:"c8720e81c7f4badb2a2e07ed7218391b3ea7d912", GitTreeState:"dirty", BuildDate:"2023-04-12T09:43:27Z", GoVersion:"go1.19.7", Compiler:"gc", Platform:"linux/amd64"}
I0523 16:55:22.392216  883664 cmdinit.go:149] No default release version found. build version: version.Info{GitVersion:"", GitCommit:"c8720e81c7f4badb2a2e07ed7218391b3ea7d912", GitTreeState:"dirty", BuildDate:"2023-04-12T09:43:27Z", GoVersion:"go1.19.7", Compiler:"gc", Platform:"linux/amd64"}
I0523 16:55:22.392349  883664 register.go:160] No default release version found. build version: version.Info{GitVersion:"", GitCommit:"c8720e81c7f4badb2a2e07ed7218391b3ea7d912", GitTreeState:"dirty", BuildDate:"2023-04-12T09:43:27Z", GoVersion:"go1.19.7", Compiler:"gc", Platform:"linux/amd64"}
---
# [1/1] retained:
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
    labels:
        app: sample
    name: sample
    namespace: test-workflow
spec:
    entrypoint: volumes-existing-example
    imagePullSecrets:
        - name: my-secret
    parallelism: 2
    serviceAccountName: my-service-account
    suspend: true
    templates:
        - name: volumes-existing-example
          steps:
            - - name: hello
                template: whalesay
        - container:
            args:
                - hello world
            command:
                - cowsay
            image: docker/whalesay:latest
            resources:
                limits:
                    memory: 1Gi
                requests:
                    memory: 1Mi
            volumeMounts:
                - mountPath: /mnt/vol
                  name: workdir
          name: whalesay
    tolerations:
        - effect: NoSchedule
          key: key
          operation: Equal
          value: value
    volumes:
        - name: workdir
          persistentVolumeClaim:
            claimName: my-existing-volume
status:
    artifactGCStatus:
        notSpecified: true
    artifactRepositoryRef:
        default: true
    conditions:
        - status: "False"
          type: PodRunning
        - status: "True"
          type: Completed
    finishedAt: "2023-04-20T11:26:47Z"
    nodes:
        sample:
            children:
                - sample-2754495299
            displayName: sample
            finishedAt: "2023-04-20T11:26:47Z"
            id: sample
            name: sample
            outboundNodes:
                - sample-410485805
            phase: Succeeded
            progress: 1/1
            resourcesDuration:
                cpu: 19
                memory: 18
            startedAt: "2023-04-20T11:26:17Z"
            templateName: volumes-existing-example
            templateScope: local/sample
            type: Steps
        sample-410485805:
            boundaryID: sample
            displayName: hello
            finishedAt: "2023-04-20T11:26:38Z"
            hostNodeName: member1-control-plane
            id: sample-410485805
            name: sample[0].hello
            outputs:
                exitCode: "0"
            phase: Succeeded
            progress: 1/1
            resourcesDuration:
                cpu: 19
                memory: 18
            startedAt: "2023-04-20T11:26:17Z"
            templateName: whalesay
            templateScope: local/sample
            type: Pod
        sample-2754495299:
            boundaryID: sample
            children:
                - sample-410485805
            displayName: '[0]'
            finishedAt: "2023-04-20T11:26:47Z"
            id: sample-2754495299
            name: sample[0]
            phase: Succeeded
            progress: 1/1
            resourcesDuration:
                cpu: 19
                memory: 18
            startedAt: "2023-04-20T11:26:17Z"
            templateScope: local/sample
            type: StepGroup
    phase: Succeeded
    progress: 1/1
    resourcesDuration:
        cpu: 19
        memory: 18
    startedAt: "2023-04-20T11:26:17Z"

### Execute the replicaResource rule
$ karmadactl interpret -f customizations.yaml --operation interpretReplica --observed-file testdata/observed-workflow.yaml 
I0523 16:56:31.555243  884583 deploy.go:75] No default release version found. build version: version.Info{GitVersion:"", GitCommit:"c8720e81c7f4badb2a2e07ed7218391b3ea7d912", GitTreeState:"dirty", BuildDate:"2023-04-12T09:43:27Z", GoVersion:"go1.19.7", Compiler:"gc", Platform:"linux/amd64"}
I0523 16:56:31.555426  884583 enable_option.go:63] No default release version found. build version: version.Info{GitVersion:"", GitCommit:"c8720e81c7f4badb2a2e07ed7218391b3ea7d912", GitTreeState:"dirty", BuildDate:"2023-04-12T09:43:27Z", GoVersion:"go1.19.7", Compiler:"gc", Platform:"linux/amd64"}
I0523 16:56:31.567790  884583 cmdinit.go:149] No default release version found. build version: version.Info{GitVersion:"", GitCommit:"c8720e81c7f4badb2a2e07ed7218391b3ea7d912", GitTreeState:"dirty", BuildDate:"2023-04-12T09:43:27Z", GoVersion:"go1.19.7", Compiler:"gc", Platform:"linux/amd64"}
I0523 16:56:31.568217  884583 register.go:160] No default release version found. build version: version.Info{GitVersion:"", GitCommit:"c8720e81c7f4badb2a2e07ed7218391b3ea7d912", GitTreeState:"dirty", BuildDate:"2023-04-12T09:43:27Z", GoVersion:"go1.19.7", Compiler:"gc", Platform:"linux/amd64"}
---
# [1/2] replica:
1
---
# [2/2] requires:
nodeClaim:
    tolerations:
        - effect: NoSchedule
          key: key
          value: value

### Execute the replicaRevision rule
$ karmadactl interpret -f customizations.yaml --operation reviseReplica --observed-file testdata/observed-workflow.yaml --desired-replica 4
I0523 16:56:51.016943  884851 deploy.go:75] No default release version found. build version: version.Info{GitVersion:"", GitCommit:"c8720e81c7f4badb2a2e07ed7218391b3ea7d912", GitTreeState:"dirty", BuildDate:"2023-04-12T09:43:27Z", GoVersion:"go1.19.7", Compiler:"gc", Platform:"linux/amd64"}
I0523 16:56:51.017105  884851 enable_option.go:63] No default release version found. build version: version.Info{GitVersion:"", GitCommit:"c8720e81c7f4badb2a2e07ed7218391b3ea7d912", GitTreeState:"dirty", BuildDate:"2023-04-12T09:43:27Z", GoVersion:"go1.19.7", Compiler:"gc", Platform:"linux/amd64"}
I0523 16:56:51.026940  884851 cmdinit.go:149] No default release version found. build version: version.Info{GitVersion:"", GitCommit:"c8720e81c7f4badb2a2e07ed7218391b3ea7d912", GitTreeState:"dirty", BuildDate:"2023-04-12T09:43:27Z", GoVersion:"go1.19.7", Compiler:"gc", Platform:"linux/amd64"}
I0523 16:56:51.027044  884851 register.go:160] No default release version found. build version: version.Info{GitVersion:"", GitCommit:"c8720e81c7f4badb2a2e07ed7218391b3ea7d912", GitTreeState:"dirty", BuildDate:"2023-04-12T09:43:27Z", GoVersion:"go1.19.7", Compiler:"gc", Platform:"linux/amd64"}
---
# [1/1] revised:
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
    labels:
        app: sample
    name: sample
    namespace: test-workflow
spec:
    entrypoint: volumes-existing-example
    imagePullSecrets:
        - name: my-secret
    parallelism: 4
    serviceAccountName: my-service-account
    suspend: true
    templates:
        - name: volumes-existing-example
          steps:
            - - name: hello
                template: whalesay
        - container:
            args:
                - hello world
            command:
                - cowsay
            image: docker/whalesay:latest
            resources:
                limits:
                    memory: 1Gi
                requests:
                    memory: 1Mi
            volumeMounts:
                - mountPath: /mnt/vol
                  name: workdir
          name: whalesay
    tolerations:
        - effect: NoSchedule
          key: key
          operation: Equal
          value: value
    volumes:
        - name: workdir
          persistentVolumeClaim:
            claimName: my-existing-volume
status:
    artifactGCStatus:
        notSpecified: true
    artifactRepositoryRef:
        default: true
    conditions:
        - status: "False"
          type: PodRunning
        - status: "True"
          type: Completed
    finishedAt: "2023-04-20T11:26:47Z"
    nodes:
        sample:
            children:
                - sample-2754495299
            displayName: sample
            finishedAt: "2023-04-20T11:26:47Z"
            id: sample
            name: sample
            outboundNodes:
                - sample-410485805
            phase: Succeeded
            progress: 1/1
            resourcesDuration:
                cpu: 19
                memory: 18
            startedAt: "2023-04-20T11:26:17Z"
            templateName: volumes-existing-example
            templateScope: local/sample
            type: Steps
        sample-410485805:
            boundaryID: sample
            displayName: hello
            finishedAt: "2023-04-20T11:26:38Z"
            hostNodeName: member1-control-plane
            id: sample-410485805
            name: sample[0].hello
            outputs:
                exitCode: "0"
            phase: Succeeded
            progress: 1/1
            resourcesDuration:
                cpu: 19
                memory: 18
            startedAt: "2023-04-20T11:26:17Z"
            templateName: whalesay
            templateScope: local/sample
            type: Pod
        sample-2754495299:
            boundaryID: sample
            children:
                - sample-410485805
            displayName: '[0]'
            finishedAt: "2023-04-20T11:26:47Z"
            id: sample-2754495299
            name: sample[0]
            phase: Succeeded
            progress: 1/1
            resourcesDuration:
                cpu: 19
                memory: 18
            startedAt: "2023-04-20T11:26:17Z"
            templateScope: local/sample
            type: StepGroup
    phase: Succeeded
    progress: 1/1
    resourcesDuration:
        cpu: 19
        memory: 18
    startedAt: "2023-04-20T11:26:17Z"

### Execute the healthInterpretation rule
$ karmadactl interpret -f customizations.yaml --operation interpretHealth --observed-file testdata/observed-workflow.yaml 
I0523 16:57:40.453487  885517 deploy.go:75] No default release version found. build version: version.Info{GitVersion:"", GitCommit:"c8720e81c7f4badb2a2e07ed7218391b3ea7d912", GitTreeState:"dirty", BuildDate:"2023-04-12T09:43:27Z", GoVersion:"go1.19.7", Compiler:"gc", Platform:"linux/amd64"}
I0523 16:57:40.453668  885517 enable_option.go:63] No default release version found. build version: version.Info{GitVersion:"", GitCommit:"c8720e81c7f4badb2a2e07ed7218391b3ea7d912", GitTreeState:"dirty", BuildDate:"2023-04-12T09:43:27Z", GoVersion:"go1.19.7", Compiler:"gc", Platform:"linux/amd64"}
I0523 16:57:40.464938  885517 cmdinit.go:149] No default release version found. build version: version.Info{GitVersion:"", GitCommit:"c8720e81c7f4badb2a2e07ed7218391b3ea7d912", GitTreeState:"dirty", BuildDate:"2023-04-12T09:43:27Z", GoVersion:"go1.19.7", Compiler:"gc", Platform:"linux/amd64"}
I0523 16:57:40.465092  885517 register.go:160] No default release version found. build version: version.Info{GitVersion:"", GitCommit:"c8720e81c7f4badb2a2e07ed7218391b3ea7d912", GitTreeState:"dirty", BuildDate:"2023-04-12T09:43:27Z", GoVersion:"go1.19.7", Compiler:"gc", Platform:"linux/amd64"}
---
# [1/1] healthy:
true

### Execute the dependencyInterpretation rule
$ karmadactl interpret -f customizations.yaml --operation interpretDependency --observed-file testdata/observed-workflow.yaml 
I0523 16:57:57.480740  885709 deploy.go:75] No default release version found. build version: version.Info{GitVersion:"", GitCommit:"c8720e81c7f4badb2a2e07ed7218391b3ea7d912", GitTreeState:"dirty", BuildDate:"2023-04-12T09:43:27Z", GoVersion:"go1.19.7", Compiler:"gc", Platform:"linux/amd64"}
I0523 16:57:57.481213  885709 enable_option.go:63] No default release version found. build version: version.Info{GitVersion:"", GitCommit:"c8720e81c7f4badb2a2e07ed7218391b3ea7d912", GitTreeState:"dirty", BuildDate:"2023-04-12T09:43:27Z", GoVersion:"go1.19.7", Compiler:"gc", Platform:"linux/amd64"}
I0523 16:57:57.513695  885709 cmdinit.go:149] No default release version found. build version: version.Info{GitVersion:"", GitCommit:"c8720e81c7f4badb2a2e07ed7218391b3ea7d912", GitTreeState:"dirty", BuildDate:"2023-04-12T09:43:27Z", GoVersion:"go1.19.7", Compiler:"gc", Platform:"linux/amd64"}
I0523 16:57:57.514098  885709 register.go:160] No default release version found. build version: version.Info{GitVersion:"", GitCommit:"c8720e81c7f4badb2a2e07ed7218391b3ea7d912", GitTreeState:"dirty", BuildDate:"2023-04-12T09:43:27Z", GoVersion:"go1.19.7", Compiler:"gc", Platform:"linux/amd64"}
---
# [1/1] dependencies:
- apiVersion: v1
  kind: PersistentVolumeClaim
  name: my-existing-volume
  namespace: test-workflow
- apiVersion: v1
  kind: Secret
  name: my-secret
  namespace: test-workflow
- apiVersion: v1
  kind: ServiceAccount
  name: my-service-account
  namespace: test-workflow

Poor12 · 2023-05-23T09:14:36Z

/hold cancel

yike21 · 2023-05-23T09:16:22Z

.../default/thirdparty/resourcecustomizations/argoproj.io/v1alpha1/Workflow/customizations.yaml

+          if observedObj.status ~= nil then
+            desiredObj.status = observedObj.status
+          end


Based on the discussion at the community meeting, it is decided to retain the entire .status. And since workflow does not support subresource of status, the AggregateStatus operation is not implemented.
/cc @Poor12 @chaunceyjiang

Poor12 · 2023-05-23T09:18:40Z

Thanks all for hard work.
/lgtm
/approve

karmada-bot · 2023-05-23T09:18:46Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Poor12

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~pkg/resourceinterpreter/default/thirdparty/OWNERS~~ [Poor12]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

karmada-bot added the kind/cleanup Categorizes issue or PR as related to cleaning up code, process, or technical debt. label Apr 20, 2023

karmada-bot requested review from chaunceyjiang and Poor12 April 20, 2023 12:47

karmada-bot added the size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. label Apr 20, 2023

karmada-bot assigned chaunceyjiang Apr 21, 2023

karmada-bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Apr 24, 2023

yike21 mentioned this pull request Apr 24, 2023

Add third-party resources into the Resource Interpreter framework #3331

Closed

17 tasks

chaunceyjiang reviewed Apr 25, 2023

View reviewed changes

yike21 force-pushed the bundle-argo-workflow-resources branch from 32974bc to f7787d5 Compare May 10, 2023 15:46

yike21 force-pushed the bundle-argo-workflow-resources branch from f7787d5 to 43ade98 Compare May 10, 2023 16:17

yike21 commented May 11, 2023

View reviewed changes

yike21 force-pushed the bundle-argo-workflow-resources branch from 43ade98 to d24a133 Compare May 14, 2023 07:19

This was referenced May 16, 2023

Do we need to change the method from UpdateStatus to Update？ #3551

Closed

Do we need to optimize the method c.ObjectWatcher.NeedsUpdate? #3564

Closed

add customizations for argoproj.io/v1alpha1/Workflow

9471d16

Signed-off-by: yike21 <yike21@qq.com>

yike21 force-pushed the bundle-argo-workflow-resources branch from d24a133 to 9471d16 Compare May 23, 2023 08:59

karmada-bot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels May 23, 2023

karmada-bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label May 23, 2023

yike21 commented May 23, 2023

View reviewed changes

karmada-bot requested a review from chaunceyjiang May 23, 2023 09:16

karmada-bot assigned Poor12 May 23, 2023

karmada-bot added the lgtm Indicates that a PR is ready to be merged. label May 23, 2023

karmada-bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label May 23, 2023

karmada-bot merged commit ba5945f into karmada-io:master May 23, 2023
12 checks passed

yike21 deleted the bundle-argo-workflow-resources branch May 23, 2023 10:01

yike21 mentioned this pull request Jun 4, 2023

REQUEST: New membership for yike21 karmada-io/community#33

Closed

8 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add customizations for argoproj.io/v1alpha1/Workflow #3438

add customizations for argoproj.io/v1alpha1/Workflow #3438

yike21 commented Apr 20, 2023 •

edited

codecov-commenter commented Apr 20, 2023 •

edited

chaunceyjiang commented Apr 21, 2023

chaunceyjiang commented Apr 24, 2023

yike21 commented Apr 24, 2023

chaunceyjiang commented Apr 24, 2023

chaunceyjiang Apr 25, 2023

yike21 Apr 25, 2023

yike21 commented May 10, 2023 •

edited

yike21 May 11, 2023

yike21 May 11, 2023

yike21 commented May 13, 2023 •

edited

yike21 commented May 14, 2023 •

edited

chaunceyjiang commented May 14, 2023 •

edited

chaunceyjiang commented May 14, 2023

yike21 commented May 16, 2023 •

edited

yike21 commented May 23, 2023

Poor12 commented May 23, 2023

yike21 May 23, 2023 •

edited

chaunceyjiang May 23, 2023

Poor12 commented May 23, 2023

karmada-bot commented May 23, 2023

add customizations for argoproj.io/v1alpha1/Workflow #3438

add customizations for argoproj.io/v1alpha1/Workflow #3438

Conversation

yike21 commented Apr 20, 2023 • edited

codecov-commenter commented Apr 20, 2023 • edited

Codecov Report

chaunceyjiang commented Apr 21, 2023

chaunceyjiang commented Apr 24, 2023

yike21 commented Apr 24, 2023

chaunceyjiang commented Apr 24, 2023

chaunceyjiang Apr 25, 2023

Choose a reason for hiding this comment

yike21 Apr 25, 2023

Choose a reason for hiding this comment

yike21 commented May 10, 2023 • edited

yike21 May 11, 2023

Choose a reason for hiding this comment

yike21 May 11, 2023

Choose a reason for hiding this comment

yike21 commented May 13, 2023 • edited

yike21 commented May 14, 2023 • edited

chaunceyjiang commented May 14, 2023 • edited

chaunceyjiang commented May 14, 2023

yike21 commented May 16, 2023 • edited

yike21 commented May 23, 2023

Poor12 commented May 23, 2023

yike21 May 23, 2023 • edited

Choose a reason for hiding this comment

chaunceyjiang May 23, 2023

Choose a reason for hiding this comment

Poor12 commented May 23, 2023

karmada-bot commented May 23, 2023

yike21 commented Apr 20, 2023 •

edited

codecov-commenter commented Apr 20, 2023 •

edited

yike21 commented May 10, 2023 •

edited

yike21 commented May 13, 2023 •

edited

yike21 commented May 14, 2023 •

edited

chaunceyjiang commented May 14, 2023 •

edited

yike21 commented May 16, 2023 •

edited

yike21 May 23, 2023 •

edited