Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Memoization doesn't work in templates that are dag or use retries #10426

Closed
2 of 3 tasks
gmeshkov opened this issue Jan 30, 2023 · 3 comments · Fixed by #12491
Closed
2 of 3 tasks

Memoization doesn't work in templates that are dag or use retries #10426

gmeshkov opened this issue Jan 30, 2023 · 3 comments · Fixed by #12491
Assignees
@Joibel
Labels
area/memoization area/retryStrategy Template-level retryStrategy area/templates/dag P1 High priority. All bugs with >=5 thumbs up that aren’t P0, plus: Any other bugs deemed high priority type/bug

Comments

@gmeshkov
Copy link

gmeshkov commented Jan 30, 2023 •
edited

Pre-requisites

  • I have double-checked my configuration
  • I can confirm the issues exists when I tested with :latest
  • I'd like to contribute the fix myself (see contributing guide)

What happened/what you expected to happen?

When I add memoization to a step with retryStrategy, the memoization doesn't work.
Checked with retryStrategy: Always and the default OnFailure value.

Also, the memoization doesn't work for a step which is a dag containing other steps.

From the logs of the controller I can conclude that the entry in ConfigMap is not created for some reason after the retry/dag steps finish.

Version

3.4.4

Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

# This test shows that memoization doesn't work for templates with retryStrategy and dag templates
# `regular-1` and `regular-2` tasks show that memoization works for regular templates
# `with-retries-1` and `with-retries-2` show that if you add the retryStrategy to the template, memoization no longer works
# `with-dag-1` and `with-dag-2` show that memoization doesn't work if a memoized template contains multiple steps (is a dag)
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: memoized-workflow-test
spec:
  entrypoint: main
#  podGC:
#    strategy: OnPodCompletion
  templates:
  - name: main
    dag:
      tasks:
      - name: regular-1
        template: run
        arguments:
          parameters:
          - name: id
            value: 1
          - name: cache-key
            value: '{{workflow.name}}'
      - name: regular-2
        template: run
        depends: regular-1.Succeeded
        arguments:
          parameters:
          - name: id
            value: 2
          - name: cache-key
            value: '{{workflow.name}}'
      - name: with-retries-1
        template: run-with-retries
        arguments:
          parameters:
          - name: id
            value: 3
          - name: cache-key
            value: '{{workflow.name}}'
      - name: with-retries-2
        template: run-with-retries
        depends: with-retries-1.Succeeded
        arguments:
          parameters:
          - name: id
            value: 4
          - name: cache-key
            value: '{{workflow.name}}'
      - name: with-dag-1
        template: run-with-dag
        arguments:
          parameters:
          - name: id
            value: 5
          - name: cache-key
            value: '{{workflow.name}}'
      - name: with-dag-2
        template: run-with-dag
        depends: with-dag-1.Succeeded
        arguments:
          parameters:
          - name: id
            value: 6
          - name: cache-key
            value: '{{workflow.name}}'

  - name: run
    inputs:
      parameters:
      - name: id
      - name: cache-key
    script:
      image: ubuntu:22.04
      command: [bash]
      source: |
        sleep 30
        echo result: {{inputs.parameters.id}}
    memoize:
      key: "regular-{{inputs.parameters.cache-key}}"
      cache:
        configMap:
          name: memoization-test-cache

  - name: run-with-retries
    inputs:
      parameters:
      - name: id
      - name: cache-key
    script:
      image: ubuntu:22.04
      command: [bash]
      source: |
        sleep 30
        echo result: {{inputs.parameters.id}}
    memoize:
      key: "retry-{{inputs.parameters.cache-key}}"
      cache:
        configMap:
          name: memoization-test-cache
    retryStrategy:
      limit: '1'
      retryPolicy: Always

  - name: run-raw
    inputs:
      parameters:
      - name: id
      - name: cache-key
    script:
      image: ubuntu:22.04
      command: [bash]
      source: |
        sleep 30
        echo result: {{inputs.parameters.id}}

  - name: run-with-dag
    inputs:
      parameters:
      - name: id
      - name: cache-key
    dag:
      tasks:
      - name: run-raw-step
        template: run-raw
        arguments:
          parameters:
          - name: id
            value: '{{inputs.parameters.id}}'
          - name: cache-key
            value: '{{inputs.parameters.cache-key}}'
    memoize:
      key: "dag-{{inputs.parameters.cache-key}}"
      cache:
        configMap:
          name: memoization-test-cache

Logs from the workflow controller

time="2023-01-30T13:36:36.431Z" level=info msg="Processing workflow" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:36:36.461Z" level=info msg="Updated phase  -> Running" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:36:36.461Z" level=info msg="DAG node memoized-workflow-testwttf6 initialized Running" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:36:36.461Z" level=info msg="All of node memoized-workflow-testwttf6.regular-1 dependencies [] completed" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:36:36.565Z" level=info msg="Pod node memoized-workflow-testwttf6-2651566501 initialized Pending" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:36:36.624Z" level=info msg="Created pod: memoized-workflow-testwttf6.regular-1 (memoized-workflow-testwttf6-run-2651566501)" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:36:36.624Z" level=info msg="All of node memoized-workflow-testwttf6.with-dag-1 dependencies [] completed" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:36:36.661Z" level=info msg="DAG node memoized-workflow-testwttf6-3689047750 initialized Pending" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:36:36.661Z" level=info msg="All of node memoized-workflow-testwttf6.with-dag-1.run-raw-step dependencies [] completed" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:36:36.661Z" level=info msg="Pod node memoized-workflow-testwttf6-1860721307 initialized Pending" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:36:36.773Z" level=info msg="Created pod: memoized-workflow-testwttf6.with-dag-1.run-raw-step (memoized-workflow-testwttf6-run-raw-1860721307)" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:36:36.773Z" level=info msg="All of node memoized-workflow-testwttf6.with-retries-1 dependencies [] completed" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:36:36.962Z" level=info msg="Retry node memoized-workflow-testwttf6-412563200 initialized Pending" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:36:36.962Z" level=info msg="Pod node memoized-workflow-testwttf6-643257443 initialized Pending" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:36:37.126Z" level=info msg="Created pod: memoized-workflow-testwttf6.with-retries-1(0) (memoized-workflow-testwttf6-run-with-retries-643257443)" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:36:37.126Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:36:37.126Z" level=info msg=reconcileAgentPod namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:36:37.137Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=314195848 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:36:46.625Z" level=info msg="Processing workflow" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:36:46.626Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=0 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:36:46.626Z" level=info msg="node changed" namespace=argo new.message= new.phase=Running new.progress=0/1 nodeID=memoized-workflow-testwttf6-643257443 old.message= old.phase=Pending old.progress=0/1 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:36:46.626Z" level=info msg="node changed" namespace=argo new.message= new.phase=Running new.progress=0/1 nodeID=memoized-workflow-testwttf6-2651566501 old.message= old.phase=Pending old.progress=0/1 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:36:46.626Z" level=info msg="node changed" namespace=argo new.message= new.phase=Running new.progress=0/1 nodeID=memoized-workflow-testwttf6-1860721307 old.message= old.phase=Pending old.progress=0/1 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:36:46.627Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:36:46.627Z" level=info msg=reconcileAgentPod namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:36:46.638Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=314197033 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:37:23.984Z" level=info msg="Processing workflow" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:37:23.985Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=3 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:37:23.985Z" level=info msg="task-result changed" namespace=argo nodeID=memoized-workflow-testwttf6-643257443 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:37:23.985Z" level=info msg="task-result changed" namespace=argo nodeID=memoized-workflow-testwttf6-1860721307 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:37:23.985Z" level=info msg="task-result changed" namespace=argo nodeID=memoized-workflow-testwttf6-2651566501 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:37:23.985Z" level=info msg="node changed" namespace=argo new.message= new.phase=Running new.progress=0/1 nodeID=memoized-workflow-testwttf6-643257443 old.message= old.phase=Running old.progress=0/1 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:37:23.986Z" level=info msg="node unchanged" namespace=argo nodeID=memoized-workflow-testwttf6-2651566501 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:37:23.986Z" level=info msg="node unchanged" namespace=argo nodeID=memoized-workflow-testwttf6-1860721307 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:37:23.986Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:37:23.987Z" level=info msg=reconcileAgentPod namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:37:23.991Z" level=info msg="cleaning up pod" action=terminateContainers key=argo/memoized-workflow-testwttf6-run-with-retries-643257443/terminateContainers
time="2023-01-30T13:37:23.998Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=314200811 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:37:33.993Z" level=info msg="Processing workflow" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:37:33.994Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=3 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:37:33.995Z" level=info msg="node unchanged" namespace=argo nodeID=memoized-workflow-testwttf6-2651566501 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:37:33.995Z" level=info msg="node unchanged" namespace=argo nodeID=memoized-workflow-testwttf6-1860721307 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:37:33.995Z" level=info msg="node changed" namespace=argo new.message= new.phase=Succeeded new.progress=0/1 nodeID=memoized-workflow-testwttf6-643257443 old.message= old.phase=Running old.progress=0/1 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:37:33.995Z" level=info msg="node memoized-workflow-testwttf6-412563200 phase Pending -> Succeeded" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:37:33.995Z" level=info msg="node memoized-workflow-testwttf6-412563200 finished: 2023-01-30 13:37:33.99591818 +0000 UTC" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:37:33.995Z" level=info msg="All of node memoized-workflow-testwttf6.with-retries-2 dependencies [with-retries-1] completed" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:37:34.002Z" level=info msg="Retry node memoized-workflow-testwttf6-462896057 initialized Pending" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:37:34.002Z" level=info msg="Pod node memoized-workflow-testwttf6-2914119264 initialized Pending" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:37:34.019Z" level=info msg="Created pod: memoized-workflow-testwttf6.with-retries-2(0) (memoized-workflow-testwttf6-run-with-retries-2914119264)" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:37:34.019Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:37:34.019Z" level=info msg=reconcileAgentPod namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:37:34.031Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=314201878 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:37:44.020Z" level=info msg="Processing workflow" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:37:44.021Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=3 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:37:44.021Z" level=info msg="node unchanged" namespace=argo nodeID=memoized-workflow-testwttf6-1860721307 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:37:44.021Z" level=info msg="node changed" namespace=argo new.message= new.phase=Running new.progress=0/1 nodeID=memoized-workflow-testwttf6-2914119264 old.message= old.phase=Pending old.progress=0/1 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:37:44.021Z" level=info msg="node unchanged" namespace=argo nodeID=memoized-workflow-testwttf6-643257443 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:37:44.021Z" level=info msg="node unchanged" namespace=argo nodeID=memoized-workflow-testwttf6-2651566501 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:37:44.022Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:37:44.022Z" level=info msg=reconcileAgentPod namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:37:44.034Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=314202945 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:37:53.992Z" level=info msg="cleaning up pod" action=killContainers key=argo/memoized-workflow-testwttf6-run-with-retries-643257443/killContainers
time="2023-01-30T13:37:57.960Z" level=info msg="Processing workflow" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:37:57.961Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=3 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:37:57.961Z" level=info msg="node unchanged" namespace=argo nodeID=memoized-workflow-testwttf6-2914119264 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:37:57.961Z" level=info msg="node changed" namespace=argo new.message= new.phase=Succeeded new.progress=0/1 nodeID=memoized-workflow-testwttf6-1860721307 old.message= old.phase=Running old.progress=0/1 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:37:57.961Z" level=info msg="node unchanged" namespace=argo nodeID=memoized-workflow-testwttf6-643257443 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:37:57.961Z" level=info msg="node changed" namespace=argo new.message= new.phase=Running new.progress=0/1 nodeID=memoized-workflow-testwttf6-2651566501 old.message= old.phase=Running old.progress=0/1 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:37:57.962Z" level=info msg="Outbound nodes of memoized-workflow-testwttf6-3689047750 set to [memoized-workflow-testwttf6-1860721307]" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:37:57.962Z" level=info msg="node memoized-workflow-testwttf6-3689047750 phase Pending -> Succeeded" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:37:57.962Z" level=info msg="node memoized-workflow-testwttf6-3689047750 finished: 2023-01-30 13:37:57.962392225 +0000 UTC" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:37:57.962Z" level=info msg="Checking daemoned children of memoized-workflow-testwttf6-3689047750" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:37:57.962Z" level=info msg="All of node memoized-workflow-testwttf6.with-dag-2 dependencies [with-dag-1] completed" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:37:57.967Z" level=info msg="cleaning up pod" action=terminateContainers key=argo/memoized-workflow-testwttf6-run-2651566501/terminateContainers
time="2023-01-30T13:37:57.967Z" level=info msg="DAG node memoized-workflow-testwttf6-3672270131 initialized Pending" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:37:57.968Z" level=info msg="All of node memoized-workflow-testwttf6.with-dag-2.run-raw-step dependencies [] completed" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:37:57.968Z" level=info msg="Pod node memoized-workflow-testwttf6-3956060252 initialized Pending" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:37:57.987Z" level=info msg="Created pod: memoized-workflow-testwttf6.with-dag-2.run-raw-step (memoized-workflow-testwttf6-run-raw-3956060252)" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:37:57.988Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:37:57.988Z" level=info msg=reconcileAgentPod namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:37:58.000Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=314204100 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:07.990Z" level=info msg="Processing workflow" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:07.992Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=3 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:07.992Z" level=info msg="node unchanged" namespace=argo nodeID=memoized-workflow-testwttf6-1860721307 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:07.992Z" level=info msg="node unchanged" namespace=argo nodeID=memoized-workflow-testwttf6-2914119264 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:07.992Z" level=info msg="node unchanged" namespace=argo nodeID=memoized-workflow-testwttf6-643257443 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:07.992Z" level=info msg="node changed" namespace=argo new.message= new.phase=Succeeded new.progress=0/1 nodeID=memoized-workflow-testwttf6-2651566501 old.message= old.phase=Running old.progress=0/1 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:07.992Z" level=info msg="Saving ConfigMap cache entry" key=regular-memoized-workflow-testwttf6 name=memoization-test-cache namespace=argo nodeId=memoized-workflow-testwttf6-2651566501
time="2023-01-30T13:38:08.003Z" level=info msg="node changed" namespace=argo new.message= new.phase=Running new.progress=0/1 nodeID=memoized-workflow-testwttf6-3956060252 old.message= old.phase=Pending old.progress=0/1 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:08.003Z" level=info msg="All of node memoized-workflow-testwttf6.regular-2 dependencies [regular-1] completed" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:08.015Z" level=info msg="Pod node memoized-workflow-testwttf6-2601233644 initialized Pending" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:08.016Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:08.016Z" level=info msg=reconcileAgentPod namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:08.029Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=314205057 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:20.067Z" level=info msg="Processing workflow" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:20.069Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=4 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:20.069Z" level=info msg="task-result changed" namespace=argo nodeID=memoized-workflow-testwttf6-2914119264 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:20.069Z" level=info msg="node changed" namespace=argo new.message= new.phase=Succeeded new.progress=0/1 nodeID=memoized-workflow-testwttf6-2914119264 old.message= old.phase=Running old.progress=0/1 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:20.070Z" level=info msg="node unchanged" namespace=argo nodeID=memoized-workflow-testwttf6-2651566501 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:20.070Z" level=info msg="node unchanged" namespace=argo nodeID=memoized-workflow-testwttf6-1860721307 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:20.070Z" level=info msg="node unchanged" namespace=argo nodeID=memoized-workflow-testwttf6-3956060252 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:20.070Z" level=info msg="node unchanged" namespace=argo nodeID=memoized-workflow-testwttf6-643257443 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:20.071Z" level=info msg="node memoized-workflow-testwttf6-462896057 phase Pending -> Succeeded" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:20.071Z" level=info msg="node memoized-workflow-testwttf6-462896057 finished: 2023-01-30 13:38:20.071530014 +0000 UTC" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:20.071Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:20.071Z" level=info msg=reconcileAgentPod namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:20.085Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=314206244 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:27.968Z" level=info msg="cleaning up pod" action=killContainers key=argo/memoized-workflow-testwttf6-run-2651566501/killContainers
time="2023-01-30T13:38:30.087Z" level=info msg="Processing workflow" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:30.089Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=4 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:30.089Z" level=info msg="node unchanged" namespace=argo nodeID=memoized-workflow-testwttf6-3956060252 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:30.089Z" level=info msg="node unchanged" namespace=argo nodeID=memoized-workflow-testwttf6-2651566501 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:30.089Z" level=info msg="node unchanged" namespace=argo nodeID=memoized-workflow-testwttf6-643257443 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:30.089Z" level=info msg="node unchanged" namespace=argo nodeID=memoized-workflow-testwttf6-1860721307 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:30.089Z" level=info msg="node unchanged" namespace=argo nodeID=memoized-workflow-testwttf6-2914119264 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:30.090Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:30.090Z" level=info msg=reconcileAgentPod namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:42.387Z" level=info msg="Processing workflow" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:42.389Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=5 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:42.389Z" level=info msg="task-result changed" namespace=argo nodeID=memoized-workflow-testwttf6-3956060252 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:42.389Z" level=info msg="node changed" namespace=argo new.message= new.phase=Succeeded new.progress=0/1 nodeID=memoized-workflow-testwttf6-3956060252 old.message= old.phase=Running old.progress=0/1 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:42.389Z" level=info msg="node unchanged" namespace=argo nodeID=memoized-workflow-testwttf6-1860721307 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:42.390Z" level=info msg="node unchanged" namespace=argo nodeID=memoized-workflow-testwttf6-643257443 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:42.390Z" level=info msg="node unchanged" namespace=argo nodeID=memoized-workflow-testwttf6-2651566501 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:42.390Z" level=info msg="node unchanged" namespace=argo nodeID=memoized-workflow-testwttf6-2914119264 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:42.391Z" level=info msg="Outbound nodes of memoized-workflow-testwttf6-3672270131 set to [memoized-workflow-testwttf6-3956060252]" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:42.391Z" level=info msg="node memoized-workflow-testwttf6-3672270131 phase Pending -> Succeeded" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:42.391Z" level=info msg="node memoized-workflow-testwttf6-3672270131 finished: 2023-01-30 13:38:42.391258267 +0000 UTC" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:42.391Z" level=info msg="Checking daemoned children of memoized-workflow-testwttf6-3672270131" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:42.391Z" level=info msg="Outbound nodes of memoized-workflow-testwttf6 set to [memoized-workflow-testwttf6-2601233644 memoized-workflow-testwttf6-3956060252 memoized-workflow-testwttf6-2914119264]" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:42.391Z" level=info msg="node memoized-workflow-testwttf6 phase Running -> Succeeded" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:42.391Z" level=info msg="node memoized-workflow-testwttf6 finished: 2023-01-30 13:38:42.391603635 +0000 UTC" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:42.391Z" level=info msg="Checking daemoned children of memoized-workflow-testwttf6" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:42.391Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:42.391Z" level=info msg=reconcileAgentPod namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:42.391Z" level=info msg="Updated phase Running -> Succeeded" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:42.391Z" level=info msg="Marking workflow completed" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:42.391Z" level=info msg="Marking workflow as pending archiving" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:42.391Z" level=info msg="Checking daemoned children of " namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:42.397Z" level=info msg="cleaning up pod" action=deletePod key=argo/memoized-workflow-testwttf6-1340600742-agent/deletePod
time="2023-01-30T13:38:42.405Z" level=info msg="Workflow update successful" namespace=argo phase=Succeeded resourceVersion=314208768 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:42.643Z" level=info msg="archiving workflow" namespace=argo uid=3b66145e-a4ba-4045-adec-c9ada69395fa workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:42.669Z" level=info msg="Queueing Succeeded workflow argo/memoized-workflow-testwttf6 for delete in 2h0m0s due to TTL"
time="2023-01-30T13:38:47.643Z" level=info msg="cleaning up pod" action=deletePod key=argo/memoized-workflow-testwttf6-run-2651566501/deletePod
time="2023-01-30T13:38:47.643Z" level=info msg="cleaning up pod" action=deletePod key=argo/memoized-workflow-testwttf6-run-raw-1860721307/deletePod
time="2023-01-30T13:38:47.643Z" level=info msg="cleaning up pod" action=deletePod key=argo/memoized-workflow-testwttf6-run-with-retries-2914119264/deletePod
time="2023-01-30T13:38:47.643Z" level=info msg="cleaning up pod" action=deletePod key=argo/memoized-workflow-testwttf6-run-with-retries-643257443/deletePod
time="2023-01-30T13:38:47.657Z" level=info msg="cleaning up pod" action=deletePod key=argo/memoized-workflow-testwttf6-run-raw-3956060252/deletePod

Logs from in your workflow's wait container

No resources found in argo namespace.
@sarabala1979 sarabala1979 added the P3 Low priority label Jan 30, 2023
@stale
Copy link

stale bot commented Mar 25, 2023

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If this is a mentoring request, please provide an update here. Thank you for your contributions.

@stale stale bot added the problem/stale This has not had a response in some time label Mar 25, 2023
@moleskin-smile moleskin-smile mentioned this issue Mar 28, 2023
Closed
3 tasks
@JPZ13 JPZ13 added P2 Important. All bugs with >=3 thumbs up that aren’t P0 or P1, plus: Any other bugs deemed important and removed P3 Low priority labels Mar 30, 2023
@stale stale bot removed problem/stale This has not had a response in some time labels Mar 30, 2023
@JPZ13 JPZ13 mentioned this issue Jun 8, 2023
Open
3 tasks
Joibel added a commit to Joibel/argo-workflows that referenced this issue Jun 23, 2023
@Joibel
docs: Link memoization and work-avoidance
2903c39 
Despite argoproj#10769 and argoproj#10426 both having examples of memoization not
working with the examples having no output, no-one has picked up on
this.

To address this improve the documentation for memoization and
work-avoidance, linking the two ideas and pointing people who want to
skip steps towards work-avoidance unless they are really doing what
memoization was designed to do.

Issue argoproj#10426 is problematic in that some steps get memoized when
perhaps they should't, so this commit shouldn't close it.

Fixes argoproj#10769

Signed-off-by: Alan Clucas <alan@clucas.org>
@Joibel Joibel mentioned this issue Jun 23, 2023
Merged
@juliev0 juliev0 added P1 High priority. All bugs with >=5 thumbs up that aren’t P0, plus: Any other bugs deemed high priority and removed P2 Important. All bugs with >=3 thumbs up that aren’t P0 or P1, plus: Any other bugs deemed important labels Jun 29, 2023
Joibel added a commit to Joibel/argo-workflows that referenced this issue Jul 17, 2023
@Joibel
feat: Allow memoization without outputs
8b71792 
Please note: I am not totally in favour of this change, it makes
memoization work without outputs and I'm not conviced this is actually
healthy, but it's what people seem to expect. See the linked issues in
the fixes list below.

This change will cause memoization to work for all step and dag tasks
even without outputs. Note: They were working semi-erroneously for
some dag tasks due to implicit outputs before this change.

Fixes argoproj#11280 (raised to cover this desire)
Fixes argoproj#10769 (already closed by documentation)
Partially addresses argoproj#10426: Dags will memoize now, but retries still
won't

Signed-off-by: Alan Clucas <alan@clucas.org>
Joibel added a commit to Joibel/argo-workflows that referenced this issue Jul 17, 2023
@Joibel
feat: Allow memoization without outputs
b52c4b7 
Please note: I am not totally in favour of this change, it makes
memoization work without outputs and I'm not conviced this is actually
healthy, but it's what people seem to expect. See the linked issues in
the fixes list below.

This change will cause memoization to work for all step and dag tasks
even without outputs. Note: They were working semi-erroneously for
some dag tasks due to implicit outputs before this change.

Fixes argoproj#11280 (raised to cover this desire)
Fixes argoproj#10769 (already closed by documentation)
Partially addresses argoproj#10426: Dags will memoize now, but retries still
won't

Signed-off-by: Alan Clucas <alan@clucas.org>
@Joibel Joibel mentioned this issue Jul 17, 2023
Merged
@Joibel
Copy link
Member

Joibel commented Jul 25, 2023

This should be improved but not fixed in 3.5, via #11379. The dag case now memoizes, but the retries don't.

@stale
Copy link

stale bot commented Sep 17, 2023

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

@stale stale bot added the problem/stale This has not had a response in some time label Sep 17, 2023
@terrytangyuan terrytangyuan removed the problem/stale This has not had a response in some time label Sep 20, 2023
@agilgur5 agilgur5 added area/templates/dag area/retry-manual Manual workflow "Retry" Action (API/CLI/UI). See retryStrategy for template-level retries labels Nov 2, 2023
This was referenced Jan 9, 2024
Closed
Merged
@Joibel Joibel self-assigned this Jan 9, 2024
terrytangyuan pushed a commit that referenced this issue Jan 19, 2024
@shuangkun
fix: cache configmap don't create with workflow has retrystrategy. Fixes
46c1324 
: #12490 #10426 (#12491)
isubasinghe pushed a commit to isubasinghe/argo-workflows that referenced this issue Feb 20, 2024
@shuangkun @isubasinghe
fix: cache configmap don't create with workflow has retrystrategy. Fixes
1afc2fa 
: argoproj#12490 argoproj#10426 (argoproj#12491)
isubasinghe pushed a commit to isubasinghe/argo-workflows that referenced this issue Feb 27, 2024
@shuangkun @isubasinghe
fix: cache configmap don't create with workflow has retrystrategy. Fixes
7e2a34f 
: argoproj#12490 argoproj#10426 (argoproj#12491)
@isubasinghe isubasinghe mentioned this issue Feb 27, 2024
Open
isubasinghe pushed a commit to isubasinghe/argo-workflows that referenced this issue Feb 28, 2024
@shuangkun @isubasinghe
fix: cache configmap don't create with workflow has retrystrategy. Fixes
8c75a72 
: argoproj#12490 argoproj#10426 (argoproj#12491)

Signed-off-by: Isitha Subasinghe <isubasinghe@student.unimelb.edu.au>
@agilgur5 agilgur5 added area/retryStrategy Template-level retryStrategy and removed area/retry-manual Manual workflow "Retry" Action (API/CLI/UI). See retryStrategy for template-level retries labels Apr 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Joibel Joibel

Labels
area/memoization area/retryStrategy Template-level retryStrategy area/templates/dag P1 High priority. All bugs with >=5 thumbs up that aren’t P0, plus: Any other bugs deemed high priority type/bug
Projects
None yet
Milestone
No milestone
8 participants
@Joibel @terrytangyuan @agilgur5 @JPZ13 @juliev0 @sarabala1979 @caelan-io @gmeshkov