Argo hangs when handling multiple with parameter steps (some empty)

[cchanley2003]

What happened:

This is running argo version 2.4.3.

I'll provide templates but basically argo get stuck if I submit a nested workflow with multiple parameters. Not sure of all the specifics of what is going wrong but in this case the first step produces multiple json results and uses withParameters across multiple steps.

In this case list-one is empty and list-two has values. It correctly starts echo2 and skips echo but then hangs.

What you expected to happen:
If I run this same setup using argo 2.4.0-rc1 this works as expected which is that the workflow finishes running through all steps in echo2. Running with 2.4.3 the workflow gets stuck in running forever.

How to reproduce it (as minimally and precisely as possible):

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: argo-test-
spec:
  entrypoint: process-steps 
  templates:
  - name: process-steps
    parallelism: 1
    steps:
    - - name: create-list 
        templateRef:
          name: create-list-template 
          template: create-list 
    - - name: echo 
        templateRef:
          name: echo-steps-template 
          template: echo-steps 
        withParam: "{{steps.create-list.outputs.parameters.list-one}}"
        arguments:
          parameters:
          - name: first 
            value: "{{item.first}}"
          - name: second 
            value: "{{item.second}}"
    - - name: echo2 
        templateRef:
          name: echo-steps-template 
          template: echo-steps 
        withParam: "{{steps.create-list.outputs.parameters.list-two}}"
        arguments:
          parameters:
          - name: first 
            value: "{{item.second}}"
          - name: second 
            value: "{{item.first}}"

apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
  name: create-list-template
spec:
  templates:
  - name: create-list
    outputs:
      parameters:
      - name: list-one
        valueFrom:
          path: /tmp/list-one.json
      - name: list-two
        valueFrom:
          path: /tmp/list-two.json
    script:
      image: groovy
      command: [groovy]
      securityContext:
        runAsUser: 0
      source: |

        import groovy.json.JsonOutput
        import java.nio.file.Paths

        // Convert yaml to object
        class Item {
          String first 
          String second 
        }

        List<Item> items = []
        items << new Item(first: "One", second: "Two")
        items << new Item(first: "Uno", second: "Dos")
        def json = JsonOutput.toJson(items)
        new File("/tmp/list-two.json").write(json)
       
        def empty = []
        json = JsonOutput.toJson(empty)
        new File("/tmp/list-one.json").write(json)

apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
  name: echo-steps-template
spec:
  templates:
  - name: echo-steps 
    inputs:
      parameters:
      - name: first 
      - name: second 
    steps:
    - - name: order 
        templateRef:
          name: echo-template
          template: echo 
        arguments:
          parameters:
          - name: first 
            value: "{{inputs.parameters.first}}"
          - name: second 
            value: "{{inputs.parameters.second}}"
    - - name: reverse 
        templateRef:
          name: echo-template
          template: echo 
        arguments:
          parameters:
          - name: second 
            value: "{{inputs.parameters.first}}"
          - name:  first 
            value: "{{inputs.parameters.second}}"

apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
  name: echo-template
spec:
  templates:
  - name: echo
    retryStrategy:
      limit: 4
    inputs:
      parameters:
      - name: first
      - name: second
    container:
      image: alpine:latest
      command: [sh, -c]
      args: ["echo result was: {{inputs.parameters.first}} {{inputs.parameters.second}}"]

Anything else we need to know?:
Environment:

• Argo version:

2.4.3

• Kubernetes version :

1.15.1

Other debugging information (if applicable):

• workflow result:
  It will be in this state indefinitely. Running but not doing the next step which would be reverse.

Name:                argo-test-kcbm6
Namespace:           argo
ServiceAccount:      default
Status:              Running
Created:             Mon Dec 16 16:45:54 -0700 (7 minutes ago)
Started:             Mon Dec 16 16:45:54 -0700 (7 minutes ago)
Duration:            7 minutes 21 seconds

STEP                                                                     PODNAME                     DURATION  MESSAGE
 ● argo-test-kcbm6 (process-steps)                                                                             
 ├---✔ create-list (create-list-template/create-list)                    argo-test-kcbm6-837177584   6s        
 └-----● echo2(0:first:One,second:Two) (echo-steps-template/echo-steps)                                        
       └---✔ order(0) (echo-template/echo)                               argo-test-kcbm6-4074628442  5s 

• executor logs:
  All logs look normal. No errors.

[alexec]

@dtaniwaki @sarabala1979 what is best for bugs with workflow templates?

[cchanley2003]

Just to clarify. I ran this with 2.4.0-rc1 with workflow templates and it works. So it worked with templates prior to the 2.4.0 official release. For us we have been still running rc1 (we really like the template concept) because of the previous issues with parameter passing and nesting that was introduced with the 2.4.0 official release.

[dtaniwaki]

Sorry for the late reply. I'll check the issue soon.

[dtaniwaki]

I tried the example above without workflow templates as below and I can reproduce the same issue.

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: argo-test-
spec:
  entrypoint: process-steps
  templates:
  - name: process-steps
    parallelism: 1
    steps:
    - - name: create-list
        template: create-list
    - - name: echo
        template: echo-steps
        withParam: "{{steps.create-list.outputs.parameters.list-one}}"
        arguments:
          parameters:
          - name: first
            value: "{{item.first}}"
          - name: second
            value: "{{item.second}}"
    - - name: echo2
        template: echo-steps
        withParam: "{{steps.create-list.outputs.parameters.list-two}}"
        arguments:
          parameters:
          - name: first
            value: "{{item.second}}"
          - name: second
            value: "{{item.first}}"
  - name: create-list
    outputs:
      parameters:
      - name: list-one
        valueFrom:
          path: /tmp/list-one.json
      - name: list-two
        valueFrom:
          path: /tmp/list-two.json
    script:
      image: groovy
      command: [groovy]
      securityContext:
        runAsUser: 0
      source: |

        import groovy.json.JsonOutput
        import java.nio.file.Paths

        // Convert yaml to object
        class Item {
          String first
          String second
        }

        List<Item> items = []
        items << new Item(first: "One", second: "Two")
        items << new Item(first: "Uno", second: "Dos")
        def json = JsonOutput.toJson(items)
        new File("/tmp/list-two.json").write(json)

        def empty = []
        json = JsonOutput.toJson(empty)
        new File("/tmp/list-one.json").write(json)
  - name: echo-steps
    inputs:
      parameters:
      - name: first
      - name: second
    steps:
    - - name: order
        template: echo
        arguments:
          parameters:
          - name: first
            value: "{{inputs.parameters.first}}"
          - name: second
            value: "{{inputs.parameters.second}}"
    - - name: reverse
        template: echo
        arguments:
          parameters:
          - name: second
            value: "{{inputs.parameters.first}}"
          - name:  first
            value: "{{inputs.parameters.second}}"
  - name: echo
    retryStrategy:
      limit: 4
    inputs:
      parameters:
      - name: first
      - name: second
    container:
      image: alpine:latest
      command: [sh, -c]
      args: ["echo result was: {{inputs.parameters.first}} {{inputs.parameters.second}}"]

Here's the result.

Name:                argo-test-q44pg
Namespace:           default
ServiceAccount:      default
Status:              Running
Created:             Mon Jan 13 20:00:38 +0900 (1 minute ago)
Started:             Mon Jan 13 20:00:38 +0900 (1 minute ago)
Duration:            1 minute 55 seconds

STEP                                                 PODNAME                     DURATION  MESSAGE
 ● argo-test-q44pg (process-steps)
 ├---✔ create-list (create-list)                     argo-test-q44pg-2721427299  27s
 └-----● echo2(0:first:One,second:Two) (echo-steps)
       └---✔ order(0) (echo)                         argo-test-q44pg-3247452505  40s

I tried it without empty output with the following diff on my manifest.

@@ -59,8 +59,8 @@
         def json = JsonOutput.toJson(items)
         new File("/tmp/list-two.json").write(json)

-        def empty = []
-        json = JsonOutput.toJson(empty)
+        // def empty = []
+        // json = JsonOutput.toJson(empty)
         new File("/tmp/list-one.json").write(json)
   - name: echo-steps
     inputs:

And it succeeded.

Name:                argo-test-2zlb9
Namespace:           default
ServiceAccount:      default
Status:              Succeeded
Created:             Mon Jan 13 20:03:26 +0900 (3 minutes ago)
Started:             Mon Jan 13 20:03:26 +0900 (3 minutes ago)
Finished:            Mon Jan 13 20:05:50 +0900 (36 seconds ago)
Duration:            2 minutes 24 seconds

STEP                                               PODNAME                     DURATION  MESSAGE
 ✔ argo-test-2zlb9 (process-steps)
 ├---✔ create-list (create-list)                   argo-test-2zlb9-1147473066  16s
 ├-·-✔ echo(0:first:One,second:Two) (echo-steps)
 | | ├---✔ order(0) (echo)                         argo-test-2zlb9-3874096275  8s
 | | └---✔ reverse(0) (echo)                       argo-test-2zlb9-1295474366  8s
 | └-✔ echo(1:first:Uno,second:Dos) (echo-steps)
 |   ├---✔ order(0) (echo)                         argo-test-2zlb9-907311640   27s
 |   └---✔ reverse(0) (echo)                       argo-test-2zlb9-4068771781  28s
 └-·-✔ echo2(0:first:One,second:Two) (echo-steps)
   | ├---✔ order(0) (echo)                         argo-test-2zlb9-1065554700  7s
   | └---✔ reverse(0) (echo)                       argo-test-2zlb9-3225911873  6s
   └-✔ echo2(1:first:Uno,second:Dos) (echo-steps)
     ├---✔ order(0) (echo)                         argo-test-2zlb9-1703890455  25s
     └---✔ reverse(0) (echo)                       argo-test-2zlb9-4117493562  6s

Although this is not a workflow-template-related issue, I'm happy to fix it as I started the investigation.

[alexec]

@sarabala1979 is this added to v2.6 as it needs fixing?

[alexec]

I don't expect to make v2.6 Dropping.