Parameter outputs from withParam steps are passed as escaped strings rather than JSON

[matthew117]

Pre-requisites

• I have double-checked my configuration
• I can confirm the issue exists when I tested with :latest
• I have searched existing issues and could not find a match for this bug
• I'd like to contribute the fix myself (see contributing guide)

What happened/what did you expect to happen?

I was looking at the documentation for looping here https://argo-workflows.readthedocs.io/en/latest/walk-through/loops/#accessing-the-aggregate-results-of-a-loop as I'm essentially doing some map-reduce type stuff in my pipeline.

It was working fine when using steps.step-name.outputs.result when it was just trivially printing out some dummy json structure but the step runs a bunch of scripts that write a lot of output to the command so I switched to using a named output and valueFrom path so that I could keep the useful logs whilst only exporting the specific json I needed for the aggregation step.

However, instead of getting a list of json objects from steps.step-name.outputs.parameters.parameter-name that I can easily drop into a python or node script on the next step it's an array of strings and I therefore have to start the next step by parsing the strings, which is a pretty simple work around, but it seems strange that the default result and outputs are being treated differently.

You can see that at the point of output, the value is indeed JSON

But then when it's aggregated as an input in the next step each JSON object is an escaped string

Version

v3.5.5

Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: loop-test-
spec:
  entrypoint: main
  templates:
  - name: main
    steps:
    - - name: execute-parallel-steps
        template: print-json-entry
        arguments:
          parameters:
          - name: index
            value: '{{item}}'
        withParam: '[1, 2, 3]'
    - - name: call-print-aggregate-output
        template: print-aggregate-output
        arguments:
          parameters:
          - name: aggregate-results
            # If the value of each loop iteration isn't a valid JSON,
            # you get a JSON parse error:
            # value: '{{steps.execute-parallel-steps.outputs.result}}'
            value: '{{steps.execute-parallel-steps.outputs.parameters.content}}'

  - name: print-json-entry
    inputs:
      parameters:
      - name: index
    # The output must be a valid JSON
    script:
      image: node
      command: [node]
      source: |
        const fs = require('fs');
        const content = JSON.stringify({a: 1, b: 2});
        fs.writeFileSync('/tmp/outputs.json', content);
    outputs:
      parameters:
      - name: content
        valueFrom:
          path: /tmp/outputs.json
  - name: print-aggregate-output
    inputs:
      parameters:
      - name: aggregate-results
    script:
      image: python
      command: [python]
      source: |
        print({{inputs.parameters.aggregate-results}})
        # prints ['{"a":1,"b":2}', '{"a":1,"b":2}', '{"a":1,"b":2}']

Logs from the workflow controller

kubectl logs -n argo deploy/workflow-controller | grep loop-test-sdxww
time="2024-03-15T17:19:35.046Z" level=info msg="Processing workflow" Phase= ResourceVersion=599370637 namespace=argo workflow=loop-test-sdxww
time="2024-03-15T17:19:35.051Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=0 workflow=loop-test-sdxww
time="2024-03-15T17:19:35.051Z" level=info msg="Updated phase  -> Running" namespace=argo workflow=loop-test-sdxww
time="2024-03-15T17:19:35.051Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=argo workflow=loop-test-sdxww
time="2024-03-15T17:19:35.051Z" level=info msg="was unable to obtain node for , letting display name to be nodeName" namespace=argo workflow=loop-test-sdxww
time="2024-03-15T17:19:35.051Z" level=info msg="Steps node loop-test-sdxww initialized Running" namespace=argo workflow=loop-test-sdxww
time="2024-03-15T17:19:35.051Z" level=info msg="StepGroup node loop-test-sdxww-2416534422 initialized Running" namespace=argo workflow=loop-test-sdxww
time="2024-03-15T17:19:35.051Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=argo workflow=loop-test-sdxww
time="2024-03-15T17:19:35.051Z" level=info msg="Pod node loop-test-sdxww-697719329 initialized Pending" namespace=argo workflow=loop-test-sdxww
time="2024-03-15T17:19:35.069Z" level=info msg="Created pod: loop-test-sdxww[0].execute-parallel-steps(0:1) (loop-test-sdxww-print-json-entry-697719329)" namespace=argo workflow=loop-test-sdxww
time="2024-03-15T17:19:35.069Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=argo workflow=loop-test-sdxww
time="2024-03-15T17:19:35.069Z" level=info msg="Pod node loop-test-sdxww-1004034371 initialized Pending" namespace=argo workflow=loop-test-sdxw
time="2024-03-15T17:19:35.082Z" level=info msg="Created pod: loop-test-sdxww[0].execute-parallel-steps(1:2) (loop-test-sdxww-print-json-entry-1004034371)" namespace=argo workflow=loop-test-sdxww
time="2024-03-15T17:19:35.082Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=argo workflow=loop-test-sdxww
time="2024-03-15T17:19:35.082Z" level=info msg="Pod node loop-test-sdxww-700833777 initialized Pending" namespace=argo workflow=loop-test-sdxww
time="2024-03-15T17:19:35.097Z" level=info msg="Created pod: loop-test-sdxww[0].execute-parallel-steps(2:3) (loop-test-sdxww-print-json-entry-700833777)" namespace=argo workflow=loop-test-sdxww
time="2024-03-15T17:19:35.097Z" level=info msg="Workflow step group node loop-test-sdxww-2416534422 not yet completed" namespace=argo workflow=loop-test-sdxww
time="2024-03-15T17:19:35.097Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=loop-test-sdxww
time="2024-03-15T17:19:35.097Z" level=info msg=reconcileAgentPod namespace=argo workflow=loop-test-sdxww
time="2024-03-15T17:19:35.107Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=599370646 workflow=loop-test-sdxww
time="2024-03-15T17:19:45.046Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=599370646 namespace=argo workflow=loop-test-sdxww
time="2024-03-15T17:19:45.047Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=3 workflow=loop-test-sdxww
time="2024-03-15T17:19:45.047Z" level=info msg="task-result changed" namespace=argo nodeID=loop-test-sdxww-1004034371 workflow=loop-test-sdxww
time="2024-03-15T17:19:45.047Z" level=info msg="task-result changed" namespace=argo nodeID=loop-test-sdxww-697719329 workflow=loop-test-sdxww
time="2024-03-15T17:19:45.047Z" level=info msg="task-result changed" namespace=argo nodeID=loop-test-sdxww-700833777 workflow=loop-test-sdxww
time="2024-03-15T17:19:45.047Z" level=info msg="node changed" namespace=argo new.message= new.phase=Succeeded new.progress=0/1 nodeID=loop-test-sdxww-700833777 old.message= old.phase=Pending old.progress=0/1 workflow=loop-test-sdxww
time="2024-03-15T17:19:45.047Z" level=info msg="node changed" namespace=argo new.message= new.phase=Running new.progress=0/1 nodeID=loop-test-sdxww-697719329 old.message= old.phase=Pending old.progress=0/1 workflow=loop-test-sdxww
time="2024-03-15T17:19:45.047Z" level=info msg="node changed" namespace=argo new.message= new.phase=Succeeded new.progress=0/1 nodeID=loop-test-sdxww-1004034371 old.message= old.phase=Pending old.progress=0/1 workflow=loop-test-sdxww
time="2024-03-15T17:19:45.047Z" level=info msg="Workflow step group node loop-test-sdxww-2416534422 not yet completed" namespace=argo workflow=loop-test-sdxww
time="2024-03-15T17:19:45.047Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=loop-test-sdxww
time="2024-03-15T17:19:45.047Z" level=info msg=reconcileAgentPod namespace=argo workflow=loop-test-sdxww
time="2024-03-15T17:19:45.052Z" level=info msg="cleaning up pod" action=terminateContainers key=argo/loop-test-sdxww-print-json-entry-697719329/terminateContainers
time="2024-03-15T17:19:45.059Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=599370764 workflow=loop-test-sdxww
time="2024-03-15T17:19:45.067Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo/loop-test-sdxww-print-json-entry-1004034371/labelPodCompleted
time="2024-03-15T17:19:45.067Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo/loop-test-sdxww-print-json-entry-700833777/labelPodCompleted
time="2024-03-15T17:19:55.061Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=599370764 namespace=argo workflow=loop-test-sdxww
time="2024-03-15T17:19:55.061Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=3 workflow=loop-test-sdxww
time="2024-03-15T17:19:55.061Z" level=info msg="task-result changed" namespace=argo nodeID=loop-test-sdxww-1004034371 workflow=loop-test-sdxww
time="2024-03-15T17:19:55.061Z" level=info msg="task-result changed" namespace=argo nodeID=loop-test-sdxww-697719329 workflow=loop-test-sdxww
time="2024-03-15T17:19:55.061Z" level=info msg="task-result changed" namespace=argo nodeID=loop-test-sdxww-700833777 workflow=loop-test-sdxww
time="2024-03-15T17:19:55.061Z" level=info msg="node changed" namespace=argo new.message= new.phase=Succeeded new.progress=0/1 nodeID=loop-test-sdxww-697719329 old.message= old.phase=Running old.progress=0/1 workflow=loop-test-sdxww
time="2024-03-15T17:19:55.062Z" level=info msg="Step group node loop-test-sdxww-2416534422 successful" namespace=argo workflow=loop-test-sdxww
time="2024-03-15T17:19:55.062Z" level=info msg="node loop-test-sdxww-2416534422 phase Running -> Succeeded" namespace=argo workflow=loop-test-sdxww
time="2024-03-15T17:19:55.062Z" level=info msg="node loop-test-sdxww-2416534422 finished: 2024-03-15 17:19:55.062092745 +0000 UTC" namespace=argo workflow=loop-test-sdxww
time="2024-03-15T17:19:55.062Z" level=info msg="StepGroup node loop-test-sdxww-3422941299 initialized Running" namespace=argo workflow=loop-test-sdxww
time="2024-03-15T17:19:55.062Z" level=info msg="SG Outbound nodes of loop-test-sdxww-697719329 are [loop-test-sdxww-697719329]" namespace=argo workflow=loop-test-sdxww
time="2024-03-15T17:19:55.062Z" level=info msg="SG Outbound nodes of loop-test-sdxww-1004034371 are [loop-test-sdxww-1004034371]" namespace=argo workflow=loop-test-sdxww
time="2024-03-15T17:19:55.062Z" level=info msg="SG Outbound nodes of loop-test-sdxww-700833777 are [loop-test-sdxww-700833777]" namespace=argo workflow=loop-test-sdxww
time="2024-03-15T17:19:55.062Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=argo workflow=loop-test-sdxww
time="2024-03-15T17:19:55.062Z" level=info msg="Pod node loop-test-sdxww-1374662512 initialized Pending" namespace=argo workflow=loop-test-sdxw
time="2024-03-15T17:19:55.075Z" level=info msg="Created pod: loop-test-sdxww[1].call-access-aggregate-output (loop-test-sdxww-access-aggregate-output-1374662512)" namespace=argo workflow=loop-test-sdxww
time="2024-03-15T17:19:55.075Z" level=info msg="Workflow step group node loop-test-sdxww-3422941299 not yet completed" namespace=argo workflow=loop-test-sdxww
time="2024-03-15T17:19:55.076Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=loop-test-sdxww
time="2024-03-15T17:19:55.076Z" level=info msg=reconcileAgentPod namespace=argo workflow=loop-test-sdxww
time="2024-03-15T17:19:55.085Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=599370856 workflow=loop-test-sdxww
time="2024-03-15T17:19:55.092Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo/loop-test-sdxww-print-json-entry-697719329/labelPodCompleted
time="2024-03-15T17:20:05.077Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=599370856 namespace=argo workflow=loop-test-sdxww
time="2024-03-15T17:20:05.077Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=4 workflow=loop-test-sdxww
time="2024-03-15T17:20:05.077Z" level=info msg="task-result changed" namespace=argo nodeID=loop-test-sdxww-1004034371 workflow=loop-test-sdxww
time="2024-03-15T17:20:05.077Z" level=info msg="task-result changed" namespace=argo nodeID=loop-test-sdxww-697719329 workflow=loop-test-sdxww
time="2024-03-15T17:20:05.077Z" level=info msg="task-result changed" namespace=argo nodeID=loop-test-sdxww-700833777 workflow=loop-test-sdxww
time="2024-03-15T17:20:05.077Z" level=info msg="task-result changed" namespace=argo nodeID=loop-test-sdxww-1374662512 workflow=loop-test-sdxww
time="2024-03-15T17:20:05.077Z" level=info msg="node changed" namespace=argo new.message= new.phase=Succeeded new.progress=0/1 nodeID=loop-test-sdxww-1374662512 old.message= old.phase=Pending old.progress=0/1 workflow=loop-test-sdxww
time="2024-03-15T17:20:05.077Z" level=info msg="SG Outbound nodes of loop-test-sdxww-697719329 are [loop-test-sdxww-697719329]" namespace=argo workflow=loop-test-sdxww
time="2024-03-15T17:20:05.077Z" level=info msg="SG Outbound nodes of loop-test-sdxww-1004034371 are [loop-test-sdxww-1004034371]" namespace=argo workflow=loop-test-sdxww
time="2024-03-15T17:20:05.077Z" level=info msg="SG Outbound nodes of loop-test-sdxww-700833777 are [loop-test-sdxww-700833777]" namespace=argo workflow=loop-test-sdxww
time="2024-03-15T17:20:05.077Z" level=info msg="Step group node loop-test-sdxww-3422941299 successful" namespace=argo workflow=loop-test-sdxww
time="2024-03-15T17:20:05.077Z" level=info msg="node loop-test-sdxww-3422941299 phase Running -> Succeeded" namespace=argo workflow=loop-test-sdxww
time="2024-03-15T17:20:05.077Z" level=info msg="node loop-test-sdxww-3422941299 finished: 2024-03-15 17:20:05.077869194 +0000 UTC" namespace=argo workflow=loop-test-sdxww
time="2024-03-15T17:20:05.077Z" level=info msg="Outbound nodes of loop-test-sdxww-1374662512 is [loop-test-sdxww-1374662512]" namespace=argo workflow=loop-test-sdxww
time="2024-03-15T17:20:05.077Z" level=info msg="Outbound nodes of loop-test-sdxww is [loop-test-sdxww-1374662512]" namespace=argo workflow=loop-test-sdxww
time="2024-03-15T17:20:05.077Z" level=info msg="node loop-test-sdxww phase Running -> Succeeded" namespace=argo workflow=loop-test-sdxww
time="2024-03-15T17:20:05.077Z" level=info msg="node loop-test-sdxww finished: 2024-03-15 17:20:05.077949053 +0000 UTC" namespace=argo workflowloop-test-sdxww
time="2024-03-15T17:20:05.077Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=loop-test-sdxww
time="2024-03-15T17:20:05.077Z" level=info msg=reconcileAgentPod namespace=argo workflow=loop-test-sdxww
time="2024-03-15T17:20:05.077Z" level=info msg="Updated phase Running -> Succeeded" namespace=argo workflow=loop-test-sdxww
time="2024-03-15T17:20:05.078Z" level=info msg="Marking workflow completed" namespace=argo workflow=loop-test-sdxww
time="2024-03-15T17:20:05.083Z" level=info msg="cleaning up pod" action=deletePod key=argo/loop-test-sdxww-1340600742-agent/deletePod
time="2024-03-15T17:20:05.088Z" level=info msg="Workflow update successful" namespace=argo phase=Succeeded resourceVersion=599370959 workflow=loop-test-sdxww
time="2024-03-15T17:20:05.105Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo/loop-test-sdxww-access-aggregate-output-1374662512/labelPodCompleted

Logs from in your workflow's wait container

time="2024-03-15T17:19:59.313Z" level=info msg="No Script output reference in workflow. Capturing script output ignored"
time="2024-03-15T17:19:59.313Z" level=info msg="No output parameters"
time="2024-03-15T17:19:59.313Z" level=info msg="No output artifacts"
time="2024-03-15T17:19:59.314Z" level=info msg="S3 Save path: /tmp/argo/outputs/logs/main.log, key: loop-test-sdxww/loop-test-sdxww-access-aggregate-output-1374662512/main.log"
time="2024-03-15T17:19:59.314Z" level=info msg="Creating minio client using static credentials" endpoint="minio:9000"
time="2024-03-15T17:19:59.314Z" level=info msg="Saving file to s3" bucket=my-bucket endpoint="minio:9000" key=loop-test-sdxww/loop-test-sdxww-access-aggregate-output-1374662512/main.log path=/tmp/argo/outputs/logs/main.log
time="2024-03-15T17:19:59.325Z" level=info msg="Save artifact" artifactName=main-logs duration=11.343605ms error="<nil>" key=loop-test-sdxww/loop-test-sdxww-access-aggregate-output-1374662512/main.log
time="2024-03-15T17:19:59.325Z" level=info msg="not deleting local artifact" localArtPath=/tmp/argo/outputs/logs/main.log
time="2024-03-15T17:19:59.325Z" level=info msg="Successfully saved file: /tmp/argo/outputs/logs/main.log"
time="2024-03-15T17:19:59.338Z" level=info msg="Alloc=7794 TotalAlloc=13783 Sys=23653 NumGC=4 Goroutines=10"
time="2024-03-15T17:19:42.330Z" level=info msg="Saving path output parameter: content"
time="2024-03-15T17:19:42.330Z" level=info msg="No output artifacts"
time="2024-03-15T17:19:42.330Z" level=info msg="S3 Save path: /tmp/argo/outputs/logs/main.log, key: loop-test-sdxww/loop-test-sdxww-print-json-entry-1004034371/main.log"
time="2024-03-15T17:19:42.330Z" level=info msg="Creating minio client using static credentials" endpoint="minio:9000"
time="2024-03-15T17:19:42.330Z" level=info msg="Saving file to s3" bucket=my-bucket endpoint="minio:9000" key=loop-test-sdxww/loop-test-sdxww-print-json-entry-1004034371/main.log path=/tmp/argo/outputs/logs/main.log
time="2024-03-15T17:19:42.339Z" level=info msg="Save artifact" artifactName=main-logs duration=8.811953ms error="<nil>" key=loop-test-sdxww/loop-test-sdxww-print-json-entry-1004034371/main.log
time="2024-03-15T17:19:42.339Z" level=info msg="not deleting local artifact" localArtPath=/tmp/argo/outputs/logs/main.log
time="2024-03-15T17:19:42.339Z" level=info msg="Successfully saved file: /tmp/argo/outputs/logs/main.log"
time="2024-03-15T17:19:42.353Z" level=info msg="Alloc=7896 TotalAlloc=13870 Sys=19301 NumGC=4 Goroutines=10"
time="2024-03-15T17:19:42.360Z" level=info msg="Deadline monitor stopped"
time="2024-03-15T17:19:43.336Z" level=info msg="Saving output parameters"
time="2024-03-15T17:19:43.336Z" level=info msg="Saving path output parameter: content"
time="2024-03-15T17:19:43.336Z" level=info msg="No output artifacts"
time="2024-03-15T17:19:43.336Z" level=info msg="S3 Save path: /tmp/argo/outputs/logs/main.log, key: loop-test-sdxww/loop-test-sdxww-print-json-entry-697719329/main.log"
time="2024-03-15T17:19:43.336Z" level=info msg="Creating minio client using static credentials" endpoint="minio:9000"
time="2024-03-15T17:19:43.337Z" level=info msg="Saving file to s3" bucket=my-bucket endpoint="minio:9000" key=loop-test-sdxww/loop-test-sdxww-print-json-entry-697719329/main.log path=/tmp/argo/outputs/logs/main.log
time="2024-03-15T17:19:43.346Z" level=info msg="Save artifact" artifactName=main-logs duration=9.056455ms error="<nil>" key=loop-test-sdxww/loop-test-sdxww-print-json-entry-697719329/main.log
time="2024-03-15T17:19:43.346Z" level=info msg="not deleting local artifact" localArtPath=/tmp/argo/outputs/logs/main.log
time="2024-03-15T17:19:43.346Z" level=info msg="Successfully saved file: /tmp/argo/outputs/logs/main.log"
time="2024-03-15T17:19:43.360Z" level=info msg="Alloc=7944 TotalAlloc=13829 Sys=23653 NumGC=4 Goroutines=10"
time="2024-03-15T17:19:41.348Z" level=info msg="Saving output parameters"
time="2024-03-15T17:19:41.348Z" level=info msg="Saving path output parameter: content"
time="2024-03-15T17:19:41.348Z" level=info msg="No output artifacts"
time="2024-03-15T17:19:41.348Z" level=info msg="S3 Save path: /tmp/argo/outputs/logs/main.log, key: loop-test-sdxww/loop-test-sdxww-print-json-entry-700833777/main.log"
time="2024-03-15T17:19:41.348Z" level=info msg="Creating minio client using static credentials" endpoint="minio:9000"
time="2024-03-15T17:19:41.348Z" level=info msg="Saving file to s3" bucket=my-bucket endpoint="minio:9000" key=loop-test-sdxww/loop-test-sdxww-print-json-entry-700833777/main.log path=/tmp/argo/outputs/logs/main.log
time="2024-03-15T17:19:41.359Z" level=info msg="Save artifact" artifactName=main-logs duration=11.238036ms error="<nil>" key=loop-test-sdxww/loop-test-sdxww-print-json-entry-700833777/main.log
time="2024-03-15T17:19:41.359Z" level=info msg="not deleting local artifact" localArtPath=/tmp/argo/outputs/logs/main.log
time="2024-03-15T17:19:41.359Z" level=info msg="Successfully saved file: /tmp/argo/outputs/logs/main.log"
time="2024-03-15T17:19:41.377Z" level=info msg="Alloc=7875 TotalAlloc=13835 Sys=23653 NumGC=4 Goroutines=10"