Loops in workflows

[dylburger]

I'd like to run a set of steps or actions within a loop construct. For example, I might receive an array of 100 events in my HTTP payload, and I need to run an action on each. I can do this with code, but I have no way to loop through the elements of this array using Pipedream's actions. I'd like a "loop" action that can handle this use case.

[tinjaw]

As a temporary workaround, can you please add the ability to duplicate an existing step in a workflow? So, if I want to do something ten times, I can create the action, duplicate it nine times, and then just modify the parameters in the duplicate actions.

[pauliusuza]

This functionality is a must! If a step receives an array, it should simply run the same step in the sequence for X times.

[alexanderradahl]

What is the status on this? It's a killer for when moving from Make.

[dylburger]

@alexanderahlsen This is on the backlog for 2022! We're working on a GitHub integration now, and plan to work on control flow (loops, conditionals, etc.) soon after!

[gustavorps]

This is what is preventing me from migrating from Integromat

[ctrlaltdylan]

Here's a temporary workaround by utilizing two different workflows.

The first workflow is the fetching workflow. It grabs this array of records from Airtable, Google Sheets, Quickbooks, or some database or API.

The second workflow is the processing workflow. It's responsible for processing a single record at a time, so that way you can use all of the pre-built actions Pipedream has to offer.

How to create a processing workflow

First, create a new workflow with an HTTP trigger:

In this case, the HTTP endpoint for our processing workflow is https://eo2b9ksk3fy3nrb.m.pipedream.net.

How to connect the fetching workflow to the processing workflow

In your fetching workflow after the fetching step, whether that be a trigger or an action add a new Node.js code step and paste in this snippet:

import { axios } from '@pipedream/platform';
// To use previous step data, pass the `steps` object to the run() function
export default defineComponent({
  props: {
    records: {
      type: "any",
      label: "Records to loop",
      description: "The array of records to send to processing workflow",
    },
    workflow_url: {
      type: "string",
      label: "Processing workflow URL",
      description: "The HTTP endpoint to connect to that's processing single individual records from this workflow"
    }
  },
  async run({ steps, $ }) {
    for await(const record of this.records) {
      await axios($, {
        url: this.workflow_url,
        method: 'POST',
        data: record
      })
    }
  },
})

Then populate the Records prop with the array of records you need processed, and paste in the URL endpoint to your processing workflow. In our example above, the URL would be https://eo2b9ksk3fy3nrb.m.pipedream.net.

After clicking Test in your fetching workflow, you should see individual HTTP requests hitting your processing workflow with each record individual in the body of the request.

Now you can process each record individually, and leverage the hundreds of pre-built actions in your processing workflow.

[bacherjo]

Hey ctrlaltdylan, you made my day! This is an excellent solution to the missing iterator feature in pipedream!
Thank you!
Johannes

[djpecot]

I ran into an issue where my API came back as an object instead of an array. This modified code should handle single values relatively gracefully 🤞 @ctrlaltdylan

import { axios } from '@pipedream/platform';
// To use previous step data, pass the `steps` object to the run() function
export default defineComponent({
  props: {
    records: {
      type: "any",
      label: "Records to loop",
      description: "The array of records to send to processing workflow",
    },
    workflow_url: {
      type: "string",
      label: "Processing workflow URL",
      description: "The HTTP endpoint to connect to that's processing single individual records from this workflow"
    }
  },

  
  

  async run({ steps, $ }) {
    // Break this if a single value (Array | Object)
    if (Array.isArray(this.records)) {
      for await(const record of this.records) {
        await axios($, {
          url: this.workflow_url,
          method: 'POST',
          data: record
        })
      }
    } else {
      await axios($, {
          url: this.workflow_url,
          method: 'POST',
          data: this.records
        })
    }
  },
})

Not the cleanest grammatically but y'all get the point right?

[ctrlaltdylan]

Hi @djpecot , yup I follow!

Though I'm curious how your data is being passed to the records prop is sometimes an array and sometimes a single object.

What's the source of your data if you don't mind me asking?