Loops in workflows

[dylburger]

I'd like to run a set of steps or actions within a loop construct. For example, I might receive an array of 100 events in my HTTP payload, and I need to run an action on each. I can do this with code, but I have no way to loop through the elements of this array using Pipedream's actions. I'd like a "loop" action that can handle this use case.

[tinjaw]

As a temporary workaround, can you please add the ability to duplicate an existing step in a workflow? So, if I want to do something ten times, I can create the action, duplicate it nine times, and then just modify the parameters in the duplicate actions.

[pauliusuza]

This functionality is a must! If a step receives an array, it should simply run the same step in the sequence for X times.

[alexanderradahl]

What is the status on this? It's a killer for when moving from Make.

[dylburger]

@alexanderahlsen This is on the backlog for 2022! We're working on a GitHub integration now, and plan to work on control flow (loops, conditionals, etc.) soon after!

[gustavorps]

This is what is preventing me from migrating from Integromat

[ctrlaltdylan]

Here's a temporary workaround by utilizing two different workflows.

The first workflow is the fetching workflow. It grabs this array of records from Airtable, Google Sheets, Quickbooks, or some database or API.

The second workflow is the processing workflow. It's responsible for processing a single record at a time, so that way you can use all of the pre-built actions Pipedream has to offer.

How to create a processing workflow

First, create a new workflow with an HTTP trigger:

In this case, the HTTP endpoint for our processing workflow is https://eo2b9ksk3fy3nrb.m.pipedream.net.

How to connect the fetching workflow to the processing workflow

In your fetching workflow after the fetching step, whether that be a trigger or an action add a new Node.js code step and paste in this snippet:

import { axios } from '@pipedream/platform';
// To use previous step data, pass the `steps` object to the run() function
export default defineComponent({
  props: {
    records: {
      type: "any",
      label: "Records to loop",
      description: "The array of records to send to processing workflow",
    },
    workflow_url: {
      type: "string",
      label: "Processing workflow URL",
      description: "The HTTP endpoint to connect to that's processing single individual records from this workflow"
    }
  },
  async run({ steps, $ }) {
    for await(const record of this.records) {
      await axios($, {
        url: this.workflow_url,
        method: 'POST',
        data: record
      })
    }
  },
})

Then populate the Records prop with the array of records you need processed, and paste in the URL endpoint to your processing workflow. In our example above, the URL would be https://eo2b9ksk3fy3nrb.m.pipedream.net.

After clicking Test in your fetching workflow, you should see individual HTTP requests hitting your processing workflow with each record individual in the body of the request.

Now you can process each record individually, and leverage the hundreds of pre-built actions in your processing workflow.

[bacherjo]

Hey ctrlaltdylan, you made my day! This is an excellent solution to the missing iterator feature in pipedream!
Thank you!
Johannes

[djpecot]

I ran into an issue where my API came back as an object instead of an array. This modified code should handle single values relatively gracefully 🤞 @ctrlaltdylan

import { axios } from '@pipedream/platform';
// To use previous step data, pass the `steps` object to the run() function
export default defineComponent({
  props: {
    records: {
      type: "any",
      label: "Records to loop",
      description: "The array of records to send to processing workflow",
    },
    workflow_url: {
      type: "string",
      label: "Processing workflow URL",
      description: "The HTTP endpoint to connect to that's processing single individual records from this workflow"
    }
  },

  
  

  async run({ steps, $ }) {
    // Break this if a single value (Array | Object)
    if (Array.isArray(this.records)) {
      for await(const record of this.records) {
        await axios($, {
          url: this.workflow_url,
          method: 'POST',
          data: record
        })
      }
    } else {
      await axios($, {
          url: this.workflow_url,
          method: 'POST',
          data: this.records
        })
    }
  },
})

Not the cleanest grammatically but y'all get the point right?

[ctrlaltdylan]

Hi @djpecot , yup I follow!

Though I'm curious how your data is being passed to the records prop is sometimes an array and sometimes a single object.

What's the source of your data if you don't mind me asking?

[djpecot]

@ctrlaltdylan It's a webhook that's hitting me with a variable amount of objects (Toggl). It was hitting me with either an array or a single object as I recall, so had to account for it.

[LucBerge]

This functionality is a must! If a step receives an array, it should simply run the same step in the sequence for X times.

I'm looking for this feature as well 😄

[tqbdev]

This functionality is a must! If a step receives an array, it should simply run the same step in the sequence for X times.

Any update for this feature? Two years and I hard to see the future of this simple implementation.

[DFazekas]

Just created an account on this website an hour ago, and already deleting my account due to the lack of this feature. What a shame.

[zvictor]

Just created an account on this website an hour ago, and already deleting my account due to the lack of this feature. What a shame.

Exactly my case as well 😅.
Pipedream seems to be the best solution, otherwise! ❤️

[bawantha]

3 years and not yet implemented

[bawantha]

@alexanderahlsen This is on the backlog for 2022! We're working on a GitHub integration now, and plan to work on control flow (loops, conditionals, etc.) soon after!

like people care about GitHub integration?

[andrewjschuang]

Zapier seems to handle loops in a solid way

[blendedmarket]

I think this has to be the deal breaker for most users once they realize this limitation. Perhaps its because making an extra call to a different workflow allows them to bill you separately for each iteration? Correct me if I am wrong , but each iteration would be an additional invocation?
It's a shame, because I was really enjoying the building process only to find out I can't afford to use it.

[bacherjo]

Yes I can confirm that calling a different workflow counts as a separate invocation. I have to process orders with 20 items, so instead of having one invocation i have 21.
Still I think we need this feature and need the ability to handle this within one invocation. It gets confusing and hard to monitor with a lot of single workflows.

[Landsil]

eh, back to Zapier I guess :(
I was really looking towards switching away from them.

I can code loop myself but if have to bother with that I may as well code the rest and self host it instead of pushing data via external services...

[eigenstil]

Just a quick note for other people stumbling accross this issue:

following @ctrlaltdylan's step-by-step explanation above, I was able to solve it for now. Though, this is somewhat tedious ;-)

[shokks]

{{Steps.Not.A.Coder ;-)}}
The workaround offered with invoking another workflow by pushing it via webhook is elegant since the 2nd workflow runs independently for each item and in parallel - so I am guessing there's a huge time saver if you need to call another API for each item.
However, I do not see a way to stitch them all together at the end in the sequence of the initial array - any ideas, please? @ctrlaltdylan

My used case: Take long-form text, split into pieces, do something with each piece, and then put them back together in the order of the original long-from text.

[acreconsulting]

Just wanted to add my +1!

Use case: I want to add each person who is registered for an event in my appointment scheduler to my CRM.

[evelant]

A no-code way to do simple iteration over array elements would be really nice! There are lots of cases where you can end up with an array of data but the API you're working with won't accept array/bulk input so you need to send a request for each one.

[LucBerge]

I guess there is nothing technical preventing them to implement the loop system. What could limit them to do it is the business model: Currently, user/organization pay per event. The problem is that if a step retrieves 100 elements to process in a loop, it is still counted as one event. Since pipedream is probably using a pay as you go server (like AWS), they need to think about a new business model before going further. This is probably why this issue is in standby since 4 years...

[dylburger]

Hi y'all, just want to let you know we read all this feedback and very much appreciate the use cases! This is on our roadmap for this year, after we ship our Git integration, folders, and other big features we're working on right now. So we plan to address this soon.

[LucBerge]

Hi y'all, just want to let you know we read all this feedback and very much appreciate the use cases! This is on our roadmap for this year, after we ship our Git integration, folders, and other big features we're working on right now. So we plan to address this soon.

Hello, great news 🎉 Will the tiers change ?

[dylburger]

We haven't decided exactly how this will impact the invocations / credits model, but we'll comment here / share in future announcements on all the details.

[sammymlangeni]

There was at one stage where I was looking to get the payload data from Woocommerce. I was more interested in getting the licenses keys to uploaded to Dropbox folder in a form of txt file. My use case may not be as complicated. The payload created 40 lines of code of which will never be the real scenario for my situation. I did not see 40 invocations when I was looping through an an array, only one.

My payload looks like this

I am sure this is only applicable to me.

[DhavalW]

Seems like this is still a pipedream eh.

+1 its unfair if each looped item counts as a new invocation and gets billed. I love pipedream so far, but If this happens, I'm moving to autocode or something else.

[LucBerge]

+1 its unfair if each looped item counts as a new invocation and gets billed. I love pipedream so far, but If this happens, I'm moving to autocode or something else

Seems like invocation system is beeing replaced by a credit system.

1 credit = 30 sec (256 Mo)
100 credits/day (for free tier) = 50min/day

This is still better than other intergration platforms

[DhavalW]

Yes. Definitely. One of the key things I like about pipedream for sure. But AFAIK it's not fully replaced. Each invocation still counts as min.1 credit, regardless of how long you use it. So if you have 10 invocations of an api that last 2s each - it bills 10 credits, even though total runtime across calls is < 1 credit.

…

On Wed, 15 Mar 2023, 2:32 am LucBerge, ***@***.***> wrote: +1 its unfair if each looped item counts as a new invocation and gets billed. I love pipedream so far, but If this happens, I'm moving to autocode or something else Seems like invocation system is beeing replaced by a credit system. 1 credit = 30 sec (256 Mo) 100 credits/day (for free tier) = 50min/day This is still better than other intergration platforms — Reply to this email directly, view it on GitHub <#193 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADRLL6BF5O3INOFYWUA7YNLW4DMIFANCNFSM4OC7JNNQ> . You are receiving this because you commented.Message ID: ***@***.***>

[NebularNerd]

Just came across a workflow where I need this, I can likely create a bodge of some description with a dual workflow but this would seem to be a must have helper function.

[ExpertLearner7]

Hi, we are trying to move from Zapier to Pipedream but we use the loop action in some Zaps. Is there a no code way to iterate through an array/list? to do some actions repeatedly over each array/list elements?

[DhavalW]

Actually i realised later that you can. Needs to be custom coded through a node js block. But it's been a while, so not sure if it's still possible or killed by the recent platform updates.

…

On Thu, 29 Jun 2023, 5:24 am ExpertLearner7, ***@***.***> wrote: Hi, we are trying to move from Zapier to Pipedream but we use the loop action in some Zaps. Is there a no code way to iterate through an array/list? to do some actions repeatedly over each array/list element? — Reply to this email directly, view it on GitHub <#193 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADRLL6G6GBHE72CDMYN27D3XNS737ANCNFSM4OC7JNNQ> . You are receiving this because you commented.Message ID: ***@***.***>

[RichardBaoLei]

Just quickly looked through each posts. the no-code loops step is really a deal breaker. the current nodeJS solution is not elegant and user-friendly for most people. is there any definitive date for the development of it? @vunguyenhung @dylburger @ctrlaltdylan

[thomasbiddle]

How's this coming along? I'm considering migrating from Make to Pipedream due to some frustrations; but this is a dealbreaker.

[nadyyym]

Any estimates on this? Want to move all my automations to Pipedream, but I am not technical and would really hate to implement loops with code

[bacherjo]

Any estimates on this? Want to move all my automations to Pipedream, but I am not technical and would really hate to implement loops with code

Hi Nadyyym, I think the loop option has many potential variations and hence it takes so long for the guys to deliver it. However, I implemented the code solution (by looping in javascript and calling sub-workflows) and I can say it is just a few lines of code, gives you a lot of flexibility and was not too hard to do (and I am also not very technical).
Johannes

[nadyyym]

loop option has many potential variations

a lot of flexibility and was not too hard to do

Fair enough. Guess I'll refresh my Python skills.

[motiejunas]

@alexanderahlsen This is on the backlog for 2022! We're working on a GitHub integration now, and plan to work on control flow (loops, conditionals, etc.) soon after!

Late 2023 already, so I wonder what is the progress?

[panda-inc-1]

Pipedream will become our EXCLUSIVE low code solution (we currently pay for Make, Zapier, n8n in addition to Pipedream) if it incorporates advanced logic like this!

[keladeine]

I'm just trying to transfer tags of an item. It's a really big flaw if I have to run a workflow 4 times just because one record has multiple tags.

[jscottcronin]

@dylburger and others - This feature request was made in 2019. Basic IF-ELSE and FOR LOOP logic is table stakes. For a lot of us coming from Make or Zapier, we start playing with pipedream and it's a joke to not have these super basic features. It's been promised to add this functionality for years, and yet, it doesn't happen. I'm inclined to leave the platform because I'm losing confidence that the tech team can accomplish what customers actually need.

[ctrlaltdylan]

Thanks for the candid feedback everyone, and we really appreciate your patience for this highly requested feature.

We know that Control Flow unlocks the ability for your workflows from a linear to a complex branching and looping experience and it would be a massive productivity gain.

This feature is the highest priority for our team, and we're actively working to release a beta version in the near feature.

If you're interested in gaining early access to this major feature that would allow branching and looping, join by filling out this short beta interest form so we can contact you as soon as it's available:

https://forms.pipedream.com/control-flow-beta

[dylburger]

@jscottcronin Nothing we love more than to wake up and read comments like this!

A GitHub issue is not a promise. This was among the first requests we received when we launched the platform, so we created this issue. In the meantime, we’ve shipped hundreds of meaningful platform features and thousands of integrations. There will always be features like this that take time to prioritize and ship. Every platform deals with this.

I empathize so much with the desire for flow control and completely feel the pain. Try to articulate that feedback without calling Pipedream “a joke” and trashing our technical team. Choose kindness! You’ll get so much more mileage with it.

[mroy-seedbox]

@jscottcronin: Did you know that Zapier existed for seven years without the ability to do even the most basic branching? Let alone looping, which was released years later. They were officially founded in 2011, and only added branches ("paths") in 2018.

Pipedream is still a pretty young company, but give them some time and they'll knock it out of the park -- in half the time it took for Zapier to develop these features, and with many much more awesome ones.

On our side, we're migrating away from Zapier. And we're not looking back.

But I do understand the impatience for this feature.

I think what has probably delayed it the most is that there are rather simple workarounds:

• You can branch/loop/etc within code steps.
• You can branch out by calling other workflows.

The other features which have been developed since 2019 have no such workarounds.