Deploy / Update Workflows via Git, API, CLI

[dylburger]

I want to write and test workflows on my local machine, storing the workflow's code in my existing version control system.

I also want to programmatically deploy workflows using a CLI, for example using a command like pipedream deploy. This way, I can deploy workflows locally or as part of my CI/CD pipeline.

Related: #1139

[dylburger]

We've launched the first version of the Pipedream CLI: https://docs.pipedream.com/cli/reference/

The CLI supports managing Event Sources for now (not yet workflows), so I updated the title of this ticket to reflect the need for managing workflows via CLI, and am keeping it open to track that request.

[morganism]

@dylburger Thank you! I am finding Pipedream really useful, Workflow management via CLI would be great. I'm pretty new to this service, but picking up the concept and how to get real utility out of it is not a chore. I'll DM you if I can.

[dylburger]

Adding a comment from #857 here. When we support an API / CLI for deploying workflows, we plan to also let you serialize workflows to plaintext (e.g. JavaScript code with some config locally), which means you'd be able to store them in Git. That feature would enable the request below.

"For testing my workflow with a real case, I need to deploy my code, so everytime I test something I create a new version of the workflow. My workflow ends up with lots of versions. Most of them have bugs or are here for testing/debugging
Sometimes I want to go back to a specific version but I can't remember which one it was, it can be 10 versions back or more, it's hard to identify versions and difference between them

Describe the solution you'd like
Something closer to real code versioning, so more like git commits:

being able to add a comment to a version. This would be a "quick win", as I could just tag a revision that I know is working before creating more revisions, So I can easily come back to it even if I created many revisions after.
Nice to have: being able to compare 2 versions of a workflow"

[illusoryio]

This would be very useful.

[moltar]

Would love to get this supported, this way we can migrate our processes from unstable Zapier zaps and Integromat scenarios to proper code solution with some tests.

[sjn001tvh]

Geetings. I'm a newby to Pipedream, but after several weeks of working with workflows between Dropbox and ClickUp etc., I've been able to accomplish a lot processing information from phone to the ClickUp app. However, as I have developed these routines, there have been instances where I have deleted a step... I know, I know, even though it warns me about the delete, I just needed it to get out of the way so I could try another option, and of course when I did... I lost the routine. I only had to do that once, and fortunately it wasn't filled with a lot of functions, etc. I've looked and asked about a utility that will allow me to save a step, maybe as an archive, so it suggested I follow this issue.

At this point, my only option is to cut and paste the code between the async(events, steps, auths) {...} and place them in a .js files defined as functions. Does anyone have a better method? Then, I can test the code, and if it works, cut and paste the new version back into my .js functions file. Thanks.

[cakemountain]

👍 for this, right now it's the only thing preventing my team from using your service!

[dylburger]

@cakemountain I hear you and we are working on an early version of this now! We’ll be sharing details on that here. We’d love y’all and anyone else watching this issue to test the beta when it’s ready.

[cakemountain]

@dylburger love hearing it. Do you have an approximate release timeline for the beta?

[dylburger]

I don’t, but I’ll let you know as soon as we have clarity on that.

[cakemountain]

@dylburger okay gotcha. if you're taking applicants for early access or beta testing we're highly motivated. Reach out to me on LinkedIn if you want to connect.

[ak--47]

this is my local workflow for writing tests inside of VSCode and constantly redeploying changes to pipeDream:

one file, 4 shells:

essentially:

• pd dev (watch for changes)
• pd logs (so i don’t need the web browser)
• jest tests on my code
• a scratchpad for doing random stuff

community post: https://pipedream.com/community/t/developing-code-steps-in-a-workflow-locally/1489

[doronlh]

+1

[mmmoli]

Would LOVE ❤️ this feature.

We manage code in a monorepo and storing workflows would be 👌 👌 👌

[jondkinney]

This would be great! Looking forward to hearing some updates, hopefully.

[charlie-s]

@sjn001tvh and anyone else that needs it, you can hit https://api.pipedream.com/v1/users/me/workflows (use the Authorization: Bearer ... header from the settings page) to download a JSON array of your workflows, including custom code in any components.

[The-TT-Hacker]

I think pipedream would see much more adoption if this workflows API existed.
It would allow workflows to be managed using version control and IaC (e.g. a terraform module).
They should really be deployed by CI/CD pipelines.

[EricWu91]

Long shooting here, but wouldn't it be possible to retrieve the workflow codes from git repos?

…

On Sun, Apr 23, 2023, 12:38 AM Tim Thacker ***@***.***> wrote: I think pipedream would see much more adoption if this workflows API existed. It would allow workflows to be managed using version control and IaC (e.g. a terraform module). — Reply to this email directly, view it on GitHub <#220 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACHJMBEB7YFFCSNLWDT6JB3XCSP35ANCNFSM4OC7QD2Q> . You are receiving this because you commented.Message ID: ***@***.***>

[mroy-seedbox]

This is 100% required in order to scale out Pipedream to hundreds of workflows, especially when many of them are very similar, or even identical (but running on different schedules or triggers, or with different parameters). Ex: run job A (via external API call), and wait for it to complete, then retrieve the results, and send them to Slack -- and do the same for job B, C, D, E, and so on...

Reusable actions/components are nice, but they only apply to one step. And then it is tedious to manually apply the new version in a multitude of workflows.

Manually managing hundreds of similar workflows is a no-go. It's kinda like copy pasting the same code all over the place: no one wants to maintain that mess (which version of the code/workflow is the most up-to-date, etc.). At least if we could look at all our workflows "at once" in an IDE, it would help to manage the repetitions, and make simultaneous updates/apply refactorings. I dream of editing Pipedream workflows directly in VS Code, and then using an awesome VS Code extension with all kind of cool features to interface directly with Pipedream from VS Code.

Also, with workflow management via API/Git/CLI, we could add a layer of abstraction on top of Pipedream, allowing us to define a generic/reusable/configurable workflow once, and then deploy it multiple times (with different parameters, schedules, or triggers).

Of course, reusable workflows or multi-step actions/components would also be very helpful, and could accomplish similar goals. Zapier has something similar, called "Sub-Zaps".

[peteratticusberg]

+1

[EricWu91]

Would love to hear updates from the team!

[n0vakovic]

+1

[danhstevens]

+1

[EricWu91]

First and foremost:
Is this the feature we've all been waiting for?

If it is, it looks amazing.

But this is slightly more disheartening.

I understand your business model, and I know you guys also need to make a buck. But 99/month for a single dev, in slightly larger projects is prohibitive. That's five times as expensive as the previous pricing tag!

[mroy-seedbox]

Maybe they could make the feature usage-based for lower plans. Just charge an extra credit per invoke for workflows that are synced to GitHub.

[EricWu91]

For context, I'm a single developer working on a few personal projects. I've replaced some of my backend code with Pipedream workflows, because it is so much easier to have an app calling a fixed URL than having to worry about all that infrastructure.

That being said, my workflows are amounting to almost 800 lines each, and editing, testing, developing online becomes a big nuisance very fast. Monaco is fine for a few hundred lines tops, and shouldn't be used for slightly larger codebases!

So... how am I supposed to keep working on my PD workflows if the alternative to wasting hours working with a feature-less IDE is paying 100 dollars a month (1200 upfront)?

[Stvad]

Seems like it's indeed the feature we've been waiting for. I'm very surprised to see it gated behind paywall it's such a foundational thing 🤔

[EricWu91]

@dylburger can you please provide updates on this? Even if it means to answer dryly "Yes, we heard your requests, we developed them, and we'll paywall them. Live with it.". At least we will have an answer.

[dylburger]

First, I really appreciate all the feedback! We have a small team for the size of the business we're operating, so it can take us longer than we'd like to groom GitHub issues.

We recently reduced the base price of all of our plans for new customers (see our pricing page), and are considering further expansion of GitHub support to lower tiers (e.g. for personal vs. organizational use). Like some of y'all have mentioned, since we operate a business, we think carefully about the value we're providing at each tier, gating some features at different tiers accordingly. But we hear the feedback, and I'm sharing all of this with the broader team so they see it.

Since this issue also operated as a catch-all for Git serialization and a workflow API + CLI, I'm closing this master ticket and opening separate tickets for the remaining work:

• [FEATURE] Create, update, and delete workflows via CLI #8736
• [FEATURE] Create, update, and delete workflows via API #8735

[EricWu91]

@dylburger , Thanks for the answer.
I noticed that you created two issues - one for CLI, one for API - but not for GIT.
Should we assume that the latter won't be receiving further considerations given that Github Sync is already out?

Furthermore, can we (and if so, where should we) follow up on this?

are considering further expansion of GitHub support to lower tiers (e.g. for personal vs. organizational use)

[mroy-seedbox]

The CLI & API features will effectively allow users to roll up their own GitHub integration anyway (or use any other version control provider), by creating a very simple CI/CD pipeline/workflow.

Which kinda defeats the purpose of paywalling this feature. 🤷‍♂️

[dylburger]

@EricWu91 We aren't planning on adding a GitHub issue to track that. We generally use GitHub only for new features, bugs, and integration requests, but not requests to transition features to different plans. I did share all this feedback with the team, though.

[EricWu91]

@dylburger, sorry for the confusion.

What I meant by - one for CLI, one for API - but not for GIT. was that I expected that all three topics in this issue ("Deploy / Update Workflows via Git, API, CLI") would have a different Github issue. Since you ommited a GIT-related issue, I thought it was being disregarded in favor of (the product) Github Sync.

As for a Github issue to track internal matters (i.e. pricing tiers), that makes all sense and I didn't expect otherwise :).

The CLI & API features will effectively allow users to roll up their own GitHub integration anyway (or use any other version control provider), by creating a very simple CI/CD pipeline/workflow.

@mroy-seedbox I would still pay for the ease of not having to deal with it, though. Just (still) not as much.

[dylburger]

Got it, so the CLI / API work will allow you to develop locally and serialize any workflow code to Git automatically (or any other VCS). In that model you’d push the code to Pipedream via your CI tool. Does that help or are you talking about something else?

[EricWu91]

My needs (and hopefully the ones of others in this issue) are just as you pointed out: Being able to develop locally, automatically versioning in a VCS (Git preferred, but not mandatory), and being able to still use Pipedream's infrastructure to run workflows.

If an API/CLI integration is made available, and it accomplishes the points above without an inherent need for a Git integration, I'm fine with it :).

[mroy-seedbox]

A well built VS Code extension for Pipedream might one day become a superior development environment than the Pipedream builder itself (due to its configurability & the wide availability of extensions, which is not feasible to add to the builder).

And that would allow users to use whatever form of version control they prefer (as @EricWu91 mentioned).

[AICrafterZheng]

+1

[jcable]

FYI I moved all my workflows out of Pipedream and re-implemented them with AWS lambda. Partly to bring the source under source code control and partly as part of a wider project that moved our graphql APIs out of Hasura on Heroku.
Julian (ogauk)

So I'm not really an interested party any longer.

[dylburger]

Since this issue is closed, I'm going to go ahead and lock this conversation so we make sure feedback about missing features is added to the relevant tickets:

• [FEATURE] Create, update, and delete workflows via CLI #8736
• [FEATURE] Create, update, and delete workflows via API #8735

[ctrlaltdylan]

We have released the ability to programmatically create and update workflows by REST API.

Read this end to end guide on how to create workflows programmatically using the REST API: https://pipedream.com/docs/rest-api/workflows

API Documentation: https://pipedream.com/docs/rest-api#workflows