+
@@ -123,7 +123,6 @@ export default {
margin-right: 1.2em;
}
-
a {
line-height: 1.4rem;
color: inherit;
@@ -149,6 +148,14 @@ export default {
.repo-link {
margin-left: 0.75rem;
}
+
+ .docs-version {
+ .dropdown-title, .nav-link {
+ font-family: 'JetBrains Mono', monospace;
+ font-weight: 700;
+ font-size: 1.2em;
+ }
+ }
}
@media (max-width: $MQMobile) {
diff --git a/docs/docs/.vuepress/theme/styles/index.styl b/docs/docs/.vuepress/theme/styles/index.styl
index 7d74bee551189..85e4ae0eb12df 100644
--- a/docs/docs/.vuepress/theme/styles/index.styl
+++ b/docs/docs/.vuepress/theme/styles/index.styl
@@ -4,6 +4,7 @@
@require './arrow'
@require './wrapper'
@require './toc'
+@require './navbar'
html, body
padding 0
diff --git a/docs/docs/.vuepress/theme/styles/navbar.styl b/docs/docs/.vuepress/theme/styles/navbar.styl
new file mode 100644
index 0000000000000..90d589e29f074
--- /dev/null
+++ b/docs/docs/.vuepress/theme/styles/navbar.styl
@@ -0,0 +1,16 @@
+.nav-link
+ .badge
+ font-size: 0.8em
+ font-weight: 800
+ text-transform: uppercase
+
+ // generated from https://www.cssportal.com/css-text-gradient-generator/
+ .primary
+ background: #3EAF7C;
+ background: -webkit-linear-gradient(to right, #3EAF7C 25%, #8F33B5 100%);
+ background: -moz-linear-gradient(to right, #3EAF7C 25%, #8F33B5 100%);
+ background: linear-gradient(to right, #3EAF7C 25%, #8F33B5 100%);
+ -webkit-background-clip: text;
+ -webkit-text-fill-color: transparent;
+
+
diff --git a/docs/docs/README.md b/docs/docs/README.md
index 9e6e0c5dd1cb4..40967cdd08762 100644
--- a/docs/docs/README.md
+++ b/docs/docs/README.md
@@ -5,84 +5,29 @@ next: false
# Overview
-## What is Pipedream?
+Pipedream is the fastest way to automate any process that connects APIs. Build and run workflows with code-level control when you need it, and no code when you don't.
-Pipedream is a production-scale serverless platform to connect APIs, remarkably fast.
+The Pipedream platform includes:
-1. Connect OAuth and key-based API accounts in seconds.
-2. Use connected accounts in Node.js code steps or no-code building blocks.
-3. Build and run workflows triggered on HTTP requests, schedules, app events and more.
-
-
-
-Watch a 4-minute demo or review our [quickstart guide](/quickstart/):
-
-
-
-
-
-
-
-
-
-
-
-
-
-## Is Pipedream for Me?
-
-We make it easy to connect APIs with code-level control when you need it — and no code when you don't. If you and your team want to stop writing boilerplate code, struggling with authentication and managing infrastructure for integrations, then Pipedream is for you.
-
-Developers with a working knowledge of Node.js or Javascript will get the most value from Pipedream (Python, TypeScript, and GitHub integration are coming soon).
-
-
+- A [serverless runtime](/code/) and [workflow service](/workflows/)
+- Open source [triggers](/workflows/triggers/) and [actions](/components/actions/) for [hundreds of integrated apps](https://pipedream.com/explore/)
+- One-click [OAuth and key-based authentication](/connected-accounts/) for hundreds of APIs (use tokens directly in code or with pre-built actions)
## Getting Started
-Sign up for a [free account (no credit card required)](https://pipedream.com/auth/signup) and complete our [quickstart guide](/quickstart/) to learn the basic patterns for workflow development:
+To get started, [sign up for a free account](https://pipedream.com/auth/signup) (no credit card required) and follow our [quickstart guide](/quickstart/) to create your first workflow.
+
+
-- Trigger workflows on HTTP requests, schedules and app events
-- Return a custom response from your workflow on HTTP requests
-- Use connected accounts in actions and code steps
-- Pass data between code steps and no code actions
-- Use npm packages in Node.js code steps
-- Scaffold an API request in Node.js
-- End workflow execution early
+One you understand the basics of workflow development, learn how to get more out of Pipedream:
-
-Pipedream offers a generous free tier (no credit card required) so you can test out the platform and use it for personal projects with no risk or commitment. As your needs grow, it's easy to upgrade to [paid plans](https://pipedream.com/pricing) to run without limits for individuals, teams and enterprises.
+- [Use code in workflows](/code/)
+- [Develop custom actions](/components/quickstart/nodejs/actions/)
+- [Develop custom triggers](/components/quickstart/nodejs/sources/)
## Use Cases
-Pipedream supports use cases from prototype to production and is trusted by 150k+ developers from startups to Fortune 500 companies:
+Pipedream supports use cases from prototype to production and is trusted by 200k+ developers from startups to Fortune 500 companies:
@@ -116,6 +61,12 @@ We hope is that by providing a generous free tier, you will not only get value f
Learn about [all the ways you can contribute](https://pipedream.com/contributing).
-## Getting Help
+## Support & Community
If you have any questions or feedback, please [reach out in our community forum](https://pipedream.com/community).
+
+## Service Status
+
+Pipedream operates a status page at [https://status.pipedream.com](https://status.pipedream.com). That page displays the uptime history and current status of every Pipedream service.
+
+When incidents occur, updates are published to the **#incidents** channel of [Pipedream's Slack Community](https://pipedream.com/support) and to the [@PipedreamStatus](https://twitter.com/PipedreamStatus) account on Twitter. On the status page itself, you can also subscribe to updates directly.
\ No newline at end of file
diff --git a/docs/docs/api/rest/README.md b/docs/docs/api/rest/README.md
index c829fa193a45f..01308c1b25b49 100644
--- a/docs/docs/api/rest/README.md
+++ b/docs/docs/api/rest/README.md
@@ -29,7 +29,7 @@ curl 'https://api.pipedream.com/v1/users/me' \
-H 'Authorization: Bearer
'
```
-Learn more about [API authentication](/api/auth)
+Learn more about [API authentication](/api/auth/)
## Required headers
@@ -1054,12 +1054,12 @@ Pipedream supports webhooks as a way to deliver events to a endpoint you own.
Webhooks are managed at an account-level, and you send data to these webhooks
using [subscriptions](#subscriptions).
-For example, you can run a Twitter [event source](/event-sources) that listens
+For example, you can run a Twitter [event source](/event-sources/) that listens
for new tweets. If you [subscribe](#subscriptions) the webhook to this source,
Pipedream will deliver those tweets directly to your webhook's URL without
running a workflow.
-[**See these tutorials**](/api/rest/webhooks) for examples.
+[**See these tutorials**](/api/rest/webhooks/) for examples.
### Create a webhook
diff --git a/docs/docs/api/rest/webhooks/README.md b/docs/docs/api/rest/webhooks/README.md
index b66e73cb6edd0..2478e22309d09 100644
--- a/docs/docs/api/rest/webhooks/README.md
+++ b/docs/docs/api/rest/webhooks/README.md
@@ -2,13 +2,13 @@
Pipedream supports webhooks as a way to deliver events to a endpoint you own. Webhooks are managed at an account-level, and you send data to these webhooks using [subscriptions](#subscriptions).
-For example, you can run a Twitter [event source](/event-sources) that listens for new tweets. If you [subscribe](#subscriptions) the webhook to this source, Pipedream will deliver those tweets directly to your webhook's URL without running a workflow.
+For example, you can run a Twitter [event source](/event-sources/) that listens for new tweets. If you [subscribe](#subscriptions) the webhook to this source, Pipedream will deliver those tweets directly to your webhook's URL without running a workflow.
[[toc]]
## Send events from an existing event source to a webhook
-[Event sources](/event-sources) source data from a service / API, emitting events that can trigger Pipedream workflows. For example, you can run a Github event source that emits an event anytime someone stars your repo, triggering a workflow on each new star.
+[Event sources](/event-sources/) source data from a service / API, emitting events that can trigger Pipedream workflows. For example, you can run a Github event source that emits an event anytime someone stars your repo, triggering a workflow on each new star.
**You can also send the events emitted by an event source to a webhook**.
diff --git a/docs/docs/apps/twitter/README.md b/docs/docs/apps/twitter/README.md
index 0c7d695b8cacd..292df15e7451c 100644
--- a/docs/docs/apps/twitter/README.md
+++ b/docs/docs/apps/twitter/README.md
@@ -2,7 +2,7 @@
Pipedream facilitates access to the Twitter API and Twitter events in two ways:
-- Within [Pipedream workflows](/workflows), you can use [prebuilt actions](/components/actions/) to post tweets, search tweets by keyword, and more. You can also run [any custom Node.js code](/workflows/steps/code/) that interacts with the Twitter API.
+- Within [Pipedream workflows](/workflows/), you can use [prebuilt actions](/components/actions/) to post tweets, search tweets by keyword, and more. You can also run [any custom Node.js code](/workflows/steps/code/) that interacts with the Twitter API.
- You can also _trigger_ Pipedream workflows on new tweets, new followers, and other Twitter events. This allows you to run custom code in response to these events.
[[toc]]
@@ -84,7 +84,7 @@ return await T.post("statuses/update", {
## Run code on new tweets, followers, and more
-Pipedream [event sources](/event-sources) emit Twitter events — tweets, follows, likes, and more — in real time. Sources can trigger [Pipedream workflows](/workflows), and can be accessed via [SSE](/api/sse/) and [REST APIs](/api/rest/).
+Pipedream [event sources](/event-sources/) emit Twitter events — tweets, follows, likes, and more — in real time. Sources can trigger [Pipedream workflows](/workflows/), and can be accessed via [SSE](/api/sse/) and [REST APIs](/api/rest/).
Create a new Twitter event source at [https://pipedream.com/sources](https://pipedream.com/sources), or by clicking on any of the links below:
diff --git a/docs/docs/code/README.md b/docs/docs/code/README.md
new file mode 100644
index 0000000000000..69f139c662d7c
--- /dev/null
+++ b/docs/docs/code/README.md
@@ -0,0 +1,20 @@
+# Overview
+
+_This document details how to use code in [workflow steps](/workflows/steps/), without leaving your browser. If you're building a [component](/components/), please [reference the component API docs](/components/api/)._
+
+Pipedream comes with thousands of prebuilt [triggers](/workflows/steps/triggers/) and [actions](/components/actions/) for [hundreds of apps](https://pipedream.com/apps). Often, these will be sufficient for building simple workflows.
+
+But sometimes you need to run your own custom logic. You may need to make an API request to fetch additional metadata about the event, transform data into a custom format, or end the execution of a workflow early under some conditions. **Code steps let you do this and more**.
+
+Code steps let you execute [Node.js v{{$site.themeConfig.NODE_VERSION}}](https://nodejs.org/) (JavaScript) code, Python, Go or even Bash right in a workflow.
+
+Choose a language to get started:
+
+* [Node.js (Javascript)](/code/nodejs/)
+* [Python](/code/python/)
+* [Go](/code/go/)
+* [Bash](/code/bash/)
+
+If you'd like to see another, specific language supported, please [let us know](https://pipedream.com/community).
+
+
diff --git a/docs/docs/code/bash/README.md b/docs/docs/code/bash/README.md
new file mode 100644
index 0000000000000..39c3363a84fff
--- /dev/null
+++ b/docs/docs/code/bash/README.md
@@ -0,0 +1,201 @@
+# Bash
+
+Prefer to write quick scripts in Bash? We've got you covered.
+
+You can run any Bash in a Pipedream step within your workflows.
+
+::: warning
+Bash steps are available in a limited alpha release.
+
+You can still run arbitrary Bash scripts, including [sharing data between steps](/code/bash/#sharing-data-between-steps) as well as [accessing environment variables](/code/bash/#using-environment-variables).
+
+However, you can't connect accounts, return HTTP responses, or take advantage of other features available in the [Node.js](/code/nodejs/) environment at this time. If you have any questions, find bugs or have feedback please [contact support](https://pipedream.com/support).
+:::
+
+## Adding a Bash code step
+
+1. Click the + icon to add a new step
+2. Click **Custom Code**
+3. In the new step, select the `bash` runtime in language dropdown
+
+## Logging and debugging
+
+When it comes to debugging Bash scripts, `echo` is your friend.
+
+Your `echo` statements will print their output in the workflow step results:
+
+```bash
+MESSAGE='Hello world'
+
+# The message will now be available in the "Result > Logs" area in the workflow step
+echo $MESSAGE
+```
+
+## Available binaries
+
+Bash steps come with many common and useful binaries preinstalled and available in `$PATH` for you to use out of the box. These binaries include but aren't limited to:
+
+* `curl` for making HTTP requests
+* `jq` for manipulating and viewing JSON data
+* `git` for interacting with remote repositories
+
+Unfortunately it is not possible to install packages from a package manager like `apt` or `yum`.
+
+If you need a package pre-installed in your Bash steps, [just ask us](https://pipedream.com/support).
+
+Otherwise, you can use the `/tmp` directory to download and install software from source.
+
+## Making an HTTP request
+
+`curl` is already preinstalled in Bash steps, we recommend using it for making HTTP requests in your code for sending or requesting data from APIs or webpages.
+
+### Making a `GET` request
+
+You can use `curl` to perform `GET` requests from websites or APIs directly.
+
+```bash
+# Get the current weather in San Francisco
+WEATHER=`curl --silent https://wttr.in/San\ Francisco\?format=3`
+
+echo $WEATHER
+# Produces:
+# San Francisco: 🌫 +48°F
+```
+
+::: tip
+Use the `--silent` flag with `curl` to suppress extra extra diagnostic information that `curl` produces when making requests.
+
+This enables you to only worry about the body of the response so you can visualize it with tools like `echo` or `jq`.
+:::
+
+### Making a `POST` request
+
+`curl` can also make `POST`s requests as well. The `-X` flag allow you to specify the HTTP method you'd like to use for an HTTP request.
+
+The `-d` flag is for passing data in the `POST` request.
+
+```bash
+curl --silent -X POST https://postman-echo.com/post -d 'name=Bulbasaur&id=1'
+
+# To store the API response in a variable, interpolate the response into a string and store it in variable
+RESPONSE=`curl --silent -X POST https://postman-echo.com/post -d 'name=Bulbasaur&id=1'`
+
+# Now the response is stored as a variable
+echo $RESPONSE
+```
+
+### Using API key authentication
+
+Some APIs require you to authenticate with a secret API key.
+
+`curl` has an `-h` flag where you can pass your API key as a token.
+
+For example, here's how to retrieve mentions from the Twitter API:
+
+```bash
+# Define the "Authorization" header to include your Twitter API key
+curl --silent -X POST -h "Authorization: Bearer $()" https://api.twitter.com/2/users/@pipedream/mentions
+```
+
+## Sharing data between steps
+
+A step can accept data from other steps in the same workflow, or pass data downstream to others.
+
+### Using data from another step
+
+In Bash steps, data from the initial workflow trigger and other steps are available in the `$PIPEDREAM_STEPS` environment variable.
+
+In this example, we'll pretend this data is coming into our HTTP trigger via a POST request.
+
+```json
+{
+ "id": 1,
+ "name": "Bulbasaur",
+ "type": "plant"
+}
+```
+
+In our Bash script, we can access this data via the `$PIPEDREAM_STEPS` file. Specifically, this data from the POST request into our workflow is available in the `trigger` object.
+
+```bash
+echo $PIPEDREAM_STEPS | jq .trigger.event
+
+# Results in { id: 1, name: "Bulbasaur", type: "plant" }
+```
+
+::: tip
+The period (`.`) in front the `trigger.event` in the example is not a typo. This is to define the starting point for `jq` to traverse down the JSON in the HTTP response.
+:::
+
+### Sending data downstream to other steps
+
+To share data for future steps to use downstream, append it to the `$PIPEDREAM_EXPORTS` file.
+
+```bash
+# Retrieve the data from an API and store it in a variable
+DATA=`curl --silent https://pokeapi.co/api/v2/pokemon/charizard`
+
+# Write data to $PIPEDREAM_EXPORTS to share with steps downstream
+EXPORT="key:json=${DATA}"
+echo $EXPORT >> $PIPEDREAM_EXPORTS
+```
+
+::: warning
+Not all data types can be stored in the `$PIPEDREAM_EXPORTS` data shared between workflow steps.
+
+For the best experience, we recommend only exporting strings from Bash steps that can be serialized to JSON.
+
+[Read more details on step limitations here.](/workflows/steps/#limitations-on-step-exports)
+:::
+
+## Using environment variables
+
+You can leverage any [environment variables defined in your Pipedream account](/environment-variables/#environment-variables) in a bash step. This is useful for keeping your secrets out of code as well as keeping them flexible to swap API keys without having to update each step individually.
+
+To access them, just append the `$` in front of the environment variable name.
+
+```bash
+echo $POKEDEX_API_KEY
+```
+
+Or an even more useful example, using the stored environment variable to make an authenticated API request.
+
+```bash
+curl --silent -X POST -h "Authorization: Bearer $TWITTER_API_KEY" https://api.twitter.com/2/users/@pipedream/mentions
+```
+
+## Raising exceptions
+
+You may need to stop your step immediately. You can use the normal `exit` function available in Bash to quit the step prematurely.
+
+```bash
+echo "Exiting now!" 1>&2
+exit 1
+```
+:::warning
+Using `exit` to quit a Bash step early _won't_ stop the execution of the rest of the workflow.
+
+Exiting a Bash step will only apply that particular step in the workflow.
+:::
+
+This will exit the step and output the error message to `stderr` which will appear in the results of the step in the workflow.
+
+## File storage
+
+If you need to download and store files, you can place them in the `/tmp` directory.
+
+### Writing a file to /tmp
+
+For example, to download a file to `/tmp` using `curl`
+
+```bash
+# Download the current weather in Cleveland in PNG format
+curl --silent https://wttr.in/Cleveland.png --output /tmp/weather.png
+
+# Output the contents of /tmp to confirm the file is there
+ls /tmp
+```
+
+:::warning
+The `/tmp` directory does not have unlimited storage. Please refer to the [disk limits](/limits/#disk) for details.
+:::
diff --git a/docs/docs/code/go/README.md b/docs/docs/code/go/README.md
new file mode 100644
index 0000000000000..b8225ee06af01
--- /dev/null
+++ b/docs/docs/code/go/README.md
@@ -0,0 +1,438 @@
+# Go
+
+**Anything you can do in Go, you can do in a Pipedream Workflow**. You can use any of [Go packages available](https://pkg.go.dev/) with a simple `import` no `go get` needed.
+
+Pipedream supports [Go v{{$site.themeConfig.GO_LANG_VERSION}}](https://go.dev) in workflows.
+
+::: warning
+Go steps are available in a limited alpha release.
+
+You can still run arbitrary Go code, including [sharing data between steps](/code/go/#sharing-data-between-steps) as well as [accessing environment variables](/code/go/#using-environment-variables).
+
+However, features available in [Node.js steps](/code/nodejs) like `$.respond`, `$.end`, and `$.auth` are not yet available in bash. If you have any questions please [contact support](https://pipedream.com/support).
+:::
+
+
+## Adding a Go code step
+
+1. Click the + icon to add a new step
+2. Click "Custom Code"
+3. In the new step, select the `golang` language runtime in language dropdown
+
+## Logging and debugging
+
+You can use `fmt.Println` at any time to log information as the script is running.
+
+The output for the `fmt.Println` **Logs** will appear in the `Results` section just beneath the code editor.
+
+:::tip
+Don't forget to import the `fmt` package in order to run `fmt.Println`.
+
+```go{3}
+ package main
+
+ import "fmt"
+
+ func main() {
+ fmt.Println("Hello World!")
+ }
+```
+:::
+
+## Using third party packages
+
+You can use any packages from [Go package registry](https://pkg.go.dev). This includes popular choices such as:
+
+* [`net/http` for making HTTP requests](https://pkg.go.dev/net/http#pkg-overview/)
+* [`encoding/json` for encoding and decoding JSON](https://pkg.go.dev/encoding/json)
+* [`database/sql` for reading and writing to SQL databases](https://pkg.go.dev/database/sql@go1.17.6)
+
+To use a Go package, just `import` it in your step's code:
+
+```go
+import "net/http"
+```
+
+And that's it.
+
+## Making an HTTP request
+
+We recommend using the `http` HTTP client package included in the Go Standard Library for making HTTP requests.
+
+### Making a `GET` request
+
+You'll typically use `GET` requests to retrieve data from an API:
+
+```go
+package main
+
+import (
+ "net/http" // HTTP client
+ "io/ioutil" // Reads the body of the response
+ "log" // Logger
+)
+
+func main() {
+ resp, err := http.Get("https://swapi.dev/api/people/1")
+
+ if err != nil {
+ log.Fatalln(err)
+ }
+ defer resp.Body.Close()
+
+ body, err := ioutil.ReadAll(resp.Body)
+ if err != nil {
+ log.Fatalln(err)
+ }
+
+ // The response status code is logged in your Pipedream step results:
+ log.Println(resp.Status)
+
+ // The response is logged in your Pipedream step results:
+ sb := string(body)
+ log.Println(sb)
+}
+```
+
+### Making a `POST` request
+
+```go
+package main
+
+import (
+ "bytes"
+ "encoding/json"
+ "io/ioutil"
+ "log"
+ "net/http"
+)
+
+func main() {
+ // JSON encode our payload
+ payload, _ := json.Marshal(map[string]string{
+ "name": "Bulbasaur",
+ })
+ payloadBuf:= bytes.NewBuffer(payload)
+
+ // Send the POST request
+ resp, err := http.Post("https://postman-echo.com/post", "application/json", payloadBuf)
+
+ if err != nil {
+ log.Fatalln(err)
+ }
+ defer resp.Body.Close()
+
+ // Read the response body
+ body, err := ioutil.ReadAll(resp.Body)
+ if err != nil {
+ log.Fatalln(err)
+ }
+ // Convert the body into a string
+ sb := string(body)
+ // Log the body to our Workflow Results
+ log.Println(sb)
+}
+```
+
+### Sending files
+
+You can send files stored in the `/tmp` directory in an HTTP request:
+
+```go
+
+package main
+
+import (
+ "os"
+ "log"
+ "mime/multipart"
+ "bytes"
+ "io"
+ "net/http"
+)
+
+func main() {
+ // Instantiate a new HTTP client, body and form writer
+ client := http.Client{}
+ body := &bytes.Buffer{}
+ writer := multipart.NewWriter(body)
+
+ // Create an empty file to start
+ fw, _:= writer.CreateFormFile("file", "go-logo.svg")
+
+
+ // Retrieve a previously saved file from workflow storage
+ file, _ := os.Open("/tmp/go-logo.svg")
+
+ // Close multipart form writer
+ writer.Close()
+
+ _, _ = io.Copy(fw, file)
+
+ // Send the POST request
+ req, _:= http.NewRequest("POST", "https://postman-echo.com/post", bytes.NewReader(body.Bytes()))
+
+ req.Header.Set("Content-Type", writer.FormDataContentType())
+ _, err := client.Do(req)
+ if err != nil {
+ log.Fatalln(err)
+ }
+}
+```
+
+## Sharing data between steps
+
+A step can accept data from other steps in the same workflow, or pass data downstream to others.
+
+This makes your steps even more powerful, you can compose new workflows and reuse steps.
+
+### Using data from another step
+
+Data from the initial workflow trigger and other steps are available in the `pipedream-go` package.
+
+In this example, we'll pretend this data is coming into our HTTP trigger via POST request.
+
+```json
+{
+ "id": 1,
+ "name": "Bulbasaur",
+ "type": "plant"
+}
+```
+
+You can access this data in the `Steps` variable from the `pd` package. Specifically, this data from the POST request into our workflow is available in the `trigger` map.
+
+```go
+package main
+
+import (
+ "fmt"
+ "github.com/PipedreamHQ/pipedream-go"
+)
+
+func main() {
+ // Access previous step data using pd.Steps
+ fmt.Println(pd.Steps["trigger"])
+}
+```
+
+### Sending data downstream to other steps
+
+To share data for future steps to use, call the Export function from pd package:
+
+```go
+package main
+
+import (
+ "encoding/json"
+ "github.com/PipedreamHQ/pipedream-go"
+ "io/ioutil"
+ "net/http"
+)
+
+func main() {
+ // Use a GET request to look up the latest data on Charizard
+ resp, _ := http.Get("https://pokeapi.co/api/v2/pokemon/charizard")
+ body, _ := ioutil.ReadAll(resp.Body)
+
+ // Unmarshal the JSON into a struct
+ var data map[string]interface{}
+ json.Unmarshal(body, &data)
+
+ // Expose the pokemon data downstream to others steps in the "pokemon" key from this step
+ pd.Export("pokemon", data)
+}
+```
+
+Now this `pokemon` data is accessible to downstream steps within `pd.Steps["code"]["pokemon"]`
+
+::: warning
+Not all data types can be stored in the `Steps` data shared between workflow steps.
+
+For the best experience, we recommend only [exporting structs that can be marshalled into JSON](https://go.dev/blog/json).
+
+[Read more details on step limitations here.](/workflows/steps/#limitations-on-step-exports)
+:::
+
+## Using environment variables
+
+You can leverage any [environment variables defined in your Pipedream account](/environment-variables/#environment-variables) in a Go step. This is useful for keeping your secrets out of code as well as keeping them flexible to swap API keys without having to update each step individually.
+
+To access them, use the `os` package.
+
+```go
+package main
+
+import (
+ "log"
+ "os"
+)
+
+func main() {
+ log.PrintLn(os.Getenv('TWITTER_API_KEY'))
+}
+```
+
+### Using API key authentication
+
+If a particular service requires you to use an API key, you can pass it via the HTTP request.
+
+This proves your identity to the service so you can interact with it:
+
+```go
+package main
+
+import (
+ "os"
+ "bytes"
+ "encoding/json"
+ "io/ioutil"
+ "log"
+ "net/http"
+ "fmt"
+)
+
+func main() {
+ // Access the Twitter API key from the environment
+ const apiKey := os.Getenv('TWITTER_API_KEY'))
+
+ // JSON encode our payload
+ payload, _ := json.Marshal(map[string]string{
+ "name": "Bulbasaur",
+ })
+ payloadBuf:= bytes.NewBuffer(payload)
+
+ // Send the POST request
+ req, err := http.NewRequest("POST", "https://api.twitter.com/2/users/@pipedream/mentions'", payloadBuf)
+
+ // Build the headers in order to authenticate properly
+ req.Header = http.Header{
+ "Content-Type": []string{"application/json"},
+ "Authorization": []string{fmt.Sprintf("Bearer %s", apiKey)},
+ }
+
+ client := http.Client{}
+ resp, err := client.Do(req)
+
+ if err != nil {
+ log.Fatalln(err)
+ }
+ // Don't forget to close the request after the function is finished
+ defer resp.Body.Close()
+
+ // Read the response body
+ body, err := ioutil.ReadAll(resp.Body)
+ if err != nil {
+ log.Fatalln(err)
+ }
+ // Convert the body into a string
+ sb := string(body)
+ // Log the body to our Workflow Results
+ log.Println(sb)
+}
+```
+
+## Handling errors
+
+You may need to exit a workflow early, use the `os.Exit` to exit the `main` function with a specific error code.
+
+```go
+package main
+
+import (
+ "fmt"
+ "os"
+)
+
+func main() {
+ os.Exit(1)
+ fmt.Println("This message will not be logged.")
+}
+
+```
+
+The step will quit at the time `os.Exit` is called. In this example, the exit code `1` will appear in the **Results** of the step.
+
+
+## File storage
+
+You can also store and read files with Go steps. This means you can upload photos, retrieve datasets, accept files from an HTTP request and more.
+
+The `/tmp` directory is accessible from your workflow steps for saving and retrieving files.
+
+You have full access to read and write both files in `/tmp`.
+
+### Writing a file to `/tmp`
+
+```go
+package main
+
+import (
+ "io"
+ "net/http"
+ "os"
+ "fmt"
+)
+func main() {
+ // Define where the file is and where to save it
+ fileUrl := "https://golangcode.com/go-logo.svg"
+ filePath := "/tmp/go-logo.svg"
+
+ // Download the file
+ resp, err := http.Get(fileUrl)
+ if err != nil {
+ fmt.Println(err)
+ }
+
+ // Don't forget to the close the HTTP connection at the end of the function
+ defer resp.Body.Close()
+
+ // Create the empty file
+ out, err := os.Create(filePath)
+ if err != nil {
+ fmt.Println(err)
+ }
+
+ // Don't forget to close close the file
+ defer out.Close()
+
+ // Write the file data to file
+ _, err = io.Copy(out, resp.Body)
+ if err != nil {
+ fmt.Println(err)
+ }
+}
+```
+
+Now `/tmp/go-logo.svg` holds the official Go logo.
+
+### Reading a file from /tmp
+
+You can also open files you have previously stored in the `/tmp` directory. Let's open the `go-logo.svg` file.
+
+```go
+package main
+
+import (
+ "os"
+ "log"
+)
+
+func main() {
+ // Open the file
+ data, err := os.ReadFile("/tmp/go-logo.svg")
+
+ if e != nil {
+ log.Fatalln(e)
+ }
+
+ // Print it's contents to the logs
+ log.Println(string(data))
+}
+```
+
+### `/tmp` limitations
+
+The `/tmp` directory can store up to {{$site.themeConfig.TMP_SIZE_LIMIT}} of storage. Also the storage may be wiped or may not exist between workflow executions.
+
+To avoid errors, assume that the `/tmp` directory is empty between workflow runs.
diff --git a/docs/docs/workflows/steps/code/README.md b/docs/docs/code/nodejs/README.md
similarity index 54%
rename from docs/docs/workflows/steps/code/README.md
rename to docs/docs/code/nodejs/README.md
index 32f4256170396..d9d2028572feb 100644
--- a/docs/docs/workflows/steps/code/README.md
+++ b/docs/docs/code/nodejs/README.md
@@ -1,99 +1,128 @@
-# Code
+# Writing Node.js in Steps
-_This document details how to use Node.js code steps. If you're building a [component](/components/), please [reference the component API docs](/components/api/)._
+Pipedream supports [Node.js v{{$site.themeConfig.NODE_VERSION}}](https://nodejs.org/).
-Pipedream comes with thousands of prebuilt [triggers](/workflows/steps/triggers/) and [actions](/components/actions/). Often, these will be sufficient for building simple workflows.
-But sometimes you need to run your own custom logic. You may need to make an API request to fetch additional metadata about the event, transform data into a custom format, or end the execution of a workflow early under some conditions. **Code steps let you do this and more**.
-Code steps let you execute [Node.js v{{$site.themeConfig.NODE_VERSION}}](https://nodejs.org/) (JavaScript) code, using JavaScript's extensive [npm](https://www.npmjs.com/) package ecosystem within your code. Virtually anything you can do in Node.js, you can do in a code step.
+**Anything you can do with Node.js, you can do in a workflow**. This includes using most of [npm's 400,000+ packages](#using-npm-packages).
-[[toc]]
-## Language Support
-Pipedream supports [Node.js v{{$site.themeConfig.NODE_VERSION}}](https://nodejs.org/).
+JavaScript is one of the [most used](https://insights.stackoverflow.com/survey/2019#technology-_-programming-scripting-and-markup-languages) [languages](https://github.blog/2018-11-15-state-of-the-octoverse-top-programming-languages/) in the world, with a thriving community and [extensive package ecosystem](https://www.npmjs.com). If you work on websites and know JavaScript well, Pipedream makes you a full stack engineer. If you've never used JavaScript, see the [resources below](#new-to-javascript).
+::: tip
It's important to understand the core difference between Node.js and the JavaScript that runs in your web browser: **Node doesn't have access to some of the things a browser expects, like the HTML on the page, or its URL**. If you haven't used Node before, be aware of this limitation as you search for JavaScript examples on the web.
-
-**Anything you can do with Node.js, you can do in a workflow**. This includes using most of [npm's 400,000+ packages](#using-npm-packages).
-
-If you'd like to see another, specific language supported, please [let us know](https://pipedream.com/community).
-
-JavaScript is one of the [most used](https://insights.stackoverflow.com/survey/2019#technology-_-programming-scripting-and-markup-languages) [languages](https://github.blog/2018-11-15-state-of-the-octoverse-top-programming-languages/) in the world, with a thriving community and [extensive package ecosystem](https://www.npmjs.com). If you work on websites and know JavaScript well, Pipedream makes you a full stack engineer. If you've never used JavaScript, see the [resources below](#new-to-javascript).
+:::
## Adding a code step
1. Click the **+** button below any step of your workflow.
-2. Select the option to **Run Node.js code**.
-
-
-
-
+
+
-
-
-
-
-
-
+
+
+> The `Promise.allSettled()` method returns a promise that resolves after all of the given promises have either fulfilled or rejected, with an array of objects that each describes the outcome of each promise.
> It is typically used when you have multiple asynchronous tasks that are not dependent on one another to complete successfully, or you'd always like to know the result of each promise.
> In comparison, the Promise returned by `Promise.all()` may be more appropriate if the tasks are dependent on each other / if you'd like to immediately reject upon any of them rejecting.
@@ -234,17 +256,21 @@ The Mozilla docs expand on the difference between these methods, and when you ma
import axios from "axios";
import FormData from "form-data";
-const formData = new FormData();
-formData.append("name", "Luke Skywalker");
-
-const headers = formData.getHeaders();
-const config = {
- method: "POST",
- url: params.url,
- headers,
- data: formData,
-};
-return await axios(config);
+export default defineComponent({
+ async run({ steps, $ }) {
+ const formData = new FormData();
+ formData.append("name", "Luke Skywalker");
+
+ const headers = formData.getHeaders();
+ const config = {
+ method: "POST",
+ url: "https://example.com",
+ headers,
+ data: formData,
+ };
+ return await axios(config);
+ }
+});
```
[Copy this workflow](https://pipedream.com/@dylburger/send-a-multipart-form-data-request-p_WxCQRyr/edit) to run this example.
@@ -262,8 +288,8 @@ import got from "got";
// DOWNLOAD
const pipeline = promisify(stream.pipeline);
await pipeline(
- got.stream(params.downloadURL),
- fs.createWriteStream(params.filePath)
+ got.stream("https://example.com"),
+ fs.createWriteStream('/tmp/file.html')
);
```
@@ -280,17 +306,21 @@ import axios from "axios";
import fs from "fs";
import FormData from "form-data";
-const formData = new FormData();
-formData.append("file", fs.createReadStream(params.pathToYourFile));
-const headers = formData.getHeaders();
-
-const config = {
- method: "POST",
- url: params.url,
- headers,
- data: formData,
-};
-return await axios(config);
+export default defineComponent({
+ async run({ steps, $ }) {
+ const formData = new FormData();
+ formData.append("file", fs.createReadStream('/tmp/file.pdf'));
+ const headers = formData.getHeaders();
+
+ const config = {
+ method: "POST",
+ url: "https://example.com",
+ headers,
+ data: formData,
+ };
+ return await axios(config);
+ }
+});
```
[Copy this workflow](https://pipedream.com/@dylburger/stream-a-file-upload-p_6lC1d2Z/edit) to run this example.
@@ -303,17 +333,21 @@ By default, HTTP requests made from Pipedream can come from a range of IP addres
```javascript
import axios from "axios";
-
import httpsProxyAgent from "https-proxy-agent";
-const agent = new httpsProxyAgent(`http://${user}:${pass}@${host}:${port}`);
-const config = {
- method: "GET",
- url,
- httpsAgent: agent,
-};
+export default defineComponent({
+ async run({ steps, $ }) {
+ const agent = new httpsProxyAgent(`http://${user}:${pass}@${host}:${port}`);
-const resp = await axios.request(config);
+ const config = {
+ method: "GET",
+ url: "https://example.com",
+ httpsAgent: agent,
+ };
+
+ return await axios.request(config);
+ }
+});
```
**If you don't have access to an HTTP proxy, and you are a paying Pipedream customer, [reach out to our team](https://pipedream.com/support)**. We operate a proxy that you can use for HTTP requests made through Pipedream.
@@ -327,43 +361,6 @@ By default, [HTTP requests made from Pipedream can come from a large range of IP
- [Use an HTTP proxy to proxy requests](#use-an-http-proxy-to-proxy-requests-through-another-host)
- If you don't need to access the HTTP response data, you can [use `$send.http()`](/destinations/http/) to send requests from a [limited set of IP addresses](/destinations/http/#ip-addresses-for-pipedream-http-requests).
-## Forward an incoming HTTP request to another URL
-
-Often, you'll want to forward an incoming HTTP request from Pipedream to another service, with the same HTTP method, headers, body, etc. [This workflow](https://pipedream.com/@dylburger/forward-http-request-issue-http-response-p_BjC8Pp/edit) does just that.
-
-Once you **Copy** the workflow, enter the **URL** where you'd like to forward an HTTP request in the `forward_http_request` step. Every HTTP request you send to the workflow's HTTP endpoint will get forwarded to that URL.
-
-```javascript
-const config = {
- method: event.method || "POST",
- url: params.url,
-};
-
-const { query } = event;
-if (Object.keys(query).length) {
- config.params = query;
-}
-
-// Headers, removing the original Host
-const { headers } = event;
-delete headers.host;
-if (Object.keys(headers).length) {
- config.headers = headers;
-}
-
-if (event.body) config.data = event.body;
-
-return await require("@pipedreamhq/platform").axios(this, config);
-```
-
-You can modify this workflow in any way you'd like. For example, if you wanted to forward only certain types of requests, you could add another Node.js code step before the `forward_http_request` step, [ending the workflow early](/workflows/steps/code/#end) if the request doesn't contain a specific key in the HTTP payload:
-
-```javascript
-if (!event.body.myImportantData) {
- $end("myImportantData not present in HTTP payload. Exiting");
-}
-```
-
## Stream a downloaded file directly to another URL
Sometimes you need to upload a downloaded file directly to another service, without processing the downloaded file. You could [download the file](#download-a-file-to-the-tmp-directory) and then [upload it](#upload-a-file-from-the-tmp-directory) to the other URL, but these intermediate steps are unnecessary: you can just stream the download to the other service directly, without saving the file to disk.
@@ -375,18 +372,21 @@ This method is especially effective for large files that exceed the [limits of t
```javascript
import stream from "stream";
import { promisify } from "util";
-import fs from "fs";
import got from "got";
-const pipeline = promisify(stream.pipeline);
+export default defineComponent({
+ async run({ steps, $ }) {
+ const pipeline = promisify(stream.pipeline);
-await pipeline(
- got.stream(params.downloadURL),
- got.stream.post(params.uploadURL)
-);
+ await pipeline(
+ got.stream("https://example.com"),
+ got.stream.post("https://example2.com")
+ );
+ }
+});
```
-You'll be asked to provide the **Download URL** — the URL of the content you want to download — and the **Upload URL** — the place you want to upload the content to. `got` streams the content directly, downloading the file using a `GET` request and uploading it as a `POST` request.
+You'll want to replace `https://example.com` with the URL you'd like to stream data from, and replace `https://example2.com` with the URL you'd like to send the data _to_. `got` streams the content directly, downloading the file using a `GET` request and uploading it as a `POST` request.
If you need to modify this behavior, [see the `got` Stream API](https://github.com/sindresorhus/got#gotstreamurl-options).
@@ -397,15 +397,19 @@ By default, `axios` throws an error when the HTTP response code is in the 400-50
```javascript
import axios from "axios";
-const resp = await axios({
- url: "https://httpstat.us/400",
- validateStatus: () => true, // will not throw error when axios gets a 400+ status code (the default behavior)
-})
-if (resp.status >= 400) {
- this.debug = resp
- throw new Error(JSON.stringify(resp.data)) // This can be modified to throw any error you'd like
-}
-return resp
+export default defineComponent({
+ async run({ steps, $ }) {
+ const resp = await axios({
+ url: "https://httpstat.us/400",
+ validateStatus: () => true, // will not throw error when axios gets a 400+ status code (the default behavior)
+ });
+ if (resp.status >= 400) {
+ this.debug = resp;
+ throw new Error(JSON.stringify(resp.data)); // This can be modified to throw any error you'd like
+ }
+ return resp;
+ }
+});
```
See [the `axios` docs](https://github.com/axios/axios#request-config) for more details.
diff --git a/docs/docs/workflows/steps/code/images/console-dir.png b/docs/docs/code/nodejs/images/console-dir.png
similarity index 100%
rename from docs/docs/workflows/steps/code/images/console-dir.png
rename to docs/docs/code/nodejs/images/console-dir.png
diff --git a/docs/docs/workflows/steps/code/images/console-log-error.png b/docs/docs/code/nodejs/images/console-log-error.png
similarity index 100%
rename from docs/docs/workflows/steps/code/images/console-log-error.png
rename to docs/docs/code/nodejs/images/console-log-error.png
diff --git a/docs/docs/workflows/steps/code/images/dollar-end.png b/docs/docs/code/nodejs/images/dollar-end.png
similarity index 100%
rename from docs/docs/workflows/steps/code/images/dollar-end.png
rename to docs/docs/code/nodejs/images/dollar-end.png
diff --git a/docs/docs/workflows/steps/code/images/exception-in-code-cell.png b/docs/docs/code/nodejs/images/exception-in-code-cell.png
similarity index 100%
rename from docs/docs/workflows/steps/code/images/exception-in-code-cell.png
rename to docs/docs/code/nodejs/images/exception-in-code-cell.png
diff --git a/docs/docs/workflows/steps/code/images/exception.png b/docs/docs/code/nodejs/images/exception.png
similarity index 100%
rename from docs/docs/workflows/steps/code/images/exception.png
rename to docs/docs/code/nodejs/images/exception.png
diff --git a/docs/docs/workflows/steps/code/images/new-button.png b/docs/docs/code/nodejs/images/new-button.png
similarity index 100%
rename from docs/docs/workflows/steps/code/images/new-button.png
rename to docs/docs/code/nodejs/images/new-button.png
diff --git a/docs/docs/workflows/steps/code/images/new-code-step.png b/docs/docs/code/nodejs/images/new-code-step.png
similarity index 100%
rename from docs/docs/workflows/steps/code/images/new-code-step.png
rename to docs/docs/code/nodejs/images/new-code-step.png
diff --git a/docs/docs/workflows/steps/code/images/syntax-error.png b/docs/docs/code/nodejs/images/syntax-error.png
similarity index 100%
rename from docs/docs/workflows/steps/code/images/syntax-error.png
rename to docs/docs/code/nodejs/images/syntax-error.png
diff --git a/docs/docs/workflows/steps/code/images/this-checkpoint-observability.png b/docs/docs/code/nodejs/images/this-checkpoint-observability.png
similarity index 100%
rename from docs/docs/workflows/steps/code/images/this-checkpoint-observability.png
rename to docs/docs/code/nodejs/images/this-checkpoint-observability.png
diff --git a/docs/docs/code/nodejs/images/user-input-props-example.png b/docs/docs/code/nodejs/images/user-input-props-example.png
new file mode 100644
index 0000000000000..ef987b0cc1afb
Binary files /dev/null and b/docs/docs/code/nodejs/images/user-input-props-example.png differ
diff --git a/docs/docs/workflows/steps/code/nodejs/sharing-code/README.md b/docs/docs/code/nodejs/sharing-code/README.md
similarity index 100%
rename from docs/docs/workflows/steps/code/nodejs/sharing-code/README.md
rename to docs/docs/code/nodejs/sharing-code/README.md
diff --git a/docs/docs/workflows/steps/code/nodejs/working-with-files/README.md b/docs/docs/code/nodejs/working-with-files/README.md
similarity index 84%
rename from docs/docs/workflows/steps/code/nodejs/working-with-files/README.md
rename to docs/docs/code/nodejs/working-with-files/README.md
index 3ee8912fc0861..1bd23597bf5c9 100644
--- a/docs/docs/workflows/steps/code/nodejs/working-with-files/README.md
+++ b/docs/docs/code/nodejs/working-with-files/README.md
@@ -20,9 +20,13 @@ Use the [`fs` module](https://nodejs.org/api/fs.html) to write data to `/tmp`:
import fs from "fs"
import { file } from 'tmp-promise'
-const { path, cleanup } = await file();
-await fs.promises.appendFile(path, Buffer.from("hello, world"))
-await cleanup();
+defineComponent({
+ async run({ steps, $ }) {
+ const { path, cleanup } = await file();
+ await fs.promises.appendFile(path, Buffer.from("hello, world"))
+ await cleanup();
+ }
+});
```
## Listing files in `/tmp`
@@ -32,7 +36,11 @@ Return a list of the files saved in `/tmp`:
```javascript
import fs from "fs";
-return fs.readdirSync("/tmp");
+defineComponent({
+ async run({ steps, $ }) {
+ return fs.readdirSync("/tmp");
+ }
+});
```
## Reading a file from `/tmp`
@@ -42,7 +50,12 @@ This example uses [step exports](/workflows/steps/#step-exports) to return the c
```javascript
import fs from "fs";
-this.fileData = (await fs.promises.readFile('/tmp/your-file')).toString()
+defineComponent({
+ async run({ steps, $ }) {
+ const files = await fs.promises.readFile('/tmp/your-file');
+ this.fileData = files.toString()
+ }
+});
```
## Delete a file
@@ -50,7 +63,11 @@ this.fileData = (await fs.promises.readFile('/tmp/your-file')).toString()
```javascript
import fs from "fs";
-return await fs.promises.unlink('/tmp/your-file')
+defineComponent({
+ async run({ steps, $ }) {
+ return await fs.promises.unlink('/tmp/your-file');
+ }
+});
```
## Download a file to `/tmp`
diff --git a/docs/docs/code/python/README.md b/docs/docs/code/python/README.md
new file mode 100644
index 0000000000000..ad0832239741c
--- /dev/null
+++ b/docs/docs/code/python/README.md
@@ -0,0 +1,272 @@
+# Python
+
+**Anything you can do in Python can be done in a Pipedream Workflow**. This includes using any of the [350,000+ PyPi packages available](https://pypi.org/) in your Python powered workflows.
+
+Pipedream supports [Python v{{$site.themeConfig.PYTHON_VERSION}}](https://www.python.org) in workflows.
+
+::: warning
+Python steps are available in a limited alpha release.
+
+You can still run arbitrary Python code, including [sharing data between steps](/code/python/#sharing-data-between-steps) as well as [accessing environment variables](/code/python/#using-environment-variables).
+
+However, you can't connect accounts, return HTTP responses, or take advantage of other features available in the [Node.js](/code/nodejs/) environment at this time. If you have any questions please [contact support](https://pipedream.com/support).
+:::
+
+
+## Adding a Python code step
+
+1. Click the + icon to add a new step
+2. Click **Custom Code**
+3. In the new step, select the `python` language runtime in language dropdown
+
+## Logging and debugging
+
+You can use `print` at any time in a Python code step to log information as the script is running.
+
+The output for the `print` **logs** will appear in the `Results` section just beneath the code editor.
+
+
+
+
