Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

SQL Proxy Docs #11397

Closed
wants to merge 16 commits into from
Closed

SQL Proxy Docs #11397

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
0d7b169
first cut of sql-proxy docs
dannyroosevelt Apr 9, 2024
d9f0a2e
Update databases.mdx
dannyroosevelt Apr 10, 2024
48e897e
Merge branch 'master' into danny/sql-proxy-docs
dannyroosevelt Apr 10, 2024
7796190
Database docs
dannyroosevelt May 2, 2024
ea78e9e
Update _meta.json
dannyroosevelt May 2, 2024
fcbc857
Merge branch 'master' into danny/sql-proxy-docs
dannyroosevelt May 2, 2024
908fbf4
Merge branch 'master' into danny/sql-proxy-docs
dannyroosevelt May 14, 2024
36b6bf1
Rev on docs
dannyroosevelt May 14, 2024
a9f1d90
Update index.mdx
dannyroosevelt May 14, 2024
7375722
Docs updates
dannyroosevelt May 14, 2024
9d693dd
More docs
dannyroosevelt May 23, 2024
4660c64
Update working-with-sql.mdx
dannyroosevelt May 24, 2024
4ea91a6
BYOA and SQL Proxy Docs (#12118)
dannyroosevelt May 24, 2024
0c0e675
Update working-with-sql.mdx
dannyroosevelt May 24, 2024
b6d369f
More docs
dannyroosevelt May 25, 2024
bad66e3
Docs
dannyroosevelt May 28, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 6 additions & 3 deletions docs-v2/pages/_meta.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,12 +23,15 @@
"sources": "Sources",
"event-history": "Event History",
"http": "HTTP",
"apps": "Integrations",
"rest-api": "API Reference",
"cli": "CLI",
"environment-variables": {
"title": "Environment Variables"
},
"databases": {
"title": "Databases"
},
"apps": "Integrations",
"rest-api": "API Reference",
"cli": "CLI",
"destinations": "Destinations",
"user-settings": "User and Billing Settings",
"troubleshooting": "Troubleshooting",
Expand Down
32 changes: 32 additions & 0 deletions docs-v2/pages/databases.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
# Working with Databases
Connecting to databases is a fundamental aspect of creating powerful workflows for your applications. Whether you're storing application data, querying user information, or analyzing event logs, most workflows require querying data at some step.

Every Pipedream workflow is deployed to its own virtual machine in AWS. This means your workflow runs in its own execution environment with dedicated RAM and disk space, isolated from other users' workflows.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While this is definitely true, I don't think it's focused to how to connect to a Database in a workflow, let's keep the intro focused on the tutorial, then go into the weeds about security.

Suggested change
Every Pipedream workflow is deployed to its own virtual machine in AWS. This means your workflow runs in its own execution environment with dedicated RAM and disk space, isolated from other users' workflows.


However, outbound traffic from your workflows uses the same network as other AWS services in the us-east-1 region. **This means your workflows share AWS's `us-east-1` network, sending requests from standard AWS IP ranges.**
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same point as above, let's focus on:

  1. What the native database support in Pipedream offers
  2. The bare minimum you need to know to get started connecting to a database
  3. Then go into implementation details that you might get tripped up on (security)
Suggested change
However, outbound traffic from your workflows uses the same network as other AWS services in the us-east-1 region. **This means your workflows share AWS's `us-east-1` network, sending requests from standard AWS IP ranges.**


## How to Connect to Restricted Databases
dannyroosevelt marked this conversation as resolved.
Show resolved Hide resolved
When you need to connect to a database that restricts access to a specific set of IP addresses, you have a couple options:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider adding an intro paragraph about database security for the unaware, then go into the specific options provided by Pipedream.

Suggested change
When you need to connect to a database that restricts access to a specific set of IP addresses, you have a couple options:
It's best practice to restrict which IP addresses can access your databases. This helps prevent unauthorized access by reducing the possible IP addresses that can make a connection.
Typically this is enforced by a firewall, or a configuration within the database to only accept connections from trusted IP addresses.
Pipedream offers two options to facilitate secure connections:


### Create a Virtual Private Cloud (VPC)
- The most secure, recommended approach, since it gives you a dedicated static IP only for workflows within your workspace
- Learn more about VPCs [here](/workflows/vpcs/)

### Use Pipedream's SQL Proxy
- The Pipedream SQL Proxy routes network requests through a static IP block
- When you create a connected account with any of [the apps](#supported-databases) that are currently supported by the SQL Proxy, steps using that account will route through the SQL Proxy, using the static IPs listed below.

#### Supported Databases
The SQL Proxy currently supports [MySQL](https://pipedream.com/apps/mysql), [PostgreSQL](https://pipedream.com/apps/postgresql), and [Snowflake](https://pipedream.com/apps/snowflake). Please let us know if you'd like to see support for other database types!

#### Static IPs
For database allow-listing, add the following IP block:
```
44.223.89.56/29
```

## Frequently Asked Questions

#### What's the difference between the SQL Proxy and a VPC?
- While both the SQL Proxy and a VPC enable secure database connections, a VPC offers enhanced isolation and security by providing a dedicated static IP that for workflows within your workspace.
dannyroosevelt marked this conversation as resolved.
Show resolved Hide resolved
- When enabled on a workflow, all requests from that workflow go through the VPC. The SQL Proxy, on the other hand, routes requests for the relevant database connection through a shared static IP block.
dannyroosevelt marked this conversation as resolved.
Show resolved Hide resolved
2 changes: 1 addition & 1 deletion docs-v2/pages/workflows/_meta.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@
"errors": "Handling errors",
"concurrency-and-throttling": "Concurrency and Throttling",
"settings": "Settings",
"vpc": "Virtual Private Clouds",
"vpc": "Virtual Private Clouds (VPCs)",
"domains": "Custom Domains",
"sharing": "Sharing workflows",
"build-and-run": {
Expand Down
6 changes: 1 addition & 5 deletions docs-v2/pages/workflows/vpc.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,13 +1,9 @@
import VideoPlayer from '@/components/VideoPlayer';

# Virtual Private Clouds
# Virtual Private Clouds (VPCs)

<VideoPlayer src="https://www.youtube.com/embed/E_dfTCCccPE" title="Virtual Private Clouds" />

Every Pipedream workflow is deployed to its own virtual machine in AWS. This means your workflow's execution environment has its own RAM and disk, isolated from other users’ workflows.

However, outbound traffic shares the same network as other AWS services deployed in the `us-east-1` region. That means network requests from your workflows (e.g. an HTTP request or a connection to a database) originate from the standard range of AWS IP addresses.

Pipedream VPCs enable you to run workflows in dedicated and isolated networks with static outbound egress IP addresses that are unique to your workspace (unlike other platforms that provide static IPs common to all customers on the platform).

Outbound network requests from workflows that run in a VPC will originate from these static IP addresses, so you can whitelist access to sensitive resources (like databases and APIs) with confidence that the requests will only originate from the Pipedream workflows in your workspace.
Expand Down
Loading