Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ Yes! ClickPipes for Postgres offers two ways to connect to databases in private
- us-east-1
- us-east-2
- eu-central-1
- For detailed setup instructions, see our [PrivateLink documentation](https://clickhouse.com/docs/knowledgebase/aws-privatelink-setup-for-clickpipes#requirements)
- For detailed setup instructions, see our [PrivateLink documentation](/docs/knowledgebase/aws-privatelink-setup-for-clickpipes#requirements)
- For regions where PrivateLink is not available, please use SSH tunneling

### How do you handle UPDATEs and DELETEs?
Expand Down
40 changes: 28 additions & 12 deletions knowledgebase/aws-privatelink-setup-for-clickpipes.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -15,43 +15,59 @@ Setup steps to expose a private RDS via AWS PrivateLink to ClickPipes.

## Requirements

The VPC must be located in one of our ClickPipes regions: us-east-1, us-east-2 or eu-central-1.
(https://clickhouse.com/docs/en/integrations/clickpipes#list-of-static-ips)
The VPC must be located in one of our ClickPipes regions: us-east-1, us-east-2 or eu-central-1
(/docs/en/integrations/clickpipes#list-of-static-ips), additionally the ClickHouse Instance must
be in the same region.

## Private link creation

Follow these steps to create a **VPC endpoint service** for your RDS instance. Repeat these steps if you have multiple instances that require endpoint services:
Follow these steps to create a **VPC endpoint service** for your RDS instance. Repeat these steps if you have multiple
RDS instances that require endpoint services (OR you may have different listener ports for different instances):

1. Locate Your VPC and [Create an NLB](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/create-network-load-balancer.html)
- Navigate to your target VPC and create a Network Load Balancer (NLB).
- Navigate to your target VPC and create a Network Load Balancer (NLB). Note that the NLB should be internal (private) and not internet-facing (public).

2. Configure the Target Group
- The target group should point to the RDS instance's endpoint IP and Port (typically 5432 for PostgreSQL or 3306 for MySQL).
- Ensure that the TCP protocol is used to avoid TLS termination by the NLB.

:::note

If you would like to automate the process of updating the target group with the new RDS endpoint IP, you can use AWS Lambda functions or other automation tools.
One of the terraform modules that can be used for this purpose is [this](https://github.com/MaterializeInc/terraform-aws-rds-privatelink).

:::
- **IMPORTANT**: Make sure the RDS instance endpoint used in case of DB Cluster/Aurora is ONLY the WRITER Endpoint and NOT the common endpoint.
- Ensure that the TCP protocol is used to avoid TLS termination by the NLB.

3. Set the Listener Port
- The listener port of the load balancer must match the port used by the target group (typically 5432 for PostgreSQL or 3306 for MySQL).

4. Ensure the Load Balancer is Private
- Configure the NLB to be private, ensuring it is only accessible within the VPC.

5. Create the VPC Endpoint Service
4. Create the VPC Endpoint Service
- In the VPC, create an endpoint service that points to the NLB.
- Enable acceptance of connection requests from specific accounts.

6. Authorize ClickPipes to Use the Endpoint Service
5. Authorize ClickPipes to Use the Endpoint Service
- Grant permission to the ClickPipes account to request this endpoint service.
- Configure allowed principals by adding the following principal ID:
```
arn:aws:iam::072088201116:root
```

6. Disable "Enforce Security Group Inbound Rules on Private Link Traffic" on the NLB (if a security group is attached to the NLB)
- Navigate to the NLB's settings and disable the "Enforce Security Group Inbound Rules on Private Link Traffic" setting if a security group is attached to the NLB.
- If using Terraform, set the `enforce_security_group_inbound_rules_on_private_link_traffic` attribute to `false` for the NLB
- This setting is **required** to allow traffic from the ClickPipes VPC to the NLB.

## Initiating connection

When it's done, share details such as private DNS name, VPC service name and availability zone.
When it's done, share details such as AWS region, VPC service name and availability zone.
ClickPipes team will initiate VPC endpoints creation in ClickPipes VPC. This will require [connection request acceptance](https://docs.aws.amazon.com/vpc/latest/privatelink/configure-endpoint-service.html#accept-reject-connection-requests) on your side.

## Creating ClickPipes

Use your RDS's private DNS endpoints to create your ClickPipes.
ClickPipes team will provide the DNS name to be used in the ClickPipe creation process. You can now [create your ClickPipe](/docs/en/integrations/clickpipes/postgres).

## Dynamically updating the RDS endpoint IP

When the RDS endpoint IP changes (in case of restarts/failovers/updates), you need to update the NLB target group with
the new IP. You can automate this process using AWS Lambda functions or other automation tools.