diff --git a/docs/en/integrations/data-ingestion/clickpipes/postgres/faq.md b/docs/en/integrations/data-ingestion/clickpipes/postgres/faq.md index 0cb9f88975c..7e52f6b93c6 100644 --- a/docs/en/integrations/data-ingestion/clickpipes/postgres/faq.md +++ b/docs/en/integrations/data-ingestion/clickpipes/postgres/faq.md @@ -43,7 +43,7 @@ Yes! ClickPipes for Postgres offers two ways to connect to databases in private - us-east-1 - us-east-2 - eu-central-1 - - For detailed setup instructions, see our [PrivateLink documentation](https://clickhouse.com/docs/knowledgebase/aws-privatelink-setup-for-clickpipes#requirements) + - For detailed setup instructions, see our [PrivateLink documentation](/docs/knowledgebase/aws-privatelink-setup-for-clickpipes#requirements) - For regions where PrivateLink is not available, please use SSH tunneling ### How do you handle UPDATEs and DELETEs? diff --git a/knowledgebase/aws-privatelink-setup-for-clickpipes.mdx b/knowledgebase/aws-privatelink-setup-for-clickpipes.mdx index c09f45e8d81..68bdbe859ea 100644 --- a/knowledgebase/aws-privatelink-setup-for-clickpipes.mdx +++ b/knowledgebase/aws-privatelink-setup-for-clickpipes.mdx @@ -15,43 +15,59 @@ Setup steps to expose a private RDS via AWS PrivateLink to ClickPipes. ## Requirements -The VPC must be located in one of our ClickPipes regions: us-east-1, us-east-2 or eu-central-1. -(https://clickhouse.com/docs/en/integrations/clickpipes#list-of-static-ips) +The VPC must be located in one of our ClickPipes regions: us-east-1, us-east-2 or eu-central-1 +(/docs/en/integrations/clickpipes#list-of-static-ips), additionally the ClickHouse Instance must +be in the same region. ## Private link creation -Follow these steps to create a **VPC endpoint service** for your RDS instance. Repeat these steps if you have multiple instances that require endpoint services: +Follow these steps to create a **VPC endpoint service** for your RDS instance. Repeat these steps if you have multiple +RDS instances that require endpoint services (OR you may have different listener ports for different instances): 1. Locate Your VPC and [Create an NLB](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/create-network-load-balancer.html) - - Navigate to your target VPC and create a Network Load Balancer (NLB). + - Navigate to your target VPC and create a Network Load Balancer (NLB). Note that the NLB should be internal (private) and not internet-facing (public). 2. Configure the Target Group - The target group should point to the RDS instance's endpoint IP and Port (typically 5432 for PostgreSQL or 3306 for MySQL). - - Ensure that the TCP protocol is used to avoid TLS termination by the NLB. + + :::note + + If you would like to automate the process of updating the target group with the new RDS endpoint IP, you can use AWS Lambda functions or other automation tools. + One of the terraform modules that can be used for this purpose is [this](https://github.com/MaterializeInc/terraform-aws-rds-privatelink). + + ::: - **IMPORTANT**: Make sure the RDS instance endpoint used in case of DB Cluster/Aurora is ONLY the WRITER Endpoint and NOT the common endpoint. + - Ensure that the TCP protocol is used to avoid TLS termination by the NLB. 3. Set the Listener Port - The listener port of the load balancer must match the port used by the target group (typically 5432 for PostgreSQL or 3306 for MySQL). -4. Ensure the Load Balancer is Private - - Configure the NLB to be private, ensuring it is only accessible within the VPC. - -5. Create the VPC Endpoint Service +4. Create the VPC Endpoint Service - In the VPC, create an endpoint service that points to the NLB. - Enable acceptance of connection requests from specific accounts. -6. Authorize ClickPipes to Use the Endpoint Service +5. Authorize ClickPipes to Use the Endpoint Service - Grant permission to the ClickPipes account to request this endpoint service. - Configure allowed principals by adding the following principal ID: ``` arn:aws:iam::072088201116:root ``` +6. Disable "Enforce Security Group Inbound Rules on Private Link Traffic" on the NLB (if a security group is attached to the NLB) + - Navigate to the NLB's settings and disable the "Enforce Security Group Inbound Rules on Private Link Traffic" setting if a security group is attached to the NLB. + - If using Terraform, set the `enforce_security_group_inbound_rules_on_private_link_traffic` attribute to `false` for the NLB + - This setting is **required** to allow traffic from the ClickPipes VPC to the NLB. + ## Initiating connection -When it's done, share details such as private DNS name, VPC service name and availability zone. +When it's done, share details such as AWS region, VPC service name and availability zone. ClickPipes team will initiate VPC endpoints creation in ClickPipes VPC. This will require [connection request acceptance](https://docs.aws.amazon.com/vpc/latest/privatelink/configure-endpoint-service.html#accept-reject-connection-requests) on your side. ## Creating ClickPipes -Use your RDS's private DNS endpoints to create your ClickPipes. \ No newline at end of file +ClickPipes team will provide the DNS name to be used in the ClickPipe creation process. You can now [create your ClickPipe](/docs/en/integrations/clickpipes/postgres). + +## Dynamically updating the RDS endpoint IP + +When the RDS endpoint IP changes (in case of restarts/failovers/updates), you need to update the NLB target group with +the new IP. You can automate this process using AWS Lambda functions or other automation tools.