Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions modules/ROOT/nav.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@
*** xref:get-started:cluster-types/byoc/gcp/index.adoc[GCP]
**** xref:get-started:cluster-types/byoc/gcp/create-byoc-cluster-gcp.adoc[]
**** xref:get-started:cluster-types/byoc/gcp/vpc-byo-gcp.adoc[]
**** xref:get-started:cluster-types/byoc/gcp/enable-rpcn-byovpc-gcp.adoc[Enable Redpanda Connect on a BYOVPC Cluster on GCP]
*** xref:get-started:cluster-types/byoc/remote-read-replicas.adoc[]
** xref:get-started:cluster-types/dedicated/index.adoc[Dedicated]
*** xref:get-started:cluster-types/dedicated/quick-start-cloud.adoc[Dedicated Cluster Quickstart]
Expand Down
2 changes: 1 addition & 1 deletion modules/develop/pages/connect/connect-quickstart.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ include::develop:partial$availability-message.adoc[]

== Prerequisites

A Redpanda Cloud account for Serverless, Dedicated, or standard BYOC (not BYOVPC). If you don't already have an account, https://redpanda.com/try-redpanda/cloud-trial[sign up for a free trial^].
A Redpanda Cloud account for Serverless, Dedicated, or standard BYOC. If you don't already have an account, https://redpanda.com/try-redpanda/cloud-trial[sign up for a free trial^].

== Before you start

Expand Down
6 changes: 2 additions & 4 deletions modules/develop/partials/availability-message.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,6 @@
====

* Redpanda Connect is available in limited availability (LA) for BYOC and Dedicated clusters. Features in LA are production-ready and are covered by Redpanda Support for early adopters. To unlock Redpanda Connect for your account, contact https://support.redpanda.com/hc/en-us/requests/new[Redpanda Support^].
* Redpanda Connect is not available for BYOVPC clusters.
* Redpanda Connect is in beta for BYOVPC clusters on GCP but not on AWS or Azure.
* Redpanda Connect is available in beta for Serverless clusters. Features in beta are not covered by Redpanda Support and should not be used in production environments.
====


====
4 changes: 2 additions & 2 deletions modules/get-started/pages/cloud-overview.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -290,7 +290,7 @@ Features in limited availability are production-ready and are covered by Redpand

The following features are currently in limited availability in Redpanda Cloud:

* Redpanda Connect for Dedicated and BYOC (not BYOVPC)
* Redpanda Connect for Dedicated and BYOC (not BYOVPC on AWS or Azure)
* Serverless
* Dedicated and BYOC for Azure
* BYOVPC for GCP
Expand All @@ -302,7 +302,7 @@ Features in beta are available for testing and feedback. They are not covered by

The following features are currently in beta in Redpanda Cloud:

* Redpanda Connect for Serverless
* Redpanda Connect for BYOVPC on GCP and Serverless
* Redpanda Terraform provider
* BYOVPC for AWS and Azure
* Remote Read Replicas for AWS and GCP
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,124 @@
= Enable Redpanda Connect on an Existing BYOVPC Cluster on GCP
:description: Add Redpanda Connect to your existing BYOVPC cluster.
:page-beta: true

include::shared:partial$feature-flag-rpcn.adoc[]

To enable Redpanda Connect on an existing BYOVPC cluster, you must update your configuration. You can also create xref:get-started:cluster-types/byoc/gcp/vpc-byo-gcp.adoc[a new BYOVPC cluster] with Redpanda Connect already enabled.

Replace all `<placeholders>` with your own values.

. Create two new service accounts with the necessary permissions and roles.
+
.Show commands
[%collapsible]
====
```bash
# Account used to check for and read secrets, which are required to create Redpanda Connect pipelines.

gcloud iam service-accounts create redpanda-connect-api \
--display-name="Redpanda Connect API Service Account"

cat << EOT > redpanda-connect-api.role
{
"name": "redpanda_connect_api_role",
"title": "Redpanda Connect API Role",
"description": "Redpanda Connect API Role",
"includedPermissions": [
"resourcemanager.projects.get",
"secretmanager.secrets.get",
"secretmanager.versions.access"
],
}
EOT

gcloud iam roles create redpanda_connect_api_role --project=<service-project-id> --file redpanda-connect-api.role

gcloud projects add-iam-policy-binding <service-project-id> \
--member="serviceAccount:redpanda-connect-api@<service-project-id>.iam.gserviceaccount.com" \
--role="projects/<service-project-id>/roles/redpanda_connect_api_role"
```

```bash
# Account used to retrieve secrets and create Redpanda Connect pipelines.

gcloud iam service-accounts create redpanda-connect \
--display-name="Redpanda Connect Service Account"

cat << EOT > redpanda-connect.role
{
"name": "redpanda_connect_role",
"title": "Redpanda Connect Role",
"description": "Redpanda Connect Role",
"includedPermissions": [
"resourcemanager.projects.get",
"secretmanager.versions.access"
],
}
EOT

gcloud iam roles create redpanda_connect_role --project=<service-project-id> --file redpanda-connect.role

gcloud projects add-iam-policy-binding <service-project-id> \
--member="serviceAccount:redpanda-connect@<service-project-id>.iam.gserviceaccount.com" \
--role="projects/<service-project-id>/roles/redpanda_connect_role"
```
====

. Bind the service accounts.
+
The account ID of the GCP service account is used to configure service account bindings. This account ID is the local part of the email address for the GCP service account. For example, if the GCP service account is `my-gcp-sa@my-project.iam.gserviceaccount.com`, then the account ID is `my-gcp-sa`.
+
.Show commands
[%collapsible]
====
```
gcloud iam service-accounts add-iam-policy-binding <redpanda_connect_api-gcp-sa-account-id>@<service-project-id>.iam.gserviceaccount.com \
--role roles/iam.workloadIdentityUser \
--member "serviceAccount:<service-project-id>.svc.id.goog[redpanda-connect/<redpanda_connect_api-gcp-sa-account-id>]"
```
```
gcloud iam service-accounts add-iam-policy-binding <redpanda_connect-gcp-sa-account-id>@<service-project-id>.iam.gserviceaccount.com \
--role roles/iam.workloadIdentityUser \
--member "serviceAccount:<service-project-id>.svc.id.goog[redpanda-connect/<redpanda_connect-gcp-sa-account-id>]"
```
====

. Make a xref:api:ROOT:cloud-controlplane-api.adoc#patch-/v1/clusters/-cluster.id-[`PATCH /v1/clusters/\{cluster-id}`] request to update the cluster configuration.
+
.Show request
[%collapsible]
====
```bash
export CLUSTER_PATCH_BODY=`cat << EOF
{
"customer_managed_resources": {
"gcp": {
"redpanda_connect_api_service_account": {
"email": "<redpanda_connect-api-gcp-sa-account-id>@<service-project-id>.iam.gserviceaccount.com"
},
"redpanda_connect_service_account": {
"email": "<redpanda_connect-gcp-sa-account-id>@<service-project-id>.iam.gserviceaccount.com"
}
}
}
}
EOF`
curl -v -X PATCH \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $AUTH_TOKEN" \
-d "$CLUSTER_PATCH_BODY" $PUBLIC_API_ENDPOINT/v1/clusters/<cluster-id>
```
====

. Check Redpanda Connect is available in the Cloud UI.
.. Log in to https://cloud.redpanda.com[Redpanda Cloud^].
.. Go to the **Connect** page and you should see the **Redpanda Connect** tab alongside **Kafka Connect**.

== Next steps

* Choose xref:develop:connect/components/catalog.adoc[connectors for your use case].
* Learn how to xref:redpanda-connect:guides:getting_started.adoc[configure, test, and run a data pipeline locally].
* Try the xref:develop:connect/connect-quickstart.adoc[Redpanda Connect quickstart].
* Try one of our xref:cookbooks:index.adoc[Redpanda Connect cookbooks].
* Learn how to xref:develop:connect/configuration/secret-management.adoc[add secrets to your pipeline].
100 changes: 92 additions & 8 deletions modules/get-started/pages/cluster-types/byoc/gcp/vpc-byo-gcp.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@ gcloud compute routers create <router-name> \
--region <region> \
--network <shared-vpc-name>

gcloud compute addresses create <address-name>
gcloud compute addresses create <address-name> --region <region>

gcloud compute routers nats create <nat-config-name> \
--project <host-project-id> \
Expand Down Expand Up @@ -93,6 +93,8 @@ gcloud compute firewall-rules create gke-redpanda-cluster-webhooks \
--allow="tcp:9443,tcp:8443,tcp:6443"
```
+
Replace `<gke-master-cidr-range>` with a /28 CIDR. For example: 172.16.0.32/28. For information about the master CIDR, and how to set it using `--master-ipv4-cidr`, see the **gcloud** tab in https://cloud.google.com/kubernetes-engine/docs/how-to/legacy/network-isolation#private_cp[Creating a private cluster with no client access to the public endpoint]
+
. Grant permission to read the VPC and related resources.
+
If the host project and service project are in different projects, it's helpful for the Redpanda team to have read access
Expand Down Expand Up @@ -169,7 +171,8 @@ cat << EOT > redpanda-agent.role
"title": "Redpanda Agent Role",
"description": "A role comprising general permissions allowing the agent to manage Redpanda cluster resources.",
"includedPermissions": [
"compute.firewalls.get",
"compute.firewalls.get",
"compute.disks.get",
"compute.globalOperations.get",
"compute.instanceGroupManagers.get",
"compute.instanceGroupManagers.delete",
Expand Down Expand Up @@ -284,6 +287,62 @@ gcloud storage buckets add-iam-policy-binding gs://<tiered-storage-bucket-name>
```
====

* Redpanda Connect service accounts
+
.Show commands
[%collapsible]
====
```bash
# Account used to check for and read secrets, which are required to create Redpanda Connect pipelines.

gcloud iam service-accounts create redpanda-connect-api \
--display-name="Redpanda Connect API Service Account"

cat << EOT > redpanda-connect-api.role
{
"name": "redpanda_connect_api_role",
"title": "Redpanda Connect API Role",
"description": "Redpanda Connect API Role",
"includedPermissions": [
"resourcemanager.projects.get",
"secretmanager.secrets.get",
"secretmanager.versions.access"
],
}
EOT

gcloud iam roles create redpanda_connect_api_role --project=<service-project-id> --file redpanda-connect-api.role

gcloud projects add-iam-policy-binding <service-project-id> \
--member="serviceAccount:redpanda-connect-api@<service-project-id>.iam.gserviceaccount.com" \
--role="projects/<service-project-id>/roles/redpanda_connect_api_role"
```
```bash
# Account used to retrieve secrets and create Redpanda Connect pipelines.

gcloud iam service-accounts create redpanda-connect \
--display-name="Redpanda Connect Service Account"

cat << EOT > redpanda-connect.role
{
"name": "redpanda_connect_role",
"title": "Redpanda Connect Role",
"description": "Redpanda Connect Role",
"includedPermissions": [
"resourcemanager.projects.get",
"secretmanager.versions.access"
],
}
EOT

gcloud iam roles create redpanda_connect_role --project=<service-project-id> --file redpanda-connect.role

gcloud projects add-iam-policy-binding <service-project-id> \
--member="serviceAccount:redpanda-connect@<service-project-id>.iam.gserviceaccount.com" \
--role="projects/<service-project-id>/roles/redpanda_connect_role"
```
====

* Redpanda Console service account
+
.Show commands
Expand Down Expand Up @@ -318,11 +377,11 @@ gcloud iam roles create redpanda_console_secret_manager_role --project=<service-

gcloud projects add-iam-policy-binding <service-project-id> \
--member="serviceAccount:redpanda-console@<service-project-id>.iam.gserviceaccount.com" \
--role='projects/<service-project-id>/roles/redpanda_console_secret_manager_role'
--role="projects/<service-project-id>/roles/redpanda_console_secret_manager_role"
```
====

* Redpanda Connectors service account
* Kafka Connect connectors service account
+
.Show commands
[%collapsible]
Expand All @@ -347,7 +406,7 @@ gcloud iam roles create redpanda_connectors_role --project=<service-project-id>

gcloud projects add-iam-policy-binding <service-project-id> \
--member="serviceAccount:redpanda-connectors@<service-project-id>.iam.gserviceaccount.com" \
--role='projects/<service-project-id>/roles/redpanda_connectors_role'
--role="projects/<service-project-id>/roles/redpanda_connectors_role"
```
====

Expand Down Expand Up @@ -454,7 +513,7 @@ gcloud iam roles create redpanda_gke_utility_role --project=<service-project-id>

gcloud projects add-iam-policy-binding <service-project-id> \
--member="serviceAccount:redpanda-gke@<service-project-id>.iam.gserviceaccount.com" \
--role='projects/<service-project-id>/roles/redpanda_gke_utility_role'
--role="projects/<service-project-id>/roles/redpanda_gke_utility_role"
```
====

Expand Down Expand Up @@ -487,7 +546,24 @@ gcloud iam service-accounts add-iam-policy-binding <console-gcp-sa-account-id>@<
```
====

* Redpanda Connectors service account
* Redpanda Connect service accounts
+
.Show command
[%collapsible]
====
```
gcloud iam service-accounts add-iam-policy-binding <redpanda_connect_api-gcp-sa-account-id>@<service-project-id>.iam.gserviceaccount.com \
--role roles/iam.workloadIdentityUser \
--member "serviceAccount:<service-project-id>.svc.id.goog[redpanda-connect/<redpanda_connect_api-gcp-sa-account-id>]"
```
```
gcloud iam service-accounts add-iam-policy-binding <redpanda_connect-gcp-sa-account-id>@<service-project-id>.iam.gserviceaccount.com \
--role roles/iam.workloadIdentityUser \
--member "serviceAccount:<service-project-id>.svc.id.goog[redpanda-connect/<redpanda_connect-gcp-sa-account-id>]"
```
====

* Kafka Connect connectors service account
+
.Show command
[%collapsible]
Expand Down Expand Up @@ -530,7 +606,7 @@ gcloud iam service-accounts add-iam-policy-binding <gke-service-account-name>@<s

Log in to the https://cloud.redpanda.com[Redpanda Cloud UI^], and follow the steps to xref:get-started:cluster-types/byoc/gcp/create-byoc-cluster-gcp.adoc[create a BYOC cluster], with the following exceptions:

. On the *Network* page, select the *Customer-managed* connection type, and enter the network, service account, and storage bucket information you created.
. On the *Network* page, select the *Customer-managed* connection type, and enter the network, service account, storage bucket information, and GKE master CIDR range you created.

. With customer-managed networks, you must grant yourself (the user deploying the cluster with `rpk`) the following permissions:
+
Expand Down Expand Up @@ -591,6 +667,14 @@ rpk cloud byoc gcp apply --redpanda-id='<redpanda-id>' --project-id='<service-pr

include::get-started:partial$no-access.adoc[]

== Delete cluster

You can delete the cluster in the Cloud UI.

. Log in to https://cloud.redpanda.com[Redpanda Cloud^].
. Select your cluster.
. Go to the **Cluster settings** page and click **Delete**, then confirm your deletion.

== Next steps

xref:networking:byoc/gcp/index.adoc[Configure private networking]
2 changes: 1 addition & 1 deletion modules/shared/partials/feature-flag-rpcn.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -2,5 +2,5 @@
====

* BYOVPC is an add-on feature that may require an additional purchase. To unlock this feature for your account, contact your Redpanda account team or https://www.redpanda.com/price-estimator[Redpanda Sales^].
* Redpanda Connect is not available for BYOVPC clusters.
* Redpanda Connect is in beta for BYOVPC clusters on GCP.
====