Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions modules/ROOT/nav.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
** xref:get-started:cloud-overview.adoc[Introduction to Redpanda Cloud]
*** xref:get-started:intro-to-events.adoc[]
*** xref:get-started:architecture.adoc[]
** xref:get-started:byoc-arch.adoc[]

* xref:get-started:cluster-types/index.adoc[Deploy]
** xref:get-started:cluster-types/serverless.adoc[Serverless]
Expand Down
38 changes: 38 additions & 0 deletions modules/get-started/pages/byoc-arch.adoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
= BYOC Architecture
:description: Learn about the control plane - data plane architecture in BYOC.

For high availability, Redpanda Cloud uses the following control plane - data plane architecture:

image::shared:control_d_plane.png[Control plane and data plane]

* *Control plane*: This is where most cluster management, operations, and maintenance takes place. The control plane enforces rules in the data plane. You can use role-based access control xref:security:authorization/rbac/rbac.adoc[(RBAC) in the control plane] to manage access to organization-level resources like clusters, resource groups, and networks.

* *Data plane*: This is where your cluster lives. The term _data plane_ is sometimes used interchangeably with _cluster_. The data plane is where you manage topics, consumer groups, connectors, and schemas. You can use xref:security:authorization/rbac/rbac_dp.adoc[RBAC in the data plane] to configure cluster-level permissions for provisioned users at scale.

IAM permissions allow the agent to access the cloud provider API to create and manage cluster resources. The permissions follow the principle of least privilege, limiting access to only what is necessary.

Clusters are configured and maintained in the control plane, but they remain available even if the network connection to the control plane is lost.

TIP: In the Redpanda Cloud UI, you see a different side navigation in the control plane and the data plane. You're in the control plane when you first log in. You're at the organization (org) level, and you haven't yet selected a cluster. This is where you can select, create, and delete clusters, networks, and resource groups. You're in the data plane when you're at the cluster level working with topics, consumer groups, connectors, and schemas.

== BYOC setup

In a BYOC architecture, you deploy the data plane in your own VPC. All network connections into the data plane take place through either a public endpoint, or for private clusters, through Redpanda Cloud network connections such as VPC peering, AWS PrivateLink, Azure Private Link, or GCP Private Service Connect. Customer data never leaves the data plane.

A BYOC cluster is initially set up from the control plane. This is a two-step process performed by `rpk cloud byoc apply`:

. You bootstrap a virtual machine (VM) in your VPC.
+
This VM spins up the agent and the required infrastructure. Redpanda assigns the necessary IAM policies required to run the agent and configures workload identity. That is, it configures independent IAM roles for each workload, with only the permissions each workload requires.

. The agent communicates with the control plane to pull the cluster specifications.
+
After the agent is up and running, it connects to the control plane and starts dequeuing and applying cluster specifications that provision, configure, and maintain clusters. The agent is in constant communication with the control plane,
receiving and applying cluster specifications and exchanging cluster metadata. Agents are authenticated and authorized through opaque and ephemeral tokens, and
they have dedicated job queues in the control plane. Agents also manage VPC peering networks.
+
image::shared:byoc_apply.png[cloud_byoc_apply]

NOTE: To create a Redpanda cluster in your virtual private cloud (VPC), follow the instructions in the Redpanda Cloud UI. The UI contains the parameters necessary to successfully run `rpk cloud byoc apply` with your cloud provider.

include::get-started:partial$no-access.adoc[]
50 changes: 8 additions & 42 deletions modules/get-started/pages/cloud-overview.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@

Redpanda Cloud is a complete data streaming platform delivered as a fully-managed service with automated upgrades and patching, data and partition balancing, and 24x7 support. It continuously monitors and maintains your clusters along with the underlying infrastructure to meet strict performance, availability, reliability, and security requirements. All Redpanda Cloud clusters are deployed with an integrated glossterm:Redpanda Console[].

TIP: For more detailed information about the Redpanda platform, see xref:get-started:intro-to-events.adoc[] and xref:get-started:architecture.adoc[].
TIP: For more detailed information about the Redpanda platform, see xref:get-started:intro-to-events.adoc[], xref:get-started:architecture.adoc[], and xref:get-started:byoc-arch.adoc[].

== Redpanda Cloud cluster types

Expand Down Expand Up @@ -45,7 +45,7 @@ include::get-started:partial$get-started-serverless.adoc[]
=== Bring Your Own Cloud (BYOC)

With BYOC clusters, you deploy Redpanda in your own cloud (AWS, Azure, or GCP), and all data is
contained in your own environment. This provides an additional layer of security and isolation. (See <<BYOC architecture>>.) When you create a BYOC cluster, you select the supported xref:reference:tiers/byoc-tiers.adoc[tier] that meets your compute and storage needs. Redpanda handles provisioning, operations, and maintenance.
contained in your own environment. This provides an additional layer of security and isolation. (See xref:get-started:byoc-arch.adoc[].) When you create a BYOC cluster, you select the supported xref:reference:tiers/byoc-tiers.adoc[tier] that meets your compute and storage needs. Redpanda handles provisioning, operations, and maintenance.

NOTE: With standard BYOC clusters, Redpanda manages security policies and resources for your VPC or VNet, including subnetworks, IAM roles, and storage buckets/accounts. A Bring Your Own Virtual Private Cloud (BYOVPC) cluster allows you to deploy the Redpanda glossterm:data plane[] into your existing VPC/VNet and take full control of managing the networking lifecycle. Compared to standard BYOC, BYOVPC provides more security, but the configuration is more complex. See <<Shared responsibility model>>.

Expand Down Expand Up @@ -98,6 +98,12 @@ Consider BYOC or Dedicated if you need more control over the deployment or if yo
* Kafka Connect
* Higher limits and quotas. See xref:reference:tiers/byoc-tiers.adoc[BYOC usage tiers] and xref:reference:tiers/dedicated-tiers.adoc[Dedicated usage tiers] compared to xref:get-started:cluster-types/serverless.adoc#serverless-usage-limits[Serverless limits].

== Redpanda Cloud architecture

When you sign up for a Redpanda account, Redpanda creates an organization for you. Your organization contains all your Redpanda resources, including your clusters and networks. Within your organization, Redpanda creates a default resource group to contain your resources. You can rename this resource group, and you can create more resource groups. For example, you may want different resource groups for production and testing.

For details about the control plane - data plane framework in BYOC, see xref:get-started:byoc-arch.adoc[BYOC architecture].

== Shared responsibility model

The Redpanda Cloud shared responsibility model lists the security ownership areas for Redpanda and customers. Responsibilities depend on the type of deployment.
Expand Down Expand Up @@ -211,46 +217,6 @@ xref:develop:connect/about.adoc[Redpanda Connect] is integrated into Redpanda Cl

xref:develop:managed-connectors/index.adoc[Kafka Connect] is automatically enabled on AWS and GCP clusters. With this, there is a node running for Kafka Connect even if connectors are not used. To enable Kafka Connect on Azure clusters, see xref:get-started:cluster-types/byoc/azure/create-byoc-cluster-azure.adoc#enable-kafka-connect[Enable Kafka Connect].

== Redpanda Cloud architecture

When you sign up for a Redpanda account, Redpanda creates an organization for you. Your organization contains all your Redpanda resources, including your clusters and networks. Within your organization, Redpanda creates a default resource group to contain your resources. You can rename this resource group, and you can create more resource groups. For example, you may want different resource groups for production and testing.

For high availability, Redpanda Cloud uses the following control plane - data plane architecture:

image::shared:control_d_plane.png[Control plane and data plane]

* *Control plane*: This is where most cluster management, operations, and maintenance takes place. The control plane enforces rules in the data plane. You can use role-based access control xref:security:authorization/rbac/rbac.adoc[(RBAC) in the control plane] to manage access to organization-level resources like clusters, resource groups, and networks.

* *Data plane*: This is where your cluster lives. The term _data plane_ is sometimes used interchangeably with _cluster_. The data plane is where you manage topics, consumer groups, connectors, and schemas. You can use xref:security:authorization/rbac/rbac_dp.adoc[RBAC in the data plane] to configure cluster-level permissions for provisioned users at scale.

* *Agent*: For BYOC, Redpanda uses an agent to manage the data plane from the control plane. The agent pulls cluster specifications from the control plane. IAM permissions allow the agent to access the the cloud provider API to create and manage cluster resources. The permissions follow the principle of least privilege, limiting access to only what is necessary.

Clusters are configured and maintained in the control plane, but they remain available even if the network connection to the control plane is lost.

TIP: In the Redpanda Cloud UI, you see a different side navigation in the control plane and the data plane. You're in the control plane when you first log in. You're at the organization (org) level, and you haven't yet selected a cluster. This is where you can select, create, and delete clusters, networks, and resource groups. You're in the data plane when you're at the cluster level working with topics, consumer groups, connectors, and schemas.

=== BYOC architecture

In a BYOC architecture, you deploy the data plane in your own VPC. All network connections into the data plane take place through either a public endpoint, or for private clusters, through Redpanda Cloud network connections such as VPC peering, AWS PrivateLink, Azure Private Link, or GCP Private Service Connect. Customer data never leaves the data plane.

include::get-started:partial$no-access.adoc[]

A BYOC cluster is initially set up from the control plane. This is a two-step process performed by `rpk cloud byoc apply`:

. You bootstrap a virtual machine (VM) in your VPC.
+
This VM spins up the agent and the required infrastructure. Redpanda assigns the necessary IAM policies required to run the agent and configures workload identity. That is, it configures independent IAM roles for each workload, with only the permissions each workload requires.

. The agent communicates with the control plane to pull the cluster specifications.
+
After the agent is up and running, it connects to the control plane and starts dequeuing and applying cluster specifications that provision, configure, and maintain clusters. The agent is in constant communication with the control plane,
receiving and applying cluster specifications and exchanging cluster metadata. Agents are authenticated and authorized through opaque and ephemeral tokens, and
they have dedicated job queues in the control plane. Agents also manage VPC peering networks.
+
image::shared:byoc_apply.png[cloud_byoc_apply]

NOTE: To create a Redpanda cluster in your virtual private cloud (VPC), follow the instructions in the Redpanda Cloud UI. The UI contains the parameters necessary to successfully run `rpk cloud byoc apply` with your cloud provider.

== Redpanda Cloud vs Self-Managed feature compatibility

Because Redpanda Cloud is a fully-managed service that provides maintenance, data and partition balancing, upgrades, and recovery, much of the cluster maintenance required for Self-Managed users is not necessary for Redpanda Cloud users. Also, Redpanda Cloud is opinionated about Kafka configurations. For example, automatic topic creation is disabled. Some systems expect the Kafka service to automatically create topics when a message is produced to a topic that doesn't exist. (You can create topics in Redpanda Cloud on the *Topics* page or with `rpk topic create`.)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
:description: Use the Redpanda Cloud UI to create a BYOC cluster on AWS.
:page-aliases: get-started:cluster-types/byoc/create-byoc-cluster-aws.adoc, cloud:create-byoc-cluster-aws.adoc, deploy:deployment-option/cloud/create-byoc-cluster-aws.adoc

To create a Redpanda cluster in your virtual private cloud (VPC), follow the instructions in the Redpanda Cloud UI. The UI contains the parameters necessary to successfully run `rpk cloud byoc apply`. See also: xref:get-started:cloud-overview.adoc#redpanda-cloud-architecture[Redpanda Cloud architecture].
To create a Redpanda cluster in your virtual private cloud (VPC), follow the instructions in the Redpanda Cloud UI. The UI contains the parameters necessary to successfully run `rpk cloud byoc apply`. See also: xref:get-started:byoc-arch.adoc[BYOC architecture].

NOTE: With standard BYOC clusters, Redpanda manages security policies and resources for your VPC, including subnetworks, service accounts, IAM roles, firewall rules, and storage buckets. For the highest level of security, you can manage these resources yourself with a xref:./vpc-byo-aws.adoc[BYOVPC cluster on AWS].

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,9 @@

include::shared:partial$feature-flag-rpcn.adoc[]

This topic explains how to create a Bring Your Own Virtual Private Cloud (BYOVPC) cluster. This setup allows you to deploy the Redpanda glossterm:data plane[] into your existing VPC and take full control of managing the networking lifecycle. Compared to a standard Bring Your Own Cloud (BYOC) setup, where Redpanda manages the networking lifecycle for you, this option provides more security.
This topic explains how to create a Bring Your Own Virtual Private Cloud (BYOVPC) cluster. This setup allows you to deploy the Redpanda glossterm:data plane[] into your existing VPC and take full control of managing the networking lifecycle. See also: xref:get-started:byoc-arch.adoc[BYOC architecture].

Compared to a standard Bring Your Own Cloud (BYOC) setup, where Redpanda manages the networking lifecycle for you, BYOVPC provides more security.

When you create a BYOCVPC cluster, you specify your VPC and service account. The Redpanda Cloud agent doesn't create any new resources or alter any settings in your account. With BYOVPC:

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,8 @@

include::shared:partial$feature-flag.adoc[]

To create a Redpanda cluster in your virtual network (VNet), follow the instructions in the Redpanda Cloud UI. The UI contains the parameters necessary to successfully run `rpk cloud byoc apply`. See also: xref:get-started:byoc-arch.adoc[BYOC architecture].
Comment thread
micheleRP marked this conversation as resolved.

NOTE: With standard BYOC clusters, Redpanda manages security policies and resources for your virtual network (VNet), including subnetworks, managed identities, IAM roles, security groups, and storage accounts. For the most security, you can manage these resources yourself with a xref:get-started:cluster-types/byoc/azure/vnet-azure.adoc[BYOVPC cluster on Azure].

== Prerequisites
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,9 @@

include::shared:partial$feature-flag-rpcn.adoc[]

This topic explains how to create a Bring Your Own Virtual Private Cloud (BYOVPC) cluster. This setup allows you to deploy the Redpanda glossterm:data plane[] into your existing virtual network (VNet) and take full control of managing the networking lifecycle. Compared to a standard Bring Your Own Cloud (BYOC) setup, where Redpanda manages the networking lifecycle for you, this option provides more security.
This topic explains how to create a Bring Your Own Virtual Private Cloud (BYOVPC) cluster. This setup allows you to deploy the Redpanda glossterm:data plane[] into your existing virtual network (VNet) and take full control of managing the networking lifecycle. See also: xref:get-started:byoc-arch.adoc[BYOC architecture].
Comment thread
micheleRP marked this conversation as resolved.

Compared to a standard Bring Your Own Cloud (BYOC) setup, where Redpanda manages the networking lifecycle for you, BYOVPC provides more security.

When you create a BYOVPC cluster, you specify your VNet and managed identities. The Redpanda Cloud agent doesn't create any new resources or alter any settings in your account. With a customer-managed VNet:

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
:description: Use the Redpanda Cloud UI to create a BYOC cluster on GCP.
:page-aliases: get-started:cluster-types/byoc/create-byoc-cluster-gcp.adoc, cloud:create-byoc-cluster-gcp.adoc, deploy:deployment-option/cloud/create-byoc-cluster-gcp

To create a Redpanda cluster in your virtual private cloud (VPC), follow the instructions in the Redpanda Cloud UI. The UI contains the parameters necessary to successfully run `rpk cloud byoc apply`.
To create a Redpanda cluster in your virtual private cloud (VPC), follow the instructions in the Redpanda Cloud UI. The UI contains the parameters necessary to successfully run `rpk cloud byoc apply`. See also: xref:get-started:byoc-arch.adoc[BYOC architecture].

NOTE: With standard BYOC clusters, Redpanda manages security policies and resources for your VPC, including subnetworks, service accounts, IAM roles, firewall rules, and storage buckets. For the highest level of security, you can manage these resources yourself with a xref:get-started:cluster-types/byoc/gcp/vpc-byo-gcp.adoc[BYOVPC cluster on GCP].

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,9 @@

include::shared:partial$feature-flag-rpcn.adoc[]

This topic explains how to create a Bring Your Own Virtual Private Cloud (BYOVPC) cluster. This setup allows you to deploy the Redpanda glossterm:data plane[] into your existing VPC and take full control of managing the networking lifecycle. Compared to a standard Bring Your Own Cloud (BYOC) setup, where Redpanda manages the networking lifecycle for you, this option provides more security.
This topic explains how to create a Bring Your Own Virtual Private Cloud (BYOVPC) cluster. This setup allows you to deploy the Redpanda glossterm:data plane[] into your existing VPC and take full control of managing the networking lifecycle. See also: xref:get-started:byoc-arch.adoc[BYOC architecture].
Comment thread
micheleRP marked this conversation as resolved.

Compared to a standard Bring Your Own Cloud (BYOC) setup, where Redpanda manages the networking lifecycle for you, BYOVPC provides more security.

When you create a BYOVPC cluster, you specify your VPC and service account. The Redpanda Cloud agent doesn't create any new resources or alter any settings in your account. With BYOVPC:

Expand Down