Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
64 changes: 26 additions & 38 deletions modules/develop/pages/connect/connect-quickstart.adoc
Original file line number Diff line number Diff line change
@@ -1,9 +1,15 @@
= Redpanda Connect Quickstart
:description: Learn how to quickly start building data pipelines with Redpanda Connect.

The *Connect* page provides a wizard to create pipelines for streaming data into and out of Redpanda. The wizard populates the YAML configuration automatically, so you can get started quickly.
The *Connect* page lets you build data pipelines using a YAML editor. Add connectors and processors from the left pane and configure them in the editor. You can use slash commands (`/`) in the YAML editor to insert existing variables, secrets, topics, and users, or to create new ones.

TIP: Advanced users can skip directly to the *Edit pipeline* step in the wizard to configure the YAML file themselves.
== Prerequisites

You must have a Redpanda Cloud account with a Serverless, Dedicated, or standard BYOC cluster. If you don't already have an account, https://redpanda.com/try-redpanda/cloud-trial[sign up for a free trial^].

NOTE: Serverless clusters support up to 10 pipelines.

== Quickstart pipelines

This quickstart shows how Redpanda Connect can generate, transform, and handle streaming data end-to-end. It creates the following pipelines:

Expand Down Expand Up @@ -61,29 +67,17 @@ The *consumer pipeline* uses the following Redpanda Connect components:
|Catches errors
|===

== Prerequisites

You must have a Redpanda Cloud account with a Serverless, Dedicated, or standard BYOC cluster. If you don't already have an account, https://redpanda.com/try-redpanda/cloud-trial[sign up for a free trial^].

NOTE: Serverless clusters support up to 10 pipelines.

== Build a producer pipeline

Follow these steps to create the producer pipeline:

. Go to the **Connect** page for your cluster to create a pipeline.

. **Add an input**: Search for and select `generate` from the list of connectors. Click **Next**.
To create the producer pipeline:

. **Add an output**: Search for and select `redpanda` from the list of connectors. Click **Next**.
. Go to the *Connect* page for your cluster and click *Create pipeline*.

. **Add a topic**: Create a new topic called `dad-jokes`. This is where Redpanda will store the generated messages. Click **Next**.
. Enter this name for the pipeline: `joke-generator-producer`.

. **Add permissions**: Create a new user called `connect`. Leave the rest of the default settings. The user will be created with the necessary permissions. Click **Next**.
. In the left pane, click to add the `generate` input connector and the `redpanda` output connector. The YAML for these connectors appears in the editor.

. **Edit pipeline**:
.. Enter this name for the pipeline: `joke-generator-producer`.
.. The **Configuration** section automatically populates the YAML with your selected components. Under *Connectors*, you can add the processors `log` and `catch` to log generated jokes and monitor for errors. However, for simplicity in this quickstart, replace the entire configuration with the following YAML, which includes the processors and the `mapping` for joke generation:
. Replace the generated YAML with the following configuration, which includes the `log` and `catch` processors and the `mapping` for joke generation:
+
[source,yaml]
----
Expand Down Expand Up @@ -151,15 +145,18 @@ output:

----

.. Click **Create**.
. Click *Create*.

+
[NOTE]
====
* Notice the `$\{REDPANDA_BROKERS}` xref:develop:connect/configuration/contextual-variables.adoc[contextual variable] in the configuration. This automatically references your cluster's bootstrap server address, so you can use it in any pipeline without hardcoding connection details.
* Notice `${secrets.KAFKA_USER_CONNECT}` and `${secrets.KAFKA_PASSWORD_CONNECT}`. These were generated when you created the `connect` user.
* Notice the `$\{REDPANDA_BROKERS}` xref:develop:connect/configuration/contextual-variables.adoc[contextual variable] in the configuration. This automatically references your cluster's bootstrap server address, so you can use it in any pipeline without hardcoding connection details.
* Notice `${secrets.KAFKA_USER_CONNECT}` and `${secrets.KAFKA_PASSWORD_CONNECT}`. These reference secrets that you can create using the slash command menu in the YAML editor or the *Security* page.
* The Brave browser does not fully support code snippets.

+
TIP: In the YAML editor, type `/` to open the slash command menu. Use it to insert variables, secrets, topics, and users. You can select existing items or create new ones.

====

. Your pipeline details display, and after a few seconds, the status changes from **Starting** to **Running**. If you don't see this change, refresh the page. Once running, your pipeline generates a new joke every five seconds and writes the joke to your topic.
Expand Down Expand Up @@ -206,24 +203,15 @@ After a minute, select the pipeline and click **Stop** so you can examine the re

== Build a consumer pipeline

This pipeline rates the jokes that you generated in the first pipeline. Follow these steps to create the consumer pipeline:

. On the **Connect** page for your cluster, click **Create pipeline**.

. **Add an input**: Search for and select `redpanda` from the list of connectors. Click **Next**.
This pipeline rates the jokes that you generated in the first pipeline. To create the consumer pipeline:

. **Add an output**: Search for and select `drop` from the list of connectors. (Because this quickstart is just for testing, this output drops the message instead of sending it anywhere else. In a real scenario you'd replace the `drop` connector with your real destination.) Click **Next**.
. On the *Connect* page for your cluster, click *Create pipeline*.

. **Add a topic**: Select the existing topic called `dad-jokes`. Click **Next**.
. Enter this name for the pipeline: `joke-generator-consumer`.

. **Add permissions**:
.. Select the existing user called `connect`.
.. Add a consumer group: Enter `dad-joke-raters` as the name for the consumer group.
.. Click **Next**.
. In the left pane, click to add the `redpanda` input connector and the `drop` output connector. (Because this quickstart is just for testing, this output drops the message instead of sending it anywhere else. In a real scenario you'd replace the `drop` connector with your real destination.) Their YAML appears in the editor.

. **Edit pipeline**:
.. Enter this name for the pipeline: `joke-generator-consumer`.
.. The **Configuration** section automatically populates the YAML with your selected components. To add the `bloblang`, `log`, and `catch` processors, replace the entire configuration with the following YAML. Bloblang is Redpanda Connect's scripting language used to add logic (for example, random ratings).
. Replace the generated YAML with the following configuration, which includes the `bloblang`, `log`, and `catch` processors. Bloblang is Redpanda Connect's scripting language used to add logic (for example, random ratings).

+
[source,yaml]
Expand Down Expand Up @@ -317,7 +305,7 @@ output:
----


.. Click **Create** to start your pipeline.
. Click *Create* to start your pipeline.

+
NOTE: This example explicitly includes several optional configuration fields for the `redpanda` input. They're shown here for demonstration purposes, so you can see a range of available settings.
Expand Down Expand Up @@ -352,7 +340,7 @@ When you've finished experimenting with your data pipeline, you can delete the p

== Next steps

* Try one of our xref:cookbooks:index.adoc[Redpanda Connect cookbooks].
* Try one of the xref:cookbooks:index.adoc[Redpanda Connect cookbooks].
* Choose xref:develop:connect/components/about.adoc[connectors for your use case].
* Learn how to xref:develop:connect/configuration/secret-management.adoc[add secrets to your pipeline].
* Learn how to xref:develop:connect/configuration/monitor-connect.adoc[monitor a data pipeline on a BYOC or Dedicated cluster].
Expand Down
4 changes: 2 additions & 2 deletions modules/get-started/pages/whats-new-cloud.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -110,11 +110,11 @@ The message size setting on existing topics is not changed, but the message size

=== Redpanda Connect updates

Redpanda Connect provides a simplified xref:develop:connect/connect-quickstart.adoc[quickstart] experience in the UI that helps you to start building data pipelines. The quickstart wizard creates pipelines to stream data into and out of Redpanda, populating the required YAML configuration automatically.
Redpanda Connect provides a simplified xref:develop:connect/connect-quickstart.adoc[quickstart] experience in the UI that helps you to start building data pipelines. The quickstart creates pipelines to stream data into and out of Redpanda using the pipeline editor.

=== Get Started with Serverless

A Serverless cluster's *Overview* page now provides a *Get Started* wizard to help you start streaming your own data with a xref:develop:connect/connect-quickstart.adoc[Redpanda Connect] pipeline. The wizard lets you stream data into and out of Redpanda without writing producer/consumer code.
A Serverless cluster's *Overview* page now provides a *Get Started* guide to help you start streaming your own data with a xref:develop:connect/connect-quickstart.adoc[Redpanda Connect] pipeline. It lets you stream data into and out of Redpanda without writing producer/consumer code.

=== Remote read replicas: GA

Expand Down
2 changes: 1 addition & 1 deletion modules/get-started/partials/cloud-quickstart.adoc
Original file line number Diff line number Diff line change
@@ -1 +1 @@
TIP: The cluster's *Overview* page includes a *Get Started* wizard to quickly pipeline your data into and out of Redpanda. The wizard guides you to select an input data source (to produce data) or an output data sink (to consume data) and automatically populates the necessary permissions and configurations. See also: xref:develop:connect/connect-quickstart.adoc[Redpanda Connect Quickstart]
TIP: The cluster's *Overview* page includes a *Get Started* guide to help you start streaming data into and out of Redpanda. See also: xref:develop:connect/connect-quickstart.adoc[Redpanda Connect Quickstart]
2 changes: 1 addition & 1 deletion modules/get-started/partials/get-started-serverless.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ When either the credits expire or the days in the trial expire, the clusters mov

After you start a trial, Redpanda instantly prepares an account for you. Your account includes a `welcome` cluster with a `hello-world` demo topic you can explore. It includes sample data so you can see how real-time messaging works before sending your own data.

xref:get-started:cluster-types/serverless.adoc#interact-with-your-cluster[Get started] by following the wizard to create a Redpanda Connect glossterm:pipeline[], or by following the steps in the Console to use `rpk` to interact with your cluster from the command line:
xref:get-started:cluster-types/serverless.adoc#interact-with-your-cluster[Get started] by creating a Redpanda Connect glossterm:pipeline[], or by following the steps in the Console to use `rpk` to interact with your cluster from the command line:

. Log in with `rpk cloud login`.
. Consume from the `hello-world` topic with `rpk topic consume hello-world`.
Expand Down