Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 30 additions & 0 deletions docs/create/create-topic.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
# Create a topic

Typically, when you deploy a configured service, topics will be created automatically for you, with default settings. However, you can create topics manually and configure the topic settings.

To create a topic:

1. On the left-hand main menu, click `Topics`.
2. In the top right of the screen click `+ New topic`. The `New topic` dialog is displayed:

![New topic dialog](../images/new-topic-dialog.png)

3. Give your topic a suitable name.
4. If you want to set some specific parameters for the topic, click `Advanced settings` to display the panel with [configurable parameters](#advanced-settings) for your broker.
5. Once you've configured your topic, click `Create`.

The system will then create the topic, which might take a few moments.

## Advanced settings

The parameters in the `Advanced settings` panel are as follows:

| Parameter | Description |
|---|---|
| Partitions | The basic unit of paralellism in a Kafka topic. The more partitions the more messages can be processed concurrently. You can think of partitions in Kafka topics as lanes on a motorway. Each partition acts as an independent lane where messages (like vehicles) flow in an ordered sequence.<br/><br/>Just like lanes on a motorway, partitions enable parallel processing of messages within a Kafka topic. Multiple consumers (or subscribers) can each read from their own partition concurrently, enabling efficient and scalable message processing.<br/><br/>Additionally, partitions provide fault tolerance similar to lanes providing redundancy on a motorway. If one lane (partition) becomes congested or experiences issues, the others can continue operating independently without affecting the overall flow of traffic (messages) on the motorway (topic). **Default = 1** |
| Retention time (hours) | Messages are retained in the topic for this amount of time, regardless of whether they have been consumed or not. After the specified time period elapses, Kafka will automatically delete messages older than the retention period. **Default = 24** |
| Retention size (MB) | Messages are retained in the topic until the topic reaches this size. Once the topic size exceeds this threshold, Kafka starts deleting the oldest messages to make space for new ones. **Default = 50** |

## To change the advanced settings

In the `Topics` view, hover over the topic you want to edit until the kebab menu (vertical dots) is displayed. Click the kebab menu, and then select `Edit topic`. Change your configuration as required. Click `Save` and your changes are applied to the topic.
4 changes: 4 additions & 0 deletions docs/create/overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,10 @@ The key terminology changes are shown in the following table:
| Branch | N/A | Represents a Git branch, such as main or dev |
| Application | Project | The files for the implementation of a source, transform, or destination |

## Topics

Typically, topics are automatically created for you when you need them. They are created with default settings. However, you can [manually create a topic](./create-topic.md), if you want to specify more advanced settings for the topic, such as the number of partitions to allocate to a topic.

## 🏃‍♀️ Next step

[Create your project :material-arrow-right-circle:{ align=right }](./create-project.md)
Binary file added docs/images/new-topic-dialog.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
1 change: 1 addition & 0 deletions mkdocs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,7 @@ nav:
- 'Protected environments': 'create/protected-environment.md'
- 'Syncing an environment': 'create/syncing-environment.md'
- 'Project structure': 'create/project-structure.md'
- 'Create a topic': 'create/create-topic.md'
- '2. Develop a pipeline':
- 'Overview': 'develop/overview.md'
- 'Create an application': 'develop/create-application.md'
Expand Down