Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -27,13 +27,9 @@ endif::[]
* Workspace admin privileges to complete the steps to create a Unity Catalog storage credential and external location that connects your cluster's Tiered Storage bucket to Databricks.

== Limitations

The following data types are not currently supported for managed Iceberg tables:
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@nvartolomei this looks correct me to based on recent slack convos but can you take a quick 👀 , please?


* Databricks Managed Iceberg tables do not currently support partition evolution. If you define a xref:manage:iceberg/about-iceberg-topics.adoc#use-custom-partitioning[custom partitioning] scheme for an Iceberg topic, you must not change it later.
+
The cluster property config_ref:iceberg_default_partition_spec,true,properties/cluster-properties[`iceberg_default_partition_spec`] defines the cluster-wide partitioning scheme, by default configured to `(hour(redpanda.timestamp))`. To use a different cluster-wide default, configure this property before creating any Iceberg-enabled topics. You must only override the default partitioning scheme on the topic level for new Iceberg topics that do not yet contain data.
* The following data types are not currently supported for managed Iceberg tables:
+
--
|===
| Iceberg type | Equivalent Avro type

Expand All @@ -42,8 +38,7 @@ The cluster property config_ref:iceberg_default_partition_spec,true,properties/c
| time | time-millis, time-micros

|===
--
+

There are no limitations for Protobuf types.

== Create a Unity Catalog storage credential
Expand Down
10 changes: 10 additions & 0 deletions modules/manage/partials/iceberg/use-iceberg-catalogs.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,11 @@ To connect to a REST catalog, set the following cluster configuration properties
* config_ref:iceberg_catalog_type,true,properties/cluster-properties[`iceberg_catalog_type`]: `rest`
* config_ref:iceberg_rest_catalog_endpoint,true,properties/cluster-properties[`iceberg_rest_catalog_endpoint`]: The endpoint URL for your Iceberg catalog, which you either manage directly, or is managed by an external catalog service.
* config_ref:iceberg_rest_catalog_authentication_mode,true,properties/cluster-properties[`iceberg_rest_catalog_authentication_mode`]: The authentication mode to use for the REST catalog. Choose from `oauth2`, `bearer`, or `none` (default).
ifdef::env-cloud[]
+
Redpanda recommends using `oauth2`.

endif::[]
** For `oauth2`, also configure the following properties:
+
--
Expand Down Expand Up @@ -111,6 +116,11 @@ Run the following `rpk` command:
----
rpk security secret create --name <secret-name> --value <secret-value> --scopes redpanda_cluster
----

Replace the placeholders with your own values:

- `<secret-name>`: The name of the secret you want to add. The secret name is also its ID. Use only the following characters: `^[A-Z][A-Z0-9_]*$`.
- `<secret-value>`: The value of the secret.
--

Cloud API::
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2066,7 +2066,11 @@ endif::[]
// tag::iceberg_rest_catalog_authentication_mode[]
=== iceberg_rest_catalog_authentication_mode

The authentication mode for client requests made to the Iceberg catalog. Choose from: `none`, `bearer`, and `oauth2`. In `bearer` mode, the token specified in `iceberg_rest_catalog_token` is used unconditionally, and no attempts are made to refresh the token. In `oauth2` mode, the credentials specified in `iceberg_rest_catalog_client_id` and `iceberg_rest_catalog_client_secret` are used to obtain a bearer token from the URI defined by `iceberg_rest_catalog_oauth2_server_uri.`.
The authentication mode for client requests made to the Iceberg catalog. Choose from: `none`, `bearer`, and `oauth2`. In `oauth2` mode, the credentials specified in `iceberg_rest_catalog_client_id` and `iceberg_rest_catalog_client_secret` are used to obtain a bearer token from the URI defined by `iceberg_rest_catalog_oauth2_server_uri.`. In `bearer` mode, the token specified in `iceberg_rest_catalog_token` is used unconditionally, and no attempts are made to refresh the token.

ifdef::env-cloud[]
Redpanda recommends using `oauth2`.
endif::[]

*Requires restart:* Yes

Expand Down
31 changes: 22 additions & 9 deletions modules/reference/pages/properties/topic-properties.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -544,6 +544,7 @@ Enable the Iceberg integration for the topic. You can choose one of four modes.

**Related topics**:

- xref:manage:iceberg/choose-iceberg-mode.adoc[]
- xref:manage:iceberg/about-iceberg-topics.adoc[]

---
Expand All @@ -560,6 +561,23 @@ Whether the corresponding Iceberg table is deleted upon deleting the topic.

---

==== redpanda.iceberg.invalid.record.action

Whether to write invalid records to a dead-letter queue (DLQ).

**Default**: `dlq_table`

**Values**:

- `drop`: Disable the DLQ and drop invalid records.
- `dlq_table`: Write invalid records to a separate DLQ Iceberg table.

**Related topics**:

- xref:manage:iceberg/about-iceberg-topics.adoc#manage-dead-letter-queue[Manage dead-letter queue]

---

==== redpanda.iceberg.partition.spec

The https://iceberg.apache.org/docs/nightly/partitioning/[partitioning^] specification for the Iceberg table.
Expand All @@ -568,20 +586,15 @@ The https://iceberg.apache.org/docs/nightly/partitioning/[partitioning^] specifi

**Related topics**:

- xref:manage:iceberg/about-iceberg-topics.adoc[]
- xref:manage:iceberg/about-iceberg-topics.adoc#use-custom-partitioning[Use custom partitioning]

---

==== redpanda.iceberg.invalid.record.action

Whether to write invalid records to a dead-letter queue (DLQ).

**Default**: `dlq_table`
==== redpanda.iceberg.target.lag.ms

**Values**:
Controls how often the data in the Iceberg table is refreshed with new data from the topic. Redpanda attempts to commit all data produced to the topic within the lag target, subject to resource availability.

- `drop`: Disable the DLQ and drop invalid records.
- `dlq_table`: Write invalid records to a separate DLQ Iceberg table.
**Default**: `60000`

**Related topics**:

Expand Down