From 5cf8db73b720b65ccf5f061b7715e66de181b47f Mon Sep 17 00:00:00 2001 From: kbatuigas <36839689+kbatuigas@users.noreply.github.com> Date: Fri, 13 Jun 2025 11:36:38 -0700 Subject: [PATCH 1/4] Partition evolution in Unity supported for external Iceberg engines --- .../iceberg/iceberg-topics-databricks-unity.adoc | 11 +++-------- 1 file changed, 3 insertions(+), 8 deletions(-) diff --git a/modules/manage/pages/iceberg/iceberg-topics-databricks-unity.adoc b/modules/manage/pages/iceberg/iceberg-topics-databricks-unity.adoc index 450a54a334..10c89cd383 100644 --- a/modules/manage/pages/iceberg/iceberg-topics-databricks-unity.adoc +++ b/modules/manage/pages/iceberg/iceberg-topics-databricks-unity.adoc @@ -27,13 +27,9 @@ endif::[] * Workspace admin privileges to complete the steps to create a Unity Catalog storage credential and external location that connects your cluster's Tiered Storage bucket to Databricks. == Limitations + +The following data types are not currently supported for managed Iceberg tables: -* Databricks Managed Iceberg tables do not currently support partition evolution. If you define a xref:manage:iceberg/about-iceberg-topics.adoc#use-custom-partitioning[custom partitioning] scheme for an Iceberg topic, you must not change it later. -+ -The cluster property config_ref:iceberg_default_partition_spec,true,properties/cluster-properties[`iceberg_default_partition_spec`] defines the cluster-wide partitioning scheme, by default configured to `(hour(redpanda.timestamp))`. To use a different cluster-wide default, configure this property before creating any Iceberg-enabled topics. You must only override the default partitioning scheme on the topic level for new Iceberg topics that do not yet contain data. -* The following data types are not currently supported for managed Iceberg tables: -+ --- |=== | Iceberg type | Equivalent Avro type @@ -42,8 +38,7 @@ The cluster property config_ref:iceberg_default_partition_spec,true,properties/c | time | time-millis, time-micros |=== --- -+ + There are no limitations for Protobuf types. == Create a Unity Catalog storage credential From c7bce32bff059d69eeea38036993d1dd725e2aae Mon Sep 17 00:00:00 2001 From: kbatuigas <36839689+kbatuigas@users.noreply.github.com> Date: Fri, 13 Jun 2025 11:39:56 -0700 Subject: [PATCH 2/4] DOC-1437 Recommend oauth2 --- .../manage/partials/iceberg/use-iceberg-catalogs.adoc | 10 ++++++++++ .../reference/pages/properties/cluster-properties.adoc | 4 ++++ 2 files changed, 14 insertions(+) diff --git a/modules/manage/partials/iceberg/use-iceberg-catalogs.adoc b/modules/manage/partials/iceberg/use-iceberg-catalogs.adoc index 4a1fd93571..ef8db8c3b4 100644 --- a/modules/manage/partials/iceberg/use-iceberg-catalogs.adoc +++ b/modules/manage/partials/iceberg/use-iceberg-catalogs.adoc @@ -60,6 +60,11 @@ To connect to a REST catalog, set the following cluster configuration properties * config_ref:iceberg_catalog_type,true,properties/cluster-properties[`iceberg_catalog_type`]: `rest` * config_ref:iceberg_rest_catalog_endpoint,true,properties/cluster-properties[`iceberg_rest_catalog_endpoint`]: The endpoint URL for your Iceberg catalog, which you either manage directly, or is managed by an external catalog service. * config_ref:iceberg_rest_catalog_authentication_mode,true,properties/cluster-properties[`iceberg_rest_catalog_authentication_mode`]: The authentication mode to use for the REST catalog. Choose from `oauth2`, `bearer`, or `none` (default). +ifdef::env-cloud[] ++ +Redpanda recommends using `oauth2`. + +endif::[] ** For `oauth2`, also configure the following properties: + -- @@ -111,6 +116,11 @@ Run the following `rpk` command: ---- rpk security secret create --name --value --scopes redpanda_cluster ---- + +Replace the placeholders with your own values: + +- ``: The name of the secret you want to add. The secret name is also its ID. Use only the following characters: `^[A-Z][A-Z0-9_]*$`. +- ``: The value of the secret. -- Cloud API:: diff --git a/modules/reference/pages/properties/cluster-properties.adoc b/modules/reference/pages/properties/cluster-properties.adoc index cd39f4d37e..880373af22 100644 --- a/modules/reference/pages/properties/cluster-properties.adoc +++ b/modules/reference/pages/properties/cluster-properties.adoc @@ -2068,6 +2068,10 @@ endif::[] The authentication mode for client requests made to the Iceberg catalog. Choose from: `none`, `bearer`, and `oauth2`. In `bearer` mode, the token specified in `iceberg_rest_catalog_token` is used unconditionally, and no attempts are made to refresh the token. In `oauth2` mode, the credentials specified in `iceberg_rest_catalog_client_id` and `iceberg_rest_catalog_client_secret` are used to obtain a bearer token from the URI defined by `iceberg_rest_catalog_oauth2_server_uri.`. +ifdef::env-cloud[] +Redpanda recommends using `oauth2`. +endif::[] + *Requires restart:* Yes *Visibility:* `user` From 9d21596f5ef474f67ba24c04c477a9bf9ba025a0 Mon Sep 17 00:00:00 2001 From: kbatuigas <36839689+kbatuigas@users.noreply.github.com> Date: Fri, 13 Jun 2025 13:04:07 -0700 Subject: [PATCH 3/4] DOC-1392 Add redpanda.iceberg.target.lag.ms --- .../pages/properties/topic-properties.adoc | 31 +++++++++++++------ 1 file changed, 22 insertions(+), 9 deletions(-) diff --git a/modules/reference/pages/properties/topic-properties.adoc b/modules/reference/pages/properties/topic-properties.adoc index efc979413a..859d2b8d2f 100644 --- a/modules/reference/pages/properties/topic-properties.adoc +++ b/modules/reference/pages/properties/topic-properties.adoc @@ -544,6 +544,7 @@ Enable the Iceberg integration for the topic. You can choose one of four modes. **Related topics**: +- xref:manage:iceberg/choose-iceberg-mode.adoc[] - xref:manage:iceberg/about-iceberg-topics.adoc[] --- @@ -560,6 +561,23 @@ Whether the corresponding Iceberg table is deleted upon deleting the topic. --- +==== redpanda.iceberg.invalid.record.action + +Whether to write invalid records to a dead-letter queue (DLQ). + +**Default**: `dlq_table` + +**Values**: + +- `drop`: Disable the DLQ and drop invalid records. +- `dlq_table`: Write invalid records to a separate DLQ Iceberg table. + +**Related topics**: + +- xref:manage:iceberg/about-iceberg-topics.adoc#manage-dead-letter-queue[Manage dead-letter queue] + +--- + ==== redpanda.iceberg.partition.spec The https://iceberg.apache.org/docs/nightly/partitioning/[partitioning^] specification for the Iceberg table. @@ -568,20 +586,15 @@ The https://iceberg.apache.org/docs/nightly/partitioning/[partitioning^] specifi **Related topics**: -- xref:manage:iceberg/about-iceberg-topics.adoc[] +- xref:manage:iceberg/about-iceberg-topics.adoc#use-custom-partitioning[Use custom partitioning] --- -==== redpanda.iceberg.invalid.record.action - -Whether to write invalid records to a dead-letter queue (DLQ). - -**Default**: `dlq_table` +==== redpanda.iceberg.target.lag.ms -**Values**: +Controls how often the data in the Iceberg table is refreshed with new data from the topic. Redpanda attempts to commit all data produced to the topic within the lag target, subject to resource availability. -- `drop`: Disable the DLQ and drop invalid records. -- `dlq_table`: Write invalid records to a separate DLQ Iceberg table. +**Default**: `60000` **Related topics**: From 99f53dc871fefd23936c1ad41487ab33b59e7175 Mon Sep 17 00:00:00 2001 From: Kat Batuigas <36839689+kbatuigas@users.noreply.github.com> Date: Mon, 16 Jun 2025 09:36:54 -0700 Subject: [PATCH 4/4] Update modules/reference/pages/properties/cluster-properties.adoc Co-authored-by: Michele Cyran --- modules/reference/pages/properties/cluster-properties.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/reference/pages/properties/cluster-properties.adoc b/modules/reference/pages/properties/cluster-properties.adoc index 880373af22..8f9b8789df 100644 --- a/modules/reference/pages/properties/cluster-properties.adoc +++ b/modules/reference/pages/properties/cluster-properties.adoc @@ -2066,7 +2066,7 @@ endif::[] // tag::iceberg_rest_catalog_authentication_mode[] === iceberg_rest_catalog_authentication_mode -The authentication mode for client requests made to the Iceberg catalog. Choose from: `none`, `bearer`, and `oauth2`. In `bearer` mode, the token specified in `iceberg_rest_catalog_token` is used unconditionally, and no attempts are made to refresh the token. In `oauth2` mode, the credentials specified in `iceberg_rest_catalog_client_id` and `iceberg_rest_catalog_client_secret` are used to obtain a bearer token from the URI defined by `iceberg_rest_catalog_oauth2_server_uri.`. +The authentication mode for client requests made to the Iceberg catalog. Choose from: `none`, `bearer`, and `oauth2`. In `oauth2` mode, the credentials specified in `iceberg_rest_catalog_client_id` and `iceberg_rest_catalog_client_secret` are used to obtain a bearer token from the URI defined by `iceberg_rest_catalog_oauth2_server_uri.`. In `bearer` mode, the token specified in `iceberg_rest_catalog_token` is used unconditionally, and no attempts are made to refresh the token. ifdef::env-cloud[] Redpanda recommends using `oauth2`.