diff --git a/modules/manage/pages/iceberg/iceberg-topics-databricks-unity.adoc b/modules/manage/pages/iceberg/iceberg-topics-databricks-unity.adoc index 450a54a334..10c89cd383 100644 --- a/modules/manage/pages/iceberg/iceberg-topics-databricks-unity.adoc +++ b/modules/manage/pages/iceberg/iceberg-topics-databricks-unity.adoc @@ -27,13 +27,9 @@ endif::[] * Workspace admin privileges to complete the steps to create a Unity Catalog storage credential and external location that connects your cluster's Tiered Storage bucket to Databricks. == Limitations + +The following data types are not currently supported for managed Iceberg tables: -* Databricks Managed Iceberg tables do not currently support partition evolution. If you define a xref:manage:iceberg/about-iceberg-topics.adoc#use-custom-partitioning[custom partitioning] scheme for an Iceberg topic, you must not change it later. -+ -The cluster property config_ref:iceberg_default_partition_spec,true,properties/cluster-properties[`iceberg_default_partition_spec`] defines the cluster-wide partitioning scheme, by default configured to `(hour(redpanda.timestamp))`. To use a different cluster-wide default, configure this property before creating any Iceberg-enabled topics. You must only override the default partitioning scheme on the topic level for new Iceberg topics that do not yet contain data. -* The following data types are not currently supported for managed Iceberg tables: -+ --- |=== | Iceberg type | Equivalent Avro type @@ -42,8 +38,7 @@ The cluster property config_ref:iceberg_default_partition_spec,true,properties/c | time | time-millis, time-micros |=== --- -+ + There are no limitations for Protobuf types. == Create a Unity Catalog storage credential diff --git a/modules/manage/partials/iceberg/use-iceberg-catalogs.adoc b/modules/manage/partials/iceberg/use-iceberg-catalogs.adoc index 4a1fd93571..ef8db8c3b4 100644 --- a/modules/manage/partials/iceberg/use-iceberg-catalogs.adoc +++ b/modules/manage/partials/iceberg/use-iceberg-catalogs.adoc @@ -60,6 +60,11 @@ To connect to a REST catalog, set the following cluster configuration properties * config_ref:iceberg_catalog_type,true,properties/cluster-properties[`iceberg_catalog_type`]: `rest` * config_ref:iceberg_rest_catalog_endpoint,true,properties/cluster-properties[`iceberg_rest_catalog_endpoint`]: The endpoint URL for your Iceberg catalog, which you either manage directly, or is managed by an external catalog service. * config_ref:iceberg_rest_catalog_authentication_mode,true,properties/cluster-properties[`iceberg_rest_catalog_authentication_mode`]: The authentication mode to use for the REST catalog. Choose from `oauth2`, `bearer`, or `none` (default). +ifdef::env-cloud[] ++ +Redpanda recommends using `oauth2`. + +endif::[] ** For `oauth2`, also configure the following properties: + -- @@ -111,6 +116,11 @@ Run the following `rpk` command: ---- rpk security secret create --name --value --scopes redpanda_cluster ---- + +Replace the placeholders with your own values: + +- ``: The name of the secret you want to add. The secret name is also its ID. Use only the following characters: `^[A-Z][A-Z0-9_]*$`. +- ``: The value of the secret. -- Cloud API:: diff --git a/modules/reference/pages/properties/cluster-properties.adoc b/modules/reference/pages/properties/cluster-properties.adoc index cd39f4d37e..8f9b8789df 100644 --- a/modules/reference/pages/properties/cluster-properties.adoc +++ b/modules/reference/pages/properties/cluster-properties.adoc @@ -2066,7 +2066,11 @@ endif::[] // tag::iceberg_rest_catalog_authentication_mode[] === iceberg_rest_catalog_authentication_mode -The authentication mode for client requests made to the Iceberg catalog. Choose from: `none`, `bearer`, and `oauth2`. In `bearer` mode, the token specified in `iceberg_rest_catalog_token` is used unconditionally, and no attempts are made to refresh the token. In `oauth2` mode, the credentials specified in `iceberg_rest_catalog_client_id` and `iceberg_rest_catalog_client_secret` are used to obtain a bearer token from the URI defined by `iceberg_rest_catalog_oauth2_server_uri.`. +The authentication mode for client requests made to the Iceberg catalog. Choose from: `none`, `bearer`, and `oauth2`. In `oauth2` mode, the credentials specified in `iceberg_rest_catalog_client_id` and `iceberg_rest_catalog_client_secret` are used to obtain a bearer token from the URI defined by `iceberg_rest_catalog_oauth2_server_uri.`. In `bearer` mode, the token specified in `iceberg_rest_catalog_token` is used unconditionally, and no attempts are made to refresh the token. + +ifdef::env-cloud[] +Redpanda recommends using `oauth2`. +endif::[] *Requires restart:* Yes diff --git a/modules/reference/pages/properties/topic-properties.adoc b/modules/reference/pages/properties/topic-properties.adoc index efc979413a..859d2b8d2f 100644 --- a/modules/reference/pages/properties/topic-properties.adoc +++ b/modules/reference/pages/properties/topic-properties.adoc @@ -544,6 +544,7 @@ Enable the Iceberg integration for the topic. You can choose one of four modes. **Related topics**: +- xref:manage:iceberg/choose-iceberg-mode.adoc[] - xref:manage:iceberg/about-iceberg-topics.adoc[] --- @@ -560,6 +561,23 @@ Whether the corresponding Iceberg table is deleted upon deleting the topic. --- +==== redpanda.iceberg.invalid.record.action + +Whether to write invalid records to a dead-letter queue (DLQ). + +**Default**: `dlq_table` + +**Values**: + +- `drop`: Disable the DLQ and drop invalid records. +- `dlq_table`: Write invalid records to a separate DLQ Iceberg table. + +**Related topics**: + +- xref:manage:iceberg/about-iceberg-topics.adoc#manage-dead-letter-queue[Manage dead-letter queue] + +--- + ==== redpanda.iceberg.partition.spec The https://iceberg.apache.org/docs/nightly/partitioning/[partitioning^] specification for the Iceberg table. @@ -568,20 +586,15 @@ The https://iceberg.apache.org/docs/nightly/partitioning/[partitioning^] specifi **Related topics**: -- xref:manage:iceberg/about-iceberg-topics.adoc[] +- xref:manage:iceberg/about-iceberg-topics.adoc#use-custom-partitioning[Use custom partitioning] --- -==== redpanda.iceberg.invalid.record.action - -Whether to write invalid records to a dead-letter queue (DLQ). - -**Default**: `dlq_table` +==== redpanda.iceberg.target.lag.ms -**Values**: +Controls how often the data in the Iceberg table is refreshed with new data from the topic. Redpanda attempts to commit all data produced to the topic within the lag target, subject to resource availability. -- `drop`: Disable the DLQ and drop invalid records. -- `dlq_table`: Write invalid records to a separate DLQ Iceberg table. +**Default**: `60000` **Related topics**: