From d19a1e08b2c8083e3f1b647216a214b125270a2c Mon Sep 17 00:00:00 2001 From: Kat Batuigas Date: Thu, 30 Jul 2026 17:13:16 -0700 Subject: [PATCH 1/4] Add new decode options --- .../sql/sql-statements/create-table.adoc | 61 +++++++++++++++++++ 1 file changed, 61 insertions(+) diff --git a/modules/reference/pages/sql/sql-statements/create-table.adoc b/modules/reference/pages/sql/sql-statements/create-table.adoc index 5ca60a9cc..26dec348a 100644 --- a/modules/reference/pages/sql/sql-statements/create-table.adoc +++ b/modules/reference/pages/sql/sql-statements/create-table.adoc @@ -70,6 +70,44 @@ a|Whether records on the topic are encoded with the https://docs.confluent.io/pl * `'false'`: Records are raw Protobuf or Avro without the wire-format prefix. Only valid when `schema_lookup_policy = 'LATEST'`. + +|`key_decode_mode` +|STRING +|No +a|How to interpret record keys. Applies to both live topic records and Iceberg-committed records. + +* `binary` (default): Exposes the key as raw `bytea`. +* `string`: Decodes the key as a UTF-8 string. Invalid bytes are replaced with the Unicode replacement character (`U+FFFD`). +* `schema_latest`: Decodes every key with the latest schema registered for the key subject. +* `schema_id_prefix`: Decodes each key using the schema ID embedded in the record's Confluent wire-format header. + +|`key_schema_subject` +|STRING +|No +|Schema Registry subject for the key schema. Defaults to the topic-name strategy (`-key`). Override when the producer does not use the topic-name strategy. + +|`key_schema_message_full_name` +|STRING +|No +|Full Protobuf message name for the key schema. Required when the key schema defines more than one message. Defaults to the first message in the schema. + +|`key_confluent_wire_protocol` +|STRING +|No +a|Whether record keys carry the Confluent Schema Registry wire-format prefix. Only valid when `key_decode_mode = 'schema_latest'`. The `schema_id_prefix` mode always requires the prefix. + +* `'false'` (default): Keys are raw encoded bytes without the prefix. +* `'true'`: Keys carry the wire-format prefix. + +|`header_value_type` +|STRING +|No +a|How to store record header values. + +* `binary` (default): Exposes header values as raw `bytea`. +* `string`: Decodes header values as UTF-8 strings. Invalid bytes are replaced with `U+FFFD`. + +Header keys are always `text`. |=== [#auto-added-columns] @@ -116,6 +154,14 @@ Contains Kafka record metadata. Always present on every row. |Kafka timestamp type code. `0` for `CreateTime`, `1` for `LogAppendTime`. `NULL` when not available. |=== +By default, `key` is raw `bytea` and each header `value` is raw `bytea`. To decode them, set `key_decode_mode`, `key_schema_subject`, `key_confluent_wire_protocol`, or `header_value_type` (see <>): + +* When you decode the key with a schema mode, `key` becomes a struct that you access with `((redpanda).key).field_name`. +* When `key_decode_mode = 'string'`, `key` is `text`. +* When `header_value_type = 'string'`, each header `value` is `text`. Header keys are always `text`. + +A `NULL` key stays `NULL`. A key that fails to decode follows the table's `error_handling_policy`. Run `DESCRIBE TABLE =>` to see the decoded key type. + === `redpanda_raw` Populated only when `error_handling_policy = 'FILL_NULL'` and a record fails to decode. In all other cases, `redpanda_raw` is `NULL`. @@ -179,3 +225,18 @@ WITH ( error_handling_policy = 'DROP_RECORD' ); ---- + +=== Decode record keys and headers + +Map a topic, decoding keys with the schema ID embedded in each record and storing header values as strings: + +[source,sql] +---- +CREATE TABLE default_redpanda_catalog=>orders +WITH ( + topic = 'orders', + schema_subject = 'orders-value', + key_decode_mode = 'schema_id_prefix', + header_value_type = 'string' +); +---- From 44e2d5dca6a30f6aa6ca177861518f95592b6a40 Mon Sep 17 00:00:00 2001 From: Kat Batuigas Date: Thu, 30 Jul 2026 17:13:53 -0700 Subject: [PATCH 2/4] Draft querying decoded values section --- .../query-data/query-iceberg-topics.adoc | 43 +++++++++++++++++++ 1 file changed, 43 insertions(+) diff --git a/modules/sql/pages/query-data/query-iceberg-topics.adoc b/modules/sql/pages/query-data/query-iceberg-topics.adoc index d028c8ae5..f48b3ce7e 100644 --- a/modules/sql/pages/query-data/query-iceberg-topics.adoc +++ b/modules/sql/pages/query-data/query-iceberg-topics.adoc @@ -100,6 +100,49 @@ Redpanda SQL plans the union internally, so you don't write a `UNION ALL`. NOTE: Iceberg-committed data persists independently of Redpanda topic retention. Queries continue to return records past the Redpanda topic's retention window, provided they were committed to Iceberg first. +== Query decoded keys and headers + +By default, the `redpanda` metadata column exposes the record `key` and header values as raw `bytea`. To query them as decoded values, set the key and header options when you map the table. For the full list, see xref:reference:sql/sql-statements/create-table.adoc#options[CREATE TABLE options]. + +For an Iceberg-enabled topic, the table's key mode must match how the topic encodes keys and headers into its Iceberg table, which is set by the topic's `redpanda.iceberg.mode` property. See xref:manage:iceberg/specify-iceberg-schema.adoc#configure-key-value-and-header-translation[Configure key, value, and header translation]. Set the `CREATE TABLE` options to match: + +[cols="<55%,<45%",options="header"] +|=== +|Topic `redpanda.iceberg.mode` |`CREATE TABLE` options + +|`value_schema_id_prefix` +|Omit the key options. The key stays `bytea`. + +|`key:mode=schema_id_prefix;value:mode=schema_id_prefix` +|`key_decode_mode = 'schema_id_prefix'` + +|`key:mode=string;value:mode=schema_id_prefix` +|`key_decode_mode = 'string'` + +|`headers:value_type=string;value:mode=schema_id_prefix` +|`header_value_type = 'string'` +|=== + +If the table's key mode doesn't match the topic's Iceberg encoding, the query fails at planning time with a type mismatch instead of returning incorrect data. + +Map the table with the matching options, then access the decoded key and header values in your query: + +[source,sql] +---- +CREATE TABLE default_redpanda_catalog=>orders WITH ( + topic = 'orders', + schema_subject = 'orders-value', + key_decode_mode = 'schema_id_prefix' +); + +SELECT ((redpanda).key).customer_id, (redpanda).headers +FROM default_redpanda_catalog=>orders; +---- + +When you decode the key with a schema, `(redpanda).key` is a struct whose fields you access with `((redpanda).key).field_name`. With `key_decode_mode = 'string'`, `(redpanda).key` is `text`. Run `DESCRIBE TABLE default_redpanda_catalog=>orders` to see the decoded key type. + +NOTE: `key_decode_mode` and `header_value_type` are fixed for the lifetime of the table. `ALTER` rejects a change to either. To change how keys or header values are decoded, drop and recreate the table. + == Handle schema differences A topic schema can evolve over time. You might add or remove fields in your Schema Registry value subject as your application changes. Redpanda writes new records to the Iceberg table forward-only: it adds columns to the Iceberg table when the topic schema widens, but it does not drop columns from the Iceberg table when the topic schema narrows. As a result, the Iceberg table can carry columns the current topic schema doesn't have. From a33ffc6c7d6bd94b806cc4adbfac64161d4ab419 Mon Sep 17 00:00:00 2001 From: Kat Batuigas Date: Thu, 30 Jul 2026 17:14:01 -0700 Subject: [PATCH 3/4] Add to What's new --- modules/get-started/pages/whats-new-cloud.adoc | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/modules/get-started/pages/whats-new-cloud.adoc b/modules/get-started/pages/whats-new-cloud.adoc index baa96ac22..60198d74c 100644 --- a/modules/get-started/pages/whats-new-cloud.adoc +++ b/modules/get-started/pages/whats-new-cloud.adoc @@ -6,6 +6,12 @@ This page lists new features added to Redpanda Cloud. +== August 2026 + +=== Query decoded keys and headers from Iceberg topics with SQL + +Redpanda SQL can now read decoded record keys and header values when you query Iceberg-enabled topics, instead of only raw bytes. Set `key_decode_mode` and `header_value_type` on `CREATE TABLE` to expose the `redpanda` metadata column's `key` as a decoded struct or string and its header values as text. See xref:sql:query-data/query-iceberg-topics.adoc#query-decoded-keys-and-headers[Query decoded keys and headers]. + == July 2026 === Schema Registry contexts enabled by default From 8b959cfa72587719166e0a64d82fe90432d21501 Mon Sep 17 00:00:00 2001 From: Kat Batuigas Date: Thu, 30 Jul 2026 17:27:43 -0700 Subject: [PATCH 4/4] Specify limitations per SME --- .../reference/pages/sql/sql-statements/create-table.adoc | 8 ++++++++ modules/sql/pages/query-data/query-iceberg-topics.adoc | 2 ++ 2 files changed, 10 insertions(+) diff --git a/modules/reference/pages/sql/sql-statements/create-table.adoc b/modules/reference/pages/sql/sql-statements/create-table.adoc index 26dec348a..874df3b8e 100644 --- a/modules/reference/pages/sql/sql-statements/create-table.adoc +++ b/modules/reference/pages/sql/sql-statements/create-table.adoc @@ -81,6 +81,8 @@ a|How to interpret record keys. Applies to both live topic records and Iceberg-c * `schema_latest`: Decodes every key with the latest schema registered for the key subject. * `schema_id_prefix`: Decodes each key using the schema ID embedded in the record's Confluent wire-format header. +Fixed for the table's lifetime. `ALTER` cannot change it; recreate the table instead. + |`key_schema_subject` |STRING |No @@ -108,6 +110,8 @@ a|How to store record header values. * `string`: Decodes header values as UTF-8 strings. Invalid bytes are replaced with `U+FFFD`. Header keys are always `text`. + +Fixed for the table's lifetime. `ALTER` cannot change it; recreate the table instead. |=== [#auto-added-columns] @@ -115,6 +119,8 @@ Header keys are always `text`. Every catalog-mapped table includes two struct columns in addition to the columns derived from the topic's schema. Redpanda SQL adds these columns to both Kafka-backed and Iceberg-backed tables. The names `redpanda` and `redpanda_raw` are reserved. A topic schema cannot define columns with these names. +`REFRESH` also rejects a schema whose declared type names begin with `__redpanda_`, a prefix reserved for Redpanda's internal metadata types. For JSON schemas, a top-level property named `__json_root` is reserved and rejected as well. + === `redpanda` Contains Kafka record metadata. Always present on every row. @@ -162,6 +168,8 @@ By default, `key` is raw `bytea` and each header `value` is raw `bytea`. To deco A `NULL` key stays `NULL`. A key that fails to decode follows the table's `error_handling_policy`. Run `DESCRIBE TABLE =>
` to see the decoded key type. +The key and value can use different schemas and formats (for example, an Avro key with a Protobuf value), and key and value field names can overlap without conflict. + === `redpanda_raw` Populated only when `error_handling_policy = 'FILL_NULL'` and a record fails to decode. In all other cases, `redpanda_raw` is `NULL`. diff --git a/modules/sql/pages/query-data/query-iceberg-topics.adoc b/modules/sql/pages/query-data/query-iceberg-topics.adoc index f48b3ce7e..36bb2f1d8 100644 --- a/modules/sql/pages/query-data/query-iceberg-topics.adoc +++ b/modules/sql/pages/query-data/query-iceberg-topics.adoc @@ -125,6 +125,8 @@ For an Iceberg-enabled topic, the table's key mode must match how the topic enco If the table's key mode doesn't match the topic's Iceberg encoding, the query fails at planning time with a type mismatch instead of returning incorrect data. +Redpanda SQL exposes decoded value fields as top-level columns, so a topic whose `redpanda.iceberg.mode` nests value fields under a `value` struct (`value:layout=nested`) is not supported. The Iceberg table's `value` column has no counterpart in the flattened topic schema, so the query fails at planning time with the `Kafka schema must be a name-superset of Iceberg schema` error described in <>. + Map the table with the matching options, then access the decoded key and header values in your query: [source,sql]