SQL: struct support

[kbatuigas]

Description

This pull request introduces comprehensive documentation improvements for working with nested fields in Redpanda SQL, focusing on mapping nested Protobuf or Avro structures as SQL ROW columns and querying them directly. It clarifies the use of the struct_mapping_policy option, expands the reference for the ROW data type, and adds a dedicated how-to guide for querying nested fields.

New documentation and feature explanations:

• Added a new how-to page, query-nested-fields.adoc, detailing how to map topics with nested schemas as SQL tables using struct_mapping_policy = 'COMPOUND', how to query nested fields with ROW syntax, and how to handle recursive (cyclic) schemas.
• Updated the navigation (nav.adoc) to include the new "Query Topics with Nested Fields" guide.

Improvements to ROW data type documentation:

• Expanded the ROW data type reference to document field access (by position and name), wildcard projection, lexicographic comparison, NULL checks, text conversion, and usage in GROUP BY, ORDER BY, and JOIN clauses.
• Enhanced the ROW type summary to mention its support for field access, comparisons, and use in query clauses.

Clarifications to CREATE TABLE options:

• Clarified the struct_mapping_policy option in the CREATE TABLE documentation, emphasizing that COMPOUND maps nested structures to SQL ROW columns and noting that cyclic types are only supported in JSON mode.

Resolves https://github.com/redpanda-data/documentation-private/issues/
Review deadline: 20 May

Page previews

• Query Topics with Nested Fields (new)
• ROW (updated)
• CREATE TABLE (updated)

Checks

• New feature
• Content gap
• Support Follow-up
• Small fix (typos, links, copyedits, etc)

[netlify]

✅ Deploy Preview for rp-cloud ready!

Name	Link
🔨 Latest commit	3b589b0
🔍 Latest deploy log	https://app.netlify.com/projects/rp-cloud/deploys/6a110114dfc9ff000836eb71
😎 Deploy Preview	https://deploy-preview-586--rp-cloud.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify project configuration.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 5d28b7cd-7892-47e1-8354-59c7a7360603

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch DOC-2019-document-feature-record-structure-type-support

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[pkonrad1229]

That may be a nitpick, but stating that we are mapping as SQL ROW columns is not entirely true.

In PostgreSQL, a ROW is an anonymous record, in which you cannot explicitly set the sub-field names (they contain some generic f1, f2, f<n>... names that you cannot change).

What we do in the COMPOUND mapping is we actually create a User-Defined type, and set the names of the fields according to the schema.

Not sure if that's something that we want to explicitly state here, or maybe the ROW meaning here is something other than PostgreSQL ROW.

[kbatuigas]

Changed to

When a glossterm:topic[]'s schema includes nested Protobuf, Avro, or JSON message types, you can map those nested structures as user-defined types (UDTs) with named fields, queryable using SQL ROW field-access syntax, instead of opaque JSON. This makes nested fields queryable by name, includable in projections, and usable in WHERE, GROUP BY, and ORDER BY clauses, without parsing JSON at query time.

@mattschumpert do you have a preference on whether we explicitly mention user defined types?

[mattschumpert]

No idea. I defer to @pkonrad1229

[pkonrad1229]

I don't see an issue with why we shouldn't. Any user can check that for themselves by using e.g. pg_typeof function.

[pkonrad1229]

ditto here:

a ROW with fields

We may also say something along the lines of:

a structure/UDT with fields

[pkonrad1229]

I'm late to the party, as this was not modified in this PR :D I believe it's worth noting that the implicit tuple syntax works only when there are two or more expressions

[kbatuigas]

@pkonrad1229 Hm, that may have just surfaced as something related based on the Claude Code research... is it ok to leave on this page? The explanation is currently under the first sectionn https://deploy-preview-586--rp-cloud.netlify.app/redpanda-cloud/reference/sql/sql-data-types/row/#syntax

[pkonrad1229]

yeah sure, it's okay to leave it here. I only meant to say that we follow PostgreSQL rules for the ROW constructor, where the implicit syntax works only when there's more than 1 expression, so:

• (col) returns col extression
• (col1,col2) returns a ROW/record of those two columns

Postgres mentions this implicit syntax rule directly in their docs .

[mattschumpert]

Shouldn't we be specifying that the schema is registered for the topic using the TopicNamingStrategy naming convention @pkonrad1229 @kbatuigas ? You have to name it correctly for this to work, right? If people are not already familiar with this in SR we should educate them (point them to this naming convention)

[pkonrad1229]

Yes, the correct schema_subject is required for this to work. Not deeply familiar with the SR side, but from Oxla's perspective, the underlying naming strategy doesn't matter; only that the resolved subject matches a registered one. Two cases:

1. SR uses Confluent's default TopicNameStrategy → schema_subject can be omitted; Oxla defaults to <topic>-value.
2. SR uses a different strategy → schema_subject must be set explicitly in the CREATE TABLE options.

[pkonrad1229]

Side note: this comment is broad, not specific to nested fields. It's general CREATE TABLE + Kafka topic behavior. Question whether we should explain it inline here or just link to the CREATE TABLE reference docs where schema_subject semantics belong.

[mattschumpert]

Is schema_subject required or optional?

If it's required then maybe the naming convention is not mandatory @pkonrad1229 ?

[pkonrad1229]

From what I see, it's optional, and when omitted, Oxla resolves the subject to <topic>-value, so in this example it could be left out, and the outcome would be the same

[mattschumpert]

It says below this is optional. Comments should explain the same here (what is optional vs not)

[mattschumpert]

@grzebiel this warning would imply something very important to alert the user of (we dont really support querying iceberg topics with recursive types). However, I don't think this is the correct message here (at least not always), because Iceberg topics has a special handling encoding recursive Protobuf Struct fields as a JSON string in the Iceberg table. SO for protobuf, we do have a story for recursive fields (at least in the protobuf case).

So, how should this be adjusted.

[mattschumpert]

@kbatuigas wrong wording IMO. 'Query a topic with Iceberg history' is better.

What's here is technically incorrect because it makes it sound like you're ONLY querying the iceberg portion (tail). but in fact this link is to how to do a bridge query that queries both the live streaming data and iceberg history.

We should ensure we correct this everywhere.

[kbatuigas]

Updated

[micheleRP]

docs-team-standards review

Critical (must fix before merging to main)

1. Three unresolved // TODO markers in the source, two of which are already answered in the review thread.

   • query-nested-fields.adoc:23 — // TODO: Confirm TopicNameStrategy requirement. @pkonrad1229 answered in the review thread: "the underlying naming strategy doesn't matter; only that the resolved subject matches a registered one. (1) Confluent default → schema_subject can be omitted, Oxla defaults to <topic>-value. (2) other strategy → schema_subject must be set explicitly." Bake this answer into the body (and into the Prerequisites bullet that currently sits above the TODO).
   • query-nested-fields.adoc:42 — // TODO: Confirm schema_subject required when struct_mapping_policy=COMPOUND. Same SME reply applies: schema_subject is optional; resolves to <topic>-value by default. The bullet at line 41 currently only documents struct_mapping_policy defaulting to COMPOUND — add a parallel bullet for schema_subject.
   • row.adoc:239 — // TODO: SME — confirm whether nested array-of-struct access (...) works at GA, and whether wildcard expansion on an empty ROW ( `(ROW()).*` ) is supported. Tracked under OXLA-9444 and OXLA-9431. Still genuinely open; convert to a follow-up doc ticket so it doesn't ship as a comment.

2. SME terminology fix only partly applied — "ROW columns" framing still in two attributes/sections.

   • @pkonrad1229 pushed back on "SQL ROW columns" framing in the review thread: "What we do in the COMPOUND mapping is we actually create a User-Defined type… the page body should say 'a structure/UDT with fields'." The body of query-nested-fields.adoc was rewritten to use "user-defined types (UDTs)" — good.
   • Still leaking the old framing in two places:
     • query-nested-fields.adoc:2 (:description:): Map a topic with nested Protobuf, Avro, or JSON fields to SQL ROW columns, then query those fields directly.
     • row.adoc:243 (See also): xref:reference:sql/sql-statements/create-table.adoc[CREATE TABLE]: maps a Redpanda topic to a SQL table. Use `struct_mapping_policy = 'COMPOUND'` to surface nested topic fields as ROW columns.
   • Fix: match the body's wording. For example, description → Map a topic's nested fields to typed SQL columns and query them by name.; row.adoc:243 → ...to surface nested topic fields as user-defined types accessible with ROW field-access syntax.

Suggestions (should consider)

1. Open thread (@mattschumpert) — schema_subject and TopicNameStrategy education.

   • The review thread has an open question from @mattschumpert about whether the page should explicitly explain Schema Registry's TopicNameStrategy convention. @pkonrad1229 clarified the behavior (Oxla just resolves the subject; default is <topic>-value). Worth folding that explanation into the Prerequisites or the Map-the-topic-as-a-SQL-table section, even just a one-sentence "If your Schema Registry uses Confluent's default TopicNameStrategy, you can omit schema_subject — Redpanda SQL resolves it to <topic>-value."

2. Open thread (@mattschumpert) — Iceberg + recursive types warning may be technically wrong for Protobuf.

   • On line 99 of query-nested-fields.adoc, the warning says COMPOUND cannot map recursive types. @mattschumpert flagged that the related Iceberg behavior is more nuanced — Protobuf recursive fields are encoded as JSON strings in Iceberg, so there is a story for them. This wasn't fully resolved in the thread. Worth a follow-up with @grzebiel (who was tagged) before this lands in main.

3. Wording cleanup carryover ("Query an Iceberg topic" → "Query a topic with Iceberg history").

   • @mattschumpert asked for this rename to be applied "everywhere." The new page uses the updated wording (lines 127, 137). The nav still has *** xref:sql:query-data/query-iceberg-topics.adoc[Query Iceberg Topics] (line 357) — pre-existing, not modified by this PR, but consistent with what mattschumpert wants. Either update it now (since nav is already in this diff) or track it as a follow-up.

[micheleRP]

Suggested change

	For composite columns with declared field names — for example, columns mapped from a topic with `struct_mapping_policy = 'COMPOUND'` (see xref:reference:sql/sql-statements/create-table.adoc[CREATE TABLE]) — access fields by their declared names:
	For composite columns with declared field names, for example, columns mapped from a topic with `struct_mapping_policy = 'COMPOUND'` (see xref:reference:sql/sql-statements/create-table.adoc[CREATE TABLE]), access fields by their declared names:

[micheleRP]

left 3 little style suggestions!