Add BigLake REST Catalog doc #1443

kbatuigas · 2025-11-04T21:38:45Z

Description

Enable BigLake REST Catalog integration and query Iceberg topics in BigQuery. This guide describes deploying Redpanda using Docker on GCP Linux VMs and integrating a "prefilled" Iceberg topic in BigLake.

Resolves https://redpandadata.atlassian.net/browse/
Review deadline:

Page previews

Use Iceberg Topics using GCP BigLake

Checks

New feature
Content gap
Support Follow-up
Small fix (typos, links, copyedits, etc)

netlify · 2025-11-04T21:38:50Z

✅ Deploy Preview for redpanda-docs-preview ready!

Name	Link
🔨 Latest commit	`b5515c9`
🔍 Latest deploy log	https://app.netlify.com/projects/redpanda-docs-preview/deploys/690c43cb103aae00085d440a
😎 Deploy Preview	https://deploy-preview-1443--redpanda-docs-preview.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

coderabbitai · 2025-11-04T21:38:54Z

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

✨ Finishing touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Post copyable unit tests in a comment
Commit unit tests in branch DOC-1604-document-feature-google-biglake-bq-metastore-supp

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

modules/manage/pages/iceberg/iceberg-topics-gcp-biglake.adoc

nvartolomei · 2025-11-05T12:55:43Z

modules/manage/pages/iceberg/iceberg-topics-gcp-biglake.adoc

+
+=== Topic names
+
+BigLake does not support Iceberg table names that contain dots (`.`). When creating Iceberg topics in Redpanda that you plan to access through BigLake, ensure that the topic names do not include dots.


Direct them to the config property where they can configure a dot replacement? We can also note here that ~ (used in the default dlq naming pattern <topic>~dlq) is also not support so it should also be configured.

We should also note that these configs must be chosen carefully to avoid table name collisions i.e. for a topic named foo-bar and foo.bar if dot replacement is chosen to be - then a collision will happen.

@nvartolomei Added the property (we have not updated the cluster config reference yet, so I can't link reference page as of now, but can add it later). I also still kept "Ensure that topic names don't include dots" as a bullet point but I can remove it if it doesn't sound good

nvartolomei

looks good and mergeable

2 nits that can be fixed later

modules/manage/pages/iceberg/iceberg-topics-gcp-biglake.adoc

micheleRP · 2025-11-05T18:57:45Z

modules/manage/pages/iceberg/iceberg-topics-gcp-biglake.adoc

+
+For general information about Iceberg catalog integrations in Redpanda, see xref:manage:iceberg/use-iceberg-catalogs.adoc[].
+
+NOTE: BigLake support for the Iceberg REST Catalog API is currently in preview. See the https://cloud.google.com/biglake[BigLake product page] for the latest updates.


Suggested change

NOTE: BigLake support for the Iceberg REST Catalog API is currently in preview. See the https://cloud.google.com/biglake[BigLake product page] for the latest updates.

NOTE: BigLake support for the Iceberg REST Catalog API is currently in preview. See the https://cloud.google.com/biglake[BigLake product page^] for the latest updates.

Suggest rephrasing this note, since you or someone will need to be responsible for checking when it's out of preview. Something like: Check BigLake product page for latest availability or support for or something

modules/manage/pages/iceberg/iceberg-topics-gcp-biglake.adoc

Co-authored-by: Michele Cyran <michele@redpanda.com>

micheleRP

LGTM

mattschumpert

Ideally we should just delete all the 'create service account/bucket setup redpanda' sections.

modules/manage/pages/iceberg/iceberg-topics-gcp-biglake.adoc

mattschumpert · 2025-11-05T22:38:24Z

modules/manage/pages/iceberg/iceberg-topics-gcp-biglake.adoc

+- Use the `iceberg_topic_name_dot_replacement` cluster property to set a replacement string for dots in topic names. Ensure that the replacement value does not cause table name collisions. For example, `current.orders` and `current_orders` would both map to the same table name if you set the replacement to an underscore (`_`).
+- Ensure that the new topic names do not include dots.
+
+You must also set the `iceberg_dlq_table_suffix` property to a value that does not include dots or tildes (`~`). See <<configure-topic-for-iceberg>> for the list of cluster properties to set when enabling the BigLake REST catalog integration.


FYI @kbatuigas this property is undocumented in the cluster config reference (when I checked yesterday)

@mattschumpert for the new Iceberg properties, we'll get them into the reference once we run the import -- I believe we have to add a fix so it works with this beta.

mattschumpert · 2025-11-05T22:39:00Z

modules/manage/pages/iceberg/iceberg-topics-gcp-biglake.adoc

+
+BigLake does not support Iceberg table names that contain dots (`.`). When creating Iceberg topics in Redpanda that you plan to access through BigLake, either:
+
+- Use the `iceberg_topic_name_dot_replacement` cluster property to set a replacement string for dots in topic names. Ensure that the replacement value does not cause table name collisions. For example, `current.orders` and `current_orders` would both map to the same table name if you set the replacement to an underscore (`_`).


@kbatuigas lets also make sue this is documented

mattschumpert · 2025-11-05T22:47:41Z

modules/manage/pages/iceberg/iceberg-topics-gcp-biglake.adoc

+----
+SELECT
+    *
+FROM `<bucket-name>>redpanda`.transactions


@kbatuigas is this the right syntax? is it not a '.' before redpanda?

@mattschumpert Yes. This works, but <bucket-name>.redpanda.transactions does not. I haven't found a good reference doc for this. Perhaps @nvartolomei knows.

mattschumpert · 2025-11-05T22:58:28Z

modules/manage/pages/iceberg/iceberg-topics-gcp-biglake.adoc

+rpk topic alter-config transactions --set redpanda.iceberg.mode=value_schema_latest:subject=transactions
+----
+
+== Query Iceberg tables in BigQuery


@kbatuigas is there absolutely no configuration on the BIgLake /BQ side required? Do we not have to grant BigLake itself access to the cluster's bucket with some kind of policy/role? This is a bit surprising. There must be some setup, no? I thought at least in the BigQuery side you must create a 'Connection' in BigQuery TO BigLake itself (previously with object storage mode it was a connection to the cloud storage location). But maybe not. If there are zero steps on the GCP side that's impressive.

Also, here the hierarchy seems to be projectName.warehouseName.namespace.table: https://docs.cloud.google.com/biglake/docs/blms-rest-catalog#query_a_table

@mattschumpert updated to include a catalog creation step in BigLake.

kbatuigas · 2025-11-06T06:30:55Z

Ideally we should just delete all the 'create service account/bucket setup redpanda' sections.

@mattschumpert I updated this so that Set Up Google Cloud resources still includes the service account creation step, but added "If you don’t already have a Google Cloud service account to use." I took out the bucket creation step in placed it in the install and deploy Redpanda section that I have also marked as optional: http://localhost:5002/25.3/manage/iceberg/iceberg-topics-gcp-biglake/#optional-deploy-redpanda-quickstart-on-gcp

The idea is that they can try the quickstart if they want to set this up and test quickly on a fresh deploy of Redpanda on GCP. But if they already have a running cluster they can skip the quickstart part.

Some feedback that I got from CS is that the Redpanda install and deploy steps could go into a lab instead. If we're able to create a lab for this, then we can take out the optional steps and include a link to the lab.

…metastore-supp

Co-authored-by: Michele Cyran <michele@redpanda.com>

kbatuigas requested a review from a team as a code owner November 4, 2025 21:38

kbatuigas marked this pull request as draft November 4, 2025 21:38

kbatuigas requested a review from mattschumpert November 5, 2025 01:27

nvartolomei reviewed Nov 5, 2025

View reviewed changes

modules/manage/pages/iceberg/iceberg-topics-gcp-biglake.adoc Outdated Show resolved Hide resolved

nvartolomei reviewed Nov 5, 2025

View reviewed changes

kbatuigas marked this pull request as ready for review November 5, 2025 16:54

kbatuigas requested a review from nvartolomei November 5, 2025 18:07

kbatuigas added 7 commits November 5, 2025 10:10

Add BigLake REST Catalog doc

98737dd

Need to set DQL suffix for BigLake

3b97d97

Requirements for Iceberg topic names

0dde100

Typo

e8fce84

Add Docker install commands

10cb52b

Apply suggestions from SME review

95cbfb9

Add BigLake to What's new

72f7fa4

kbatuigas force-pushed the DOC-1604-document-feature-google-biglake-bq-metastore-supp branch from 160014c to 72f7fa4 Compare November 5, 2025 18:22

micheleRP reviewed Nov 5, 2025

View reviewed changes

modules/manage/pages/iceberg/iceberg-topics-gcp-biglake.adoc Outdated Show resolved Hide resolved

micheleRP reviewed Nov 5, 2025

View reviewed changes

modules/manage/pages/iceberg/iceberg-topics-gcp-biglake.adoc Outdated Show resolved Hide resolved

micheleRP reviewed Nov 5, 2025

View reviewed changes

modules/manage/pages/iceberg/iceberg-topics-gcp-biglake.adoc Outdated Show resolved Hide resolved

micheleRP reviewed Nov 5, 2025

View reviewed changes

modules/manage/pages/iceberg/iceberg-topics-gcp-biglake.adoc Outdated Show resolved Hide resolved

micheleRP reviewed Nov 5, 2025

View reviewed changes

modules/manage/pages/iceberg/iceberg-topics-gcp-biglake.adoc Outdated Show resolved Hide resolved

kbatuigas and others added 2 commits November 5, 2025 13:49

Apply suggestions from code review

bdd7f1e

Co-authored-by: Michele Cyran <michele@redpanda.com>

Apply suggestions from doc review

5b401db

kbatuigas requested a review from micheleRP November 5, 2025 22:17

micheleRP approved these changes Nov 5, 2025

View reviewed changes

mattschumpert approved these changes Nov 5, 2025

View reviewed changes

Indicate Redpanda docker example as optional step

be56601

Merge branch 'beta' into DOC-1604-document-feature-google-biglake-bq-…

b5515c9

…metastore-supp

kbatuigas merged commit 2500ef5 into beta Nov 6, 2025
5 checks passed

kbatuigas deleted the DOC-1604-document-feature-google-biglake-bq-metastore-supp branch November 6, 2025 06:55

paulohtb6 pushed a commit that referenced this pull request Nov 19, 2025

Add BigLake REST Catalog doc (#1443)

c02350d

Co-authored-by: Michele Cyran <michele@redpanda.com>


		=== Topic names

		BigLake does not support Iceberg table names that contain dots (`.`). When creating Iceberg topics in Redpanda that you plan to access through BigLake, ensure that the topic names do not include dots.


		For general information about Iceberg catalog integrations in Redpanda, see xref:manage:iceberg/use-iceberg-catalogs.adoc[].

		NOTE: BigLake support for the Iceberg REST Catalog API is currently in preview. See the https://cloud.google.com/biglake[BigLake product page] for the latest updates.


		BigLake does not support Iceberg table names that contain dots (`.`). When creating Iceberg topics in Redpanda that you plan to access through BigLake, either:

		- Use the `iceberg_topic_name_dot_replacement` cluster property to set a replacement string for dots in topic names. Ensure that the replacement value does not cause table name collisions. For example, `current.orders` and `current_orders` would both map to the same table name if you set the replacement to an underscore (`_`).

Add BigLake REST Catalog doc #1443

Add BigLake REST Catalog doc #1443

Uh oh!

Conversation

kbatuigas commented Nov 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Page previews

Checks

Uh oh!

netlify bot commented Nov 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for redpanda-docs-preview ready!

Uh oh!

coderabbitai bot commented Nov 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nvartolomei left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

micheleRP left a comment

Choose a reason for hiding this comment

Uh oh!

mattschumpert left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kbatuigas commented Nov 6, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

kbatuigas commented Nov 4, 2025 •

edited

Loading

netlify bot commented Nov 4, 2025 •

edited

Loading

coderabbitai bot commented Nov 4, 2025 •

edited

Loading