Skip to content

[iceberg] support iceberg rest catalog for iceberg compatibility #5817

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 14 commits into from
Jul 14, 2025

Conversation

LsomeYeah
Copy link
Contributor

@LsomeYeah LsomeYeah commented Jun 30, 2025

Purpose

Linked issue: #4394

This PR is inspired by the implementation of #5621, many thanks to @nickdelnano.

Since some iceberg rest catalog do not support registerTable or commitTransaction, we use createTable and updateTable to commit metadata to rest catalog.

When creating table, fieldId in iceberg will be forced to start from 1, while fieldId in paimon usually start from 0. If we directly use the schema extracted from paimon to create iceberg table, the fieldId will be in disorder, and this may cause incorrectness when reading by iceberg reader. So we use an emptySchema to create iceberg table, and add the corresponding schemas by updateTable later. Given the id-0 will be occupied by the empty schema, there will be 1-unit offset between the schema-id in metadata stored in rest catalog and the schema-id in paimon.

Still need to do:

  1. tag support

Tests

IcebergRestMetadataCommitterTest
IcebergRestMetadataCommitterITCase

API and Format

Documentation

@LsomeYeah
Copy link
Contributor Author

Hi, @nickdelnano. Could you please review this PR when you have a moment? Thanks a lot!

@nickdelnano
Copy link
Contributor

@LsomeYeah yes I will review it in the next few days. Thanks for making this change!

Copy link
Contributor

@nickdelnano nickdelnano left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@LsomeYeah nice job accomplishing this with a relatively simple change!

Copy link
Contributor

@JingsongLi JingsongLi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me! Thanks @LsomeYeah

@JingsongLi JingsongLi merged commit 01a85d1 into apache:master Jul 14, 2025
21 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants