-
Notifications
You must be signed in to change notification settings - Fork 1.2k
[iceberg] support iceberg rest catalog for iceberg compatibility #5817
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
Hi, @nickdelnano. Could you please review this PR when you have a moment? Thanks a lot! |
@LsomeYeah yes I will review it in the next few days. Thanks for making this change! |
This reverts commit 27485f0.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@LsomeYeah nice job accomplishing this with a relatively simple change!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good to me! Thanks @LsomeYeah
Purpose
Linked issue: #4394
This PR is inspired by the implementation of #5621, many thanks to @nickdelnano.
Since some iceberg rest catalog do not support
registerTable
orcommitTransaction
, we usecreateTable
andupdateTable
to commit metadata to rest catalog.When creating table,
fieldId
in iceberg will be forced to start from 1, whilefieldId
in paimon usually start from 0. If we directly use the schema extracted from paimon to create iceberg table, thefieldId
will be in disorder, and this may cause incorrectness when reading by iceberg reader. So we use an emptySchema to create iceberg table, and add the corresponding schemas byupdateTable
later. Given the id-0 will be occupied by the empty schema, there will be 1-unit offset between the schema-id in metadata stored in rest catalog and the schema-id in paimon.Still need to do:
Tests
IcebergRestMetadataCommitterTest
IcebergRestMetadataCommitterITCase
API and Format
Documentation