New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[HUDI-6857] Update BigQuerySyncTool docs #9710
[HUDI-6857] Update BigQuerySyncTool docs #9710
Conversation
hoodie.datasource.write.drop.partition.columns = 'true' | ||
hoodie.partition.metafile.use.base.format = 'true' |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't see this option in the hudi docs so I removed it. Also we don't use this internally when running the sync tool. Anyone know what this may have been for?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yeah. each hudi partition has a metadata file called .hoodie_partition_metadata. this has to be in parquet format just for BQ sync purpose. for other query engines or metastores, we did not have this requirement.
So, this meta file will be just a text file in general. but if user prefers to sync to BQ, the recommendation is to enable this
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
few minor comments
hoodie.datasource.write.drop.partition.columns = 'true' | ||
hoodie.partition.metafile.use.base.format = 'true' |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yeah. each hudi partition has a metadata file called .hoodie_partition_metadata. this has to be in parquet format just for BQ sync purpose. for other query engines or metastores, we did not have this requirement.
So, this meta file will be just a text file in general. but if user prefers to sync to BQ, the recommendation is to enable this
Change Logs
Impact
Better docs
Risk level (write none, low medium or high below)
None
Documentation Update
Describe any necessary documentation update if there is any new feature, config, or user-facing change
ticket number here and follow the instruction to make
changes to the website.
Contributor's checklist