-
Notifications
You must be signed in to change notification settings - Fork 2.4k
[HUDI-9528] Support database and table name for Glue/ Datahub catalog #13424
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
[HUDI-9528] Support database and table name for Glue/ Datahub catalog #13424
Conversation
Signed-off-by: Vineeth N. <37555814+vineethNaroju@users.noreply.github.com>
62c0823
to
505deb9
Compare
https://issues.apache.org/jira/browse/HUDI-9528 |
@hudi-bot run azure |
@vineethNaroju can you explain why we need a new options key for the db/table name even though the existing options already work? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
block to wait for some clarifications.
@danny0405 We support database/table names being different for other catalogs/metastores like BigQuery for example. The restriction for user right now is that for Glue/DataHub, it always gets created with |
@vinishjail97 I have no access to the link, I see there are already some options like
|
Yes, we want to have similar config for glue and datahub catalog. |
That's okay, can we add similiar inference logic just in the config option so that we only need to change the specific sync tool: public static final ConfigProperty<String> BIGQUERY_SYNC_TABLE_NAME = ConfigProperty
.key("hoodie.gcp.bigquery.sync.table_name")
.noDefaultValue()
.withInferFunction(cfg -> Option.ofNullable(cfg.getString(HOODIE_TABLE_NAME_KEY))
.or(() -> Option.ofNullable(cfg.getString(HOODIE_WRITE_TABLE_NAME_KEY))))
.markAdvanced()
.withDocumentation("Name of the target table in BigQuery"); |
Change Logs
Added separate configs for glue and datahub to set database/table name in sync client.
Impact
Hudi database/table name can be configured for glue/datahub catalog separately.
Risk level (write none, low medium or high below)
If medium or high, explain what verification was done to mitigate the risks.
Documentation Update
Contributor's checklist