[HUDI-5903]: Introduce Glue sync configs#8208
[HUDI-5903]: Introduce Glue sync configs#8208pratyakshsharma wants to merge 5 commits intoapache:masterfrom
Conversation
codope
added
component:catalog-sync
| .withDocumentation("AWS session token"); | ||
|
|
||
| public static final ConfigProperty<String> AWS_REGION = ConfigProperty | ||
| .key("hoodie.aws.region") |
There was a problem hiding this comment.
Should this config be specific to Glue ? How are we planning to use this config for pointing to S3 in different region ? Will that be a separate config ?
There was a problem hiding this comment.
The original intent of adding this config was only for glue. If S3 and glue need to be pointed to different regions, then we need a separate config for S3.
There was a problem hiding this comment.
Moved this to GlueSyncConfig class.
|
@hudi-bot run azure |
|
@bvaradar ping! |
|
@pratyakshsharma : Can you resolve the conflicts and I will look into this. |
|
@bvaradar I have resolved the conflicts, waiting for CI to pass. I have used these changelogs as reference for making all glue client related changes - https://github.com/aws/aws-sdk-java-v2/blob/master/docs/LaunchChangelog.md |
|
@bvaradar Can we review and merge this please? |
bvaradar
left a comment
bvaradar
left a comment
There was a problem hiding this comment.
@pratyakshsharma Otherwise looks ok. Can you also fix conflicts ?
| groupName = ConfigGroups.Names.META_SYNC, | ||
| subGroupName = ConfigGroups.SubGroupNames.NONE, | ||
| description = "Configs that control Glue catalog sync based client.") | ||
| public class GlueSyncConfig extends HiveSyncConfig { |
There was a problem hiding this comment.
@pratyakshsharma what is the reason for renaming this class from GlueCatalogSyncClientConfig to GlueSyncConfig ?
There was a problem hiding this comment.
I had introduced GlueSyncConfig in my initial commit, and as part of resolving conflicts later, looks like I deleted the original class by mistake. Will revert the name change.
Thank you for pointing this out.
|
@hudi-bot run azure |
5 participants
This PR is a good starting point for introducing glue sync specific configs. By default, AWS chooses the same region where other services like EMR or EC2 are running. There was no provision for selecting the AWS region so far. Please find more info here - https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/java-dg-region-selection.html.
Change Logs
Impact
Risk level (write none, low medium or high below)
None
Documentation Update
New configs need to be added. Tracking jira - https://issues.apache.org/jira/browse/HUDI-5946
Contributor's checklist