[HUDI-7005] Fix hudi-aws-bundle relocation issue with avro #9946
Conversation
Signed-off-by: Prabhu Joseph <joprabhu@amazon.com>
PrabhuJoseph
changed the title
|
@danny0405 Could you review this patch when you get time. Thanks. |
|
@umehrot2 has reviewed the patch and found a problem in relocating avro classes in the hudi-aws-bundle. It will affect the hudi-spark bundle as which has not relocated avro. The right fix would be to remove including hudi-common classes from hudi-aws-bundle and hudi-aws classes from hudi-flink-bundle. I will make a new revision. Thanks. |
| @@ -84,7 +84,6 @@ | |||
| <include>org.apache.hudi:hudi-sync-common</include> | |||
| <include>org.apache.hudi:hudi-hadoop-mr</include> | |||
| <include>org.apache.hudi:hudi-timeline-service</include> | |||
| <include>org.apache.hudi:hudi-aws</include> | |||
|
|
|||
There was a problem hiding this comment.
Flink code HiveSyncContext is dependent on AwsGlueCatalogSyncTool.
There was a problem hiding this comment.
Good Point. The idea is to have hudi-aws classes in the hudi-aws bundle. hudi-flink-bundle won't have any hudi-aws classes to avoid the below issue. We will ask users to place both hudi-flink-bundle and hudi-aws-bundle in the Flink/lib. Do you have any concerns with this approach?
Issue when hudi-flink-bundle includes hudi-aws
2023-10-07 14:47:03,463 ERROR org.apache.hudi.sink.StreamWriteOperatorCoordinator [] - Executor executes action [sync hive metadata for instant 20231007144701183] error
java.lang.NoClassDefFoundError: software/amazon/awssdk/services/glue/model/EntityNotFoundException
at org.apache.hudi.aws.sync.AwsGlueCatalogSyncTool.initSyncClient(AwsGlueCatalogSyncTool.java:52) ~[hudi-flink-bundle.jar:0.13.1-amzn-1]
at org.apache.hudi.hive.HiveSyncTool.<init>(HiveSyncTool.java:114) ~[hudi-flink-bundle.jar:0.13.1-amzn-1]
This issue happens as AwsGlueCatalogSyncTool (hudi-aws module) part of hudi-flink-bundle has not relocated AWS SDK classes whereas the hudi-aws-bundle (which includes AWS SDK) has relocated it.
There was a problem hiding this comment.
Then what is the suggested way to load the AwsGlueCatalogSyncTool, maybe by reflection?
There was a problem hiding this comment.
Yes, I also think reflection will be a better choice.
- If the user configures "hive_sync.mode" to "hms", then only hudi-flink-bundle is needed.
- If the user configures "hive_sync.mode" to "glue", then both hudi-flink-bundle and hudi-aws-bundle is needed. This needs to be documented here.
I see ReflectionUtils and SyncUtilHelpers have provided ways to easily load the class. I will use them and submit a new revision if you are also fine with this discussed approach.
There was a problem hiding this comment.
The fix is to remove including hudi-aws from hudi-flink-bundle. hudi-flink-bundle need not bring hudi-aws classes as hudi-aws-bundle jar can be used instead at runtime.
Is this the recommended or standard style how engine bundle jar should be used when AWS related function is turned on? cc @umehrot2
There was a problem hiding this comment.
Yes, the hudi-aws-bundle needs to be used to access the features present in hudi-aws and its aws dependencies.
There was a problem hiding this comment.
Fine with it, please keep @umehrot2 as noted.
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/utils/HiveSyncContext.java
Outdated
Show resolved
Hide resolved
|
Thanks, @danny0405, for the review and commit. |
Change Logs
Relocates org.apache.avro to org.apache.hudi.org.apache.avro while shading hudi-common classes in hudi-hive-sync-bundle and hudi-aws-bundle to avoid conflicts while working with hudi-flink-bundle and any other bundle where it is relocated already.
Impact
Flink SQL Client Fixes.
Risk level (write none, low medium or high below)
low
Documentation Update
NA
Contributor's checklist