fix(bundle): relocate the AWS SDK bundled into hudi-utilities-bundle - #19425
fix(bundle): relocate the AWS SDK bundled into hudi-utilities-bundle#19425rahil-c wants to merge 2 commits into
Conversation
JsonKinesisSource depends on software.amazon.awssdk:kinesis and com.amazonaws:amazon-kinesis-deaggregator, but hudi-utilities-bundle's shade config never included either, so the published bundle jar doesn't carry the classes JsonKinesisSource needs at runtime. hudi-aws-bundle already solves this exact problem for its own AWS dependencies: it includes software.amazon.awssdk:* and relocates it to org.apache.hudi.software.amazon.awssdk, avoiding classpath collisions with whatever SDK version a consuming application pins. Apply the same include + relocation to hudi-utilities-bundle so JsonKinesisSource works when the bundle is deployed on its own. Verified locally: `mvn -pl packaging/hudi-utilities-bundle -am package` succeeds, and the resulting jar has zero classes at the original software/amazon/awssdk/** paths (fully relocated under org/apache/hudi/software/amazon/awssdk/**), including the relocated META-INF/services/*SdkHttpService entries needed for the SDK's ServiceLoader-based HTTP client discovery.
rahil-c
changed the title
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #19425 +/- ##
============================================
- Coverage 75.56% 75.55% -0.01%
- Complexity 32651 32662 +11
============================================
Files 2574 2574
Lines 142995 142995
Branches 17530 17530
============================================
- Hits 108051 108045 -6
- Misses 26908 26915 +7
+ Partials 8036 8035 -1
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
hudi-agent
left a comment
hudi-agent
left a comment
There was a problem hiding this comment.
Thanks for working on this! This PR shades the AWS SDK v2 classes (relocated under org.apache.hudi.software.amazon.awssdk.**) plus the Kinesis deaggregator into hudi-utilities-bundle so JsonKinesisSource can run standalone, mirroring hudi-aws-bundle. The relocation of software.amazon.awssdk.* looks consistent with the source imports, but there's one bundling gap worth double-checking on the built artifact (see inline). Please take a look at the inline comment, and this should be ready for a Hudi committer or PMC member to take it from here.
| <!-- AWS SDK for JsonKinesisSource; relocated below, matching hudi-aws-bundle. --> | ||
| <include>software.amazon.awssdk:*</include> | ||
| <!-- KPL de-aggregation: extracts user records from Kinesis Producer Library aggregated records --> | ||
| <include>com.amazonaws:amazon-kinesis-deaggregator</include> |
There was a problem hiding this comment.
🤖 This includes amazon-kinesis-deaggregator but not com.amazonaws:amazon-kinesis-client. KinesisDeaggregator imports com.amazonaws.services.kinesis.clientlibrary.types.UserRecord, which lives in amazon-kinesis-client (pulled in transitively at 1.8.8), not in the deaggregator artifact — so the standalone bundle would still hit NoClassDefFoundError on UserRecord. Note hudi-aws-bundle includes both amazon-kinesis-deaggregator and amazon-kinesis-client. Could you add the amazon-kinesis-client include here to match?
software.amazon.awssdk:* also pulled in s3, dynamodb, glue, cloudwatch, and their protocol/support artifacts -- none of which JsonKinesisSource uses. Those come from other AWS-dependent code in hudi-utilities' dependency graph, not from Kinesis. Since that code isn't part of this bundle, the extra SDK surface was dead weight: ~17.7MB with nothing in the shaded jar actually calling it. Replaced the wildcard with the exact kinesis+sts transitive closure, verified via `mvn dependency:tree -Dverbose`: the shared AWS SDK v2 core (sdk-core, aws-core, auth, regions, etc.), kinesis's own protocol dependencies (aws-cbor-protocol, third-party-jackson-*), sts's (aws-query-protocol, profiles), and org.reactivestreams, which sdk-core needs and isn't in the software.amazon.awssdk group. Verified locally: the resulting jar has zero classes from s3/dynamodb/glue/cloudwatch/sqs/arns/aws-xml-protocol/crt-core, is still fully self-resolving (zero unrelocated software/amazon/awssdk/** classes), and JsonKinesisSource's bytecode still resolves to the relocated KinesisClient.
github-actions
Bot
added
size:S
hudi-agent
left a comment
hudi-agent
left a comment
There was a problem hiding this comment.
Thanks for the update! The reworked include list replaces the software.amazon.awssdk:* wildcard with an explicit set of AWS SDK v2 modules (core, kinesis, sts, and their protocol/transport dependencies), which is a reasonable way to keep the shaded set tight. One prior finding still looks open, though: the bundle adds com.amazonaws:amazon-kinesis-deaggregator but not com.amazonaws:amazon-kinesis-client, and KinesisDeaggregator still imports UserRecord from the latter — see the inline comment. Also worth double-checking that the now-explicit v2 include list is complete against the dependency tree so nothing needed gets dropped from the shaded jar. Please take a look at the inline comment, and this should be ready for a Hudi committer or PMC member to take it from here.
| <include>software.amazon.awssdk:aws-query-protocol</include> | ||
| <include>software.amazon.awssdk:profiles</include> | ||
| <!-- KPL de-aggregation: extracts user records from Kinesis Producer Library aggregated records --> | ||
| <include>com.amazonaws:amazon-kinesis-deaggregator</include> |
There was a problem hiding this comment.
🤖 The reworked list still only adds amazon-kinesis-deaggregator, not com.amazonaws:amazon-kinesis-client. KinesisDeaggregator imports com.amazonaws.services.kinesis.clientlibrary.types.UserRecord, which lives in amazon-kinesis-client (not the deaggregator jar), so the standalone bundle would still hit NoClassDefFoundError on UserRecord. hudi-aws-bundle includes both — could you add <include>com.amazonaws:amazon-kinesis-client</include> here to match?
4 participants
Change Logs
Summary.
hudi-utilities-bundledoesn't shade the AWS SDK thatJsonKinesisSourceneeds, so the published bundle jar can't actually runJsonKinesisSourceon its own.hudi-utilitiesdepends onsoftware.amazon.awssdk:kinesisandcom.amazonaws:amazon-kinesis-deaggregatorforJsonKinesisSource/KinesisOffsetGen/KinesisDeaggregator, butpackaging/hudi-utilities-bundle/pom.xml's shade includes never picked either up. Neither artifact ends up in the shaded jar, so a deployment running onlyhudi-utilities-bundlehitsNoClassDefFoundErroron anysoftware.amazon.awssdk.services.kinesis.*class the momentJsonKinesisSourceis used.hudi-aws-bundlealready relocates its AWS SDK dependencies the same way this PR relocateshudi-utilities-bundle's: moving thesoftware.amazon.awssdkpackage prefix toorg.apache.hudi.software.amazon.awssdk, avoiding a classpath collision with whatever SDK version a consuming application has pinned on its own classpath.Include list is scoped to kinesis/sts, not a wildcard.
kinesis/stsaren't self-contained — they're thin client artifacts that depend on a shared set of AWS SDK v2 core libraries to actually load:KinesisClientimplementsAwsClient, which lives in the separateaws-coreartifact; both needsdk-corefor request handling,authfor signing,regions,http-client-spiplusapache-client/netty-nio-clientfor transport, and so on down the chain. None of that is optional — omit any of it and the class fails to load at runtime.hudi-utilitiesalso depends onhudi-aws(for Glue sync, the DynamoDB lock provider, and CloudWatch metrics elsewhere inhudi-utilities), which transitively resolvess3,dynamodb,glue,cloudwatch, and their protocol/support artifacts. Asoftware.amazon.awssdk:*wildcard include would sweep those into this bundle too, even thoughJsonKinesisSourcenever touches them andhudi-aws's own classes aren't part of this bundle — so they'd just be dead weight (measured at ~17.7MB on the built jar).Instead, the include list enumerates exactly the
kinesis+ststransitive closure, traced withmvn dependency:tree -Dverboseagainsthudi-utilities: the shared AWS SDK v2 core (sdk-core,aws-core,auth,regions,http-client-spi,apache-client,netty-nio-client,checksums,retries, and their-spisiblings),kinesis's own protocol dependencies (aws-cbor-protocol,third-party-jackson-dataformat-cborand-core),sts's (aws-query-protocol,profiles), andorg.reactivestreams:reactive-streams, whichsdk-coreneeds and isn't in thesoftware.amazon.awssdkgroup (so a group-scoped wildcard wouldn't have covered it either).Impact
hudi-utilities-bundlenow carries thekinesis/stsdependency closureJsonKinesisSourceneeds, relocated underorg.apache.hudi.software.amazon.awssdk.**. No behavior change for anything not usingJsonKinesisSource.Risk level
Low. Additive to the bundle's shaded contents; no existing classes move or change behavior.
The one thing worth double-checking on the built artifact is AWS SDK v2's
ServiceLoader-based HTTP client discovery, sinceKinesisOffsetGen.createKinesisClientbuilds aKinesisClientwithout an explicit.httpClient(...). Verified locally that the relocated jar carries the rewritten service file,META-INF/services/org.apache.hudi.software.amazon.awssdk.http.SdkHttpService, containingorg.apache.hudi.software.amazon.awssdk.http.apache.ApacheSdkHttpService.Documentation Update
None. No configs, public API, or user-facing behavior change.
Contributor's checklist
mvn -pl packaging/hudi-utilities-bundle -am package -DskipTests -Dscala-2.12 -Dspark3.5 -Dflink1.20succeeds locally. Verified the resulting jar: zero classes remain at the originalsoftware/amazon/awssdk/**paths; the relocatedSdkHttpServiceservices file is present;s3/dynamodb/glue/cloudwatch/sqs/arns/aws-xml-protocol/crt-coreare all absent; andJsonKinesisSource's own bytecode resolves to the relocatedKinesisClient.