[#12682] improve(paimon): document optional Hadoop S3A support for Paimon catalog - #12683
Open
yuqi1129 wants to merge 3 commits into
Open
[#12682] improve(paimon): document optional Hadoop S3A support for Paimon catalog#12683yuqi1129 wants to merge 3 commits into
yuqi1129 wants to merge 3 commits into
Conversation
Code Coverage Report
|
yuqi1129
changed the title
yuqi1129
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Document Hadoop S3A as an optional filesystem for Paimon catalogs, including the required dependencies and configuration.
Add
hadoop-awsto the test runtime classpath only, and retain unit and LocalStack integration coverage for ans3a://Paimon filesystem catalog.Why are the changes needed?
Paimon native S3 support uses
s3://withpaimon-s3. Hadoop S3A is an optional Hadoop-compatible filesystem and requires version-alignedhadoop-awsandaws-java-sdk-bundleJARs in the Paimon catalog classpath.Packaging these dependencies by default would increase the Paimon catalog package by about 298 MB. Users who need
s3a://can install them explicitly instead.Fix: #12682
Does this PR introduce any user-facing change?
Yes. The Paimon catalog documentation now explains how to enable and configure optional Hadoop S3A support.
No API or property-key changes are introduced, and S3A dependencies are not included in the production runtime classpath.
How was this patch tested?
./gradlew --no-daemon :catalogs:catalog-lakehouse-paimon:test --tests org.apache.gravitino.catalog.lakehouse.paimon.TestPaimonCatalog.testS3AFileSystemIsAvailable --tests org.apache.gravitino.catalog.lakehouse.paimon.integration.test.CatalogPaimonS3AIT.testAccessS3AFileSystem -PskipDockerTests=false --rerun-tasksruntimeClasspathdoes not containhadoop-aws.testRuntimeClasspathcontainshadoop-aws:3.3.6andaws-java-sdk-bundle:1.12.367.