[SPARK-57794][BUILD] Remove hadoop-huaweicloud and its okhttp/okio dependencies - #57872
Open
zhangxinyao88 wants to merge 3 commits into
Open
[SPARK-57794][BUILD] Remove hadoop-huaweicloud and its okhttp/okio dependencies#57872zhangxinyao88 wants to merge 3 commits into
zhangxinyao88 wants to merge 3 commits into
Conversation
zhangxinyao88
force-pushed
the
remove-hadoop-huaweicloud
branch
from
August 8, 2026 17:24
031ccf8 to
d1d6733
Compare
zhangxinyao88
marked this pull request as ready for review
August 8, 2026 23:40
Author
|
Hi @anupamme — thanks for the work on #57779. I only found it after opening this PR. Full CI is green here, including the extra dependency-manifest and license cleanup I mentioned in the coordination thread. I wanted to loop you in first before involving other reviewers. Happy to follow whichever direction you think makes the most sense. |
|
Responded to PR #57779 |
Member
|
Thank you @zhangxinyao88 and @anupamme for coordinating. Shall we close the current PR at this point? |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Remove the direct
hadoop-huaweiclouddependency fromhadoop-cloud, together with itsexplicit OkHttp and Okio dependencies, version properties, dependency scope property, and
huaweicloud-providedprofile.Keep the
hadoop-huaweicloudexclusion onhadoop-cloud-storageso that the connector is notreintroduced transitively. Update
LICENSE-binaryand the Hadoop 3 dependency manifest toremove the three JARs.
Closes #57794.
Why are the changes needed?
Spark has no source code, tests, or documentation that use the Huawei OBS connector. Bundling
it pulls in the obsolete OkHttp 3.12.12 and Okio 1.17.6 dependency tree, including an OkHttp
version affected by CVE-2021-0341. Removing the unused connector eliminates that dependency
tree without changing Spark's supported default paths.
Does this PR introduce any user-facing change?
Yes. Spark distributions built with the Hadoop cloud module no longer bundle
hadoop-huaweicloud, OkHttp 3.12.12, or Okio 1.17.6. Spark APIs and documented/default cloudstorage paths are unchanged.
How was this patch tested?
No unit tests were added because this patch only removes Maven and binary-distribution metadata.
It was validated with:
Both effective POMs contain zero direct
hadoop-huaweicloud,okhttp, orokiodependencies, retain exactly one
hadoop-huaweicloudexclusion, and contain nohuaweicloud-providedprofile orhuaweicloud.deps.scopeproperty.Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex (codex-cli 0.146.0)