Skip to content

[SPARK-57794][BUILD] Remove hadoop-huaweicloud and its okhttp/okio dependencies - #57872

Open
zhangxinyao88 wants to merge 3 commits into
apache:masterfrom
zhangxinyao88:remove-hadoop-huaweicloud
Open

[SPARK-57794][BUILD] Remove hadoop-huaweicloud and its okhttp/okio dependencies#57872
zhangxinyao88 wants to merge 3 commits into
apache:masterfrom
zhangxinyao88:remove-hadoop-huaweicloud

Conversation

@zhangxinyao88

Copy link
Copy Markdown

What changes were proposed in this pull request?

Remove the direct hadoop-huaweicloud dependency from hadoop-cloud, together with its
explicit OkHttp and Okio dependencies, version properties, dependency scope property, and
huaweicloud-provided profile.

Keep the hadoop-huaweicloud exclusion on hadoop-cloud-storage so that the connector is not
reintroduced transitively. Update LICENSE-binary and the Hadoop 3 dependency manifest to
remove the three JARs.

Closes #57794.

Why are the changes needed?

Spark has no source code, tests, or documentation that use the Huawei OBS connector. Bundling
it pulls in the obsolete OkHttp 3.12.12 and Okio 1.17.6 dependency tree, including an OkHttp
version affected by CVE-2021-0341. Removing the unused connector eliminates that dependency
tree without changing Spark's supported default paths.

Does this PR introduce any user-facing change?

Yes. Spark distributions built with the Hadoop cloud module no longer bundle
hadoop-huaweicloud, OkHttp 3.12.12, or Okio 1.17.6. Spark APIs and documented/default cloud
storage paths are unchanged.

How was this patch tested?

No unit tests were added because this patch only removes Maven and binary-distribution metadata.
It was validated with:

./build/mvn -Phadoop-3,hadoop-cloud -pl hadoop-cloud help:effective-pom \
  -Doutput=/private/tmp/spark-hadoop-cloud-effective-pom.xml
./build/mvn -Phadoop-3,hadoop-cloud,hadoop-provided -pl hadoop-cloud help:effective-pom \
  -Doutput=/private/tmp/spark-hadoop-cloud-provided-effective-pom.xml -q
xmllint --noout pom.xml hadoop-cloud/pom.xml
LC_ALL=C sort -c dev/deps/spark-deps-hadoop-3-hive-2.3
git diff --check upstream/master

Both effective POMs contain zero direct hadoop-huaweicloud, okhttp, or okio
dependencies, retain exactly one hadoop-huaweicloud exclusion, and contain no
huaweicloud-provided profile or huaweicloud.deps.scope property.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: OpenAI Codex (codex-cli 0.146.0)

@zhangxinyao88
zhangxinyao88 force-pushed the remove-hadoop-huaweicloud branch from 031ccf8 to d1d6733 Compare August 8, 2026 17:24
@zhangxinyao88 zhangxinyao88 changed the title [BUILD] Remove hadoop-huaweicloud and its okhttp/okio dependencies [SPARK-57794][BUILD] Remove hadoop-huaweicloud and its okhttp/okio dependencies Aug 8, 2026
@zhangxinyao88
zhangxinyao88 marked this pull request as ready for review August 8, 2026 23:40
@zhangxinyao88

Copy link
Copy Markdown
Author

Hi @anupamme — thanks for the work on #57779. I only found it after opening this PR. Full CI is green here, including the extra dependency-manifest and license cleanup I mentioned in the coordination thread.

I wanted to loop you in first before involving other reviewers. Happy to follow whichever direction you think makes the most sense.

@anupamme

anupamme commented Aug 9, 2026

Copy link
Copy Markdown

Responded to PR #57779

@uros-b

uros-b commented Aug 9, 2026

Copy link
Copy Markdown
Member

Thank you @zhangxinyao88 and @anupamme for coordinating. Shall we close the current PR at this point?

@uros-b uros-b left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[SPARK] Remove hadoop-huaweicloud and its okhttp/okio dependencies

3 participants