[SPARK-56696][BUILD] Exclude old junit:junit from the dependency tree for sbt build#55645
Closed
sarutak wants to merge 2 commits intoapache:masterfrom
Closed
[SPARK-56696][BUILD] Exclude old junit:junit from the dependency tree for sbt build#55645sarutak wants to merge 2 commits intoapache:masterfrom
junit:junit from the dependency tree for sbt build#55645sarutak wants to merge 2 commits intoapache:masterfrom
Conversation
…cal as a workaround" This reverts commit a4ebe27.
pan3793
approved these changes
May 2, 2026
HyukjinKwon
approved these changes
May 4, 2026
Member
|
Merged to master, branch-4.x and branch-4.2. |
HyukjinKwon
pushed a commit
that referenced
this pull request
May 4, 2026
…ee for sbt build ### What changes were proposed in this pull request? This PR excludes old `junit:junit` from the dependency tree for sbt build. This is also another solution for SPARK-56478 (#55358). ### Why are the changes needed? Currently, we use `org.junit.jupiter:junit-jupiter` rather than `junit:junit`, and `junit:junit` is explicitly excluded in `pom.xml`. https://github.com/apache/spark/blob/8e37824402531f82ad1dfef415b5c29b478df760/pom.xml#L1519 But the dependency tree for sbt build still includes `junit:junit`. ``` $ build/sbt Test/dependencyTree ... [info] | +-org.apache.parquet:parquet-column:1.17.0 [info] | | +-com.carrotsearch:junit-benchmarks:0.7.2 [info] | | +-junit:junit:4.13.2 [info] | | | +-org.hamcrest:hamcrest-core:1.3 [info] | | | [info] | | +-org.apache.commons:commons-lang3:3.17.0 (evicted by: 3.20.0) [info] | | +-org.apache.commons:commons-lang3:3.20.0 [info] | | +-org.apache.parquet:parquet-common:1.17.0 [info] | | | +-junit:junit:4.13.2 [info] | | | | +-org.hamcrest:hamcrest-core:1.3 ``` `junit:junit` also depends on `org.hamcrest:hamcrest-core:1.3`, which is causes an issue. If, we build spark using Maven with `-Psparkr`, the pom file for `hamcrest-core:1.3` will be downloaded but the corresponding jar file will not. ``` $ build/mvn -DskipTests -Psparkr package $ ls ~/.m2/repository/org/hamcrest/hamcrest-core/1.3/ _remote.repositories hamcrest-core-1.3.pom hamcrest-core-1.3.pom.lastUpdated hamcrest-core-1.3.pom.sha1 ``` If we then build Spark using sbt in this situation, it will fail. ``` $ build/sbt package ... $ build/sbt package ... [error] lmcoursier.internal.shaded.coursier.error.FetchError$DownloadingArtifacts: Error fetching artifacts: [error] file:/home/release/.m2/repository/org/hamcrest/hamcrest-core/1.3/hamcrest-core-1.3.jar: not found: /home/release/. m2/repository/org/hamcrest/hamcrest-core/1.3/hamcrest-core-1.3.jar ``` For the same reason, building Scala/Java API document will fail. ``` $ cd docs $ SKIP_RDOC=1 SKIP_PYTHONDOC=1 SKIP_SQLDOC=1 bundler exec jekyll build ... [error] lmcoursier.internal.shaded.coursier.error.FetchError$DownloadingArtifacts: Error fetching artifacts: [error] file:/home/release/.m2/repository/org/hamcrest/hamcrest-core/1.3/hamcrest-core-1.3.jar: not found: /home/release/. m2/repository/org/hamcrest/hamcrest-core/1.3/hamcrest-core-1.3.jar ``` ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Confirmed both of the following commands succeeded ``` $ build/mvn -DskipTests -Psparkr package $ build/sbt package $ cd docs && SKIP_RDOC=1 SKIP_PYTHONDOC=1 SKIP_SQLDOC=1 bundler exec jekyll build ``` ### Was this patch authored or co-authored using generative AI tooling? No. Closes #55645 from sarutak/exclude-old-junit. Authored-by: Kousuke Saruta <sarutak@apache.org> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org> (cherry picked from commit 74ac7f4) Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
HyukjinKwon
pushed a commit
that referenced
this pull request
May 4, 2026
…ee for sbt build ### What changes were proposed in this pull request? This PR excludes old `junit:junit` from the dependency tree for sbt build. This is also another solution for SPARK-56478 (#55358). ### Why are the changes needed? Currently, we use `org.junit.jupiter:junit-jupiter` rather than `junit:junit`, and `junit:junit` is explicitly excluded in `pom.xml`. https://github.com/apache/spark/blob/8e37824402531f82ad1dfef415b5c29b478df760/pom.xml#L1519 But the dependency tree for sbt build still includes `junit:junit`. ``` $ build/sbt Test/dependencyTree ... [info] | +-org.apache.parquet:parquet-column:1.17.0 [info] | | +-com.carrotsearch:junit-benchmarks:0.7.2 [info] | | +-junit:junit:4.13.2 [info] | | | +-org.hamcrest:hamcrest-core:1.3 [info] | | | [info] | | +-org.apache.commons:commons-lang3:3.17.0 (evicted by: 3.20.0) [info] | | +-org.apache.commons:commons-lang3:3.20.0 [info] | | +-org.apache.parquet:parquet-common:1.17.0 [info] | | | +-junit:junit:4.13.2 [info] | | | | +-org.hamcrest:hamcrest-core:1.3 ``` `junit:junit` also depends on `org.hamcrest:hamcrest-core:1.3`, which is causes an issue. If, we build spark using Maven with `-Psparkr`, the pom file for `hamcrest-core:1.3` will be downloaded but the corresponding jar file will not. ``` $ build/mvn -DskipTests -Psparkr package $ ls ~/.m2/repository/org/hamcrest/hamcrest-core/1.3/ _remote.repositories hamcrest-core-1.3.pom hamcrest-core-1.3.pom.lastUpdated hamcrest-core-1.3.pom.sha1 ``` If we then build Spark using sbt in this situation, it will fail. ``` $ build/sbt package ... $ build/sbt package ... [error] lmcoursier.internal.shaded.coursier.error.FetchError$DownloadingArtifacts: Error fetching artifacts: [error] file:/home/release/.m2/repository/org/hamcrest/hamcrest-core/1.3/hamcrest-core-1.3.jar: not found: /home/release/. m2/repository/org/hamcrest/hamcrest-core/1.3/hamcrest-core-1.3.jar ``` For the same reason, building Scala/Java API document will fail. ``` $ cd docs $ SKIP_RDOC=1 SKIP_PYTHONDOC=1 SKIP_SQLDOC=1 bundler exec jekyll build ... [error] lmcoursier.internal.shaded.coursier.error.FetchError$DownloadingArtifacts: Error fetching artifacts: [error] file:/home/release/.m2/repository/org/hamcrest/hamcrest-core/1.3/hamcrest-core-1.3.jar: not found: /home/release/. m2/repository/org/hamcrest/hamcrest-core/1.3/hamcrest-core-1.3.jar ``` ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Confirmed both of the following commands succeeded ``` $ build/mvn -DskipTests -Psparkr package $ build/sbt package $ cd docs && SKIP_RDOC=1 SKIP_PYTHONDOC=1 SKIP_SQLDOC=1 bundler exec jekyll build ``` ### Was this patch authored or co-authored using generative AI tooling? No. Closes #55645 from sarutak/exclude-old-junit. Authored-by: Kousuke Saruta <sarutak@apache.org> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org> (cherry picked from commit 74ac7f4) Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
This PR excludes old
junit:junitfrom the dependency tree for sbt build.This is also another solution for SPARK-56478 (#55358).
Why are the changes needed?
Currently, we use
org.junit.jupiter:junit-jupiterrather thanjunit:junit, andjunit:junitis explicitly excluded inpom.xml.spark/pom.xml
Line 1519 in 8e37824
But the dependency tree for sbt build still includes
junit:junit.junit:junitalso depends onorg.hamcrest:hamcrest-core:1.3, which is causes an issue. If, we build spark using Maven with-Psparkr, the pom file forhamcrest-core:1.3will be downloaded but the corresponding jar file will not.If we then build Spark using sbt in this situation, it will fail.
For the same reason, building Scala/Java API document will fail.
Does this PR introduce any user-facing change?
No.
How was this patch tested?
Confirmed both of the following commands succeeded
Was this patch authored or co-authored using generative AI tooling?
No.