[SPARK-56478][INFRA] Remove hamcrest-core-1.3 in the Maven local as a workaround#55358
Closed
HyukjinKwon wants to merge 1 commit intoapache:masterfrom
Closed
[SPARK-56478][INFRA] Remove hamcrest-core-1.3 in the Maven local as a workaround#55358HyukjinKwon wants to merge 1 commit intoapache:masterfrom
HyukjinKwon wants to merge 1 commit intoapache:masterfrom
Conversation
Member
|
Thank you for mornitering and working on this~ |
gatorsmile
approved these changes
Apr 15, 2026
Member
Author
|
Merged to master. |
HyukjinKwon
pushed a commit
that referenced
this pull request
May 4, 2026
…ee for sbt build ### What changes were proposed in this pull request? This PR excludes old `junit:junit` from the dependency tree for sbt build. This is also another solution for SPARK-56478 (#55358). ### Why are the changes needed? Currently, we use `org.junit.jupiter:junit-jupiter` rather than `junit:junit`, and `junit:junit` is explicitly excluded in `pom.xml`. https://github.com/apache/spark/blob/8e37824402531f82ad1dfef415b5c29b478df760/pom.xml#L1519 But the dependency tree for sbt build still includes `junit:junit`. ``` $ build/sbt Test/dependencyTree ... [info] | +-org.apache.parquet:parquet-column:1.17.0 [info] | | +-com.carrotsearch:junit-benchmarks:0.7.2 [info] | | +-junit:junit:4.13.2 [info] | | | +-org.hamcrest:hamcrest-core:1.3 [info] | | | [info] | | +-org.apache.commons:commons-lang3:3.17.0 (evicted by: 3.20.0) [info] | | +-org.apache.commons:commons-lang3:3.20.0 [info] | | +-org.apache.parquet:parquet-common:1.17.0 [info] | | | +-junit:junit:4.13.2 [info] | | | | +-org.hamcrest:hamcrest-core:1.3 ``` `junit:junit` also depends on `org.hamcrest:hamcrest-core:1.3`, which is causes an issue. If, we build spark using Maven with `-Psparkr`, the pom file for `hamcrest-core:1.3` will be downloaded but the corresponding jar file will not. ``` $ build/mvn -DskipTests -Psparkr package $ ls ~/.m2/repository/org/hamcrest/hamcrest-core/1.3/ _remote.repositories hamcrest-core-1.3.pom hamcrest-core-1.3.pom.lastUpdated hamcrest-core-1.3.pom.sha1 ``` If we then build Spark using sbt in this situation, it will fail. ``` $ build/sbt package ... $ build/sbt package ... [error] lmcoursier.internal.shaded.coursier.error.FetchError$DownloadingArtifacts: Error fetching artifacts: [error] file:/home/release/.m2/repository/org/hamcrest/hamcrest-core/1.3/hamcrest-core-1.3.jar: not found: /home/release/. m2/repository/org/hamcrest/hamcrest-core/1.3/hamcrest-core-1.3.jar ``` For the same reason, building Scala/Java API document will fail. ``` $ cd docs $ SKIP_RDOC=1 SKIP_PYTHONDOC=1 SKIP_SQLDOC=1 bundler exec jekyll build ... [error] lmcoursier.internal.shaded.coursier.error.FetchError$DownloadingArtifacts: Error fetching artifacts: [error] file:/home/release/.m2/repository/org/hamcrest/hamcrest-core/1.3/hamcrest-core-1.3.jar: not found: /home/release/. m2/repository/org/hamcrest/hamcrest-core/1.3/hamcrest-core-1.3.jar ``` ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Confirmed both of the following commands succeeded ``` $ build/mvn -DskipTests -Psparkr package $ build/sbt package $ cd docs && SKIP_RDOC=1 SKIP_PYTHONDOC=1 SKIP_SQLDOC=1 bundler exec jekyll build ``` ### Was this patch authored or co-authored using generative AI tooling? No. Closes #55645 from sarutak/exclude-old-junit. Authored-by: Kousuke Saruta <sarutak@apache.org> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
HyukjinKwon
pushed a commit
that referenced
this pull request
May 4, 2026
…ee for sbt build ### What changes were proposed in this pull request? This PR excludes old `junit:junit` from the dependency tree for sbt build. This is also another solution for SPARK-56478 (#55358). ### Why are the changes needed? Currently, we use `org.junit.jupiter:junit-jupiter` rather than `junit:junit`, and `junit:junit` is explicitly excluded in `pom.xml`. https://github.com/apache/spark/blob/8e37824402531f82ad1dfef415b5c29b478df760/pom.xml#L1519 But the dependency tree for sbt build still includes `junit:junit`. ``` $ build/sbt Test/dependencyTree ... [info] | +-org.apache.parquet:parquet-column:1.17.0 [info] | | +-com.carrotsearch:junit-benchmarks:0.7.2 [info] | | +-junit:junit:4.13.2 [info] | | | +-org.hamcrest:hamcrest-core:1.3 [info] | | | [info] | | +-org.apache.commons:commons-lang3:3.17.0 (evicted by: 3.20.0) [info] | | +-org.apache.commons:commons-lang3:3.20.0 [info] | | +-org.apache.parquet:parquet-common:1.17.0 [info] | | | +-junit:junit:4.13.2 [info] | | | | +-org.hamcrest:hamcrest-core:1.3 ``` `junit:junit` also depends on `org.hamcrest:hamcrest-core:1.3`, which is causes an issue. If, we build spark using Maven with `-Psparkr`, the pom file for `hamcrest-core:1.3` will be downloaded but the corresponding jar file will not. ``` $ build/mvn -DskipTests -Psparkr package $ ls ~/.m2/repository/org/hamcrest/hamcrest-core/1.3/ _remote.repositories hamcrest-core-1.3.pom hamcrest-core-1.3.pom.lastUpdated hamcrest-core-1.3.pom.sha1 ``` If we then build Spark using sbt in this situation, it will fail. ``` $ build/sbt package ... $ build/sbt package ... [error] lmcoursier.internal.shaded.coursier.error.FetchError$DownloadingArtifacts: Error fetching artifacts: [error] file:/home/release/.m2/repository/org/hamcrest/hamcrest-core/1.3/hamcrest-core-1.3.jar: not found: /home/release/. m2/repository/org/hamcrest/hamcrest-core/1.3/hamcrest-core-1.3.jar ``` For the same reason, building Scala/Java API document will fail. ``` $ cd docs $ SKIP_RDOC=1 SKIP_PYTHONDOC=1 SKIP_SQLDOC=1 bundler exec jekyll build ... [error] lmcoursier.internal.shaded.coursier.error.FetchError$DownloadingArtifacts: Error fetching artifacts: [error] file:/home/release/.m2/repository/org/hamcrest/hamcrest-core/1.3/hamcrest-core-1.3.jar: not found: /home/release/. m2/repository/org/hamcrest/hamcrest-core/1.3/hamcrest-core-1.3.jar ``` ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Confirmed both of the following commands succeeded ``` $ build/mvn -DskipTests -Psparkr package $ build/sbt package $ cd docs && SKIP_RDOC=1 SKIP_PYTHONDOC=1 SKIP_SQLDOC=1 bundler exec jekyll build ``` ### Was this patch authored or co-authored using generative AI tooling? No. Closes #55645 from sarutak/exclude-old-junit. Authored-by: Kousuke Saruta <sarutak@apache.org> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org> (cherry picked from commit 74ac7f4) Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
HyukjinKwon
pushed a commit
that referenced
this pull request
May 4, 2026
…ee for sbt build ### What changes were proposed in this pull request? This PR excludes old `junit:junit` from the dependency tree for sbt build. This is also another solution for SPARK-56478 (#55358). ### Why are the changes needed? Currently, we use `org.junit.jupiter:junit-jupiter` rather than `junit:junit`, and `junit:junit` is explicitly excluded in `pom.xml`. https://github.com/apache/spark/blob/8e37824402531f82ad1dfef415b5c29b478df760/pom.xml#L1519 But the dependency tree for sbt build still includes `junit:junit`. ``` $ build/sbt Test/dependencyTree ... [info] | +-org.apache.parquet:parquet-column:1.17.0 [info] | | +-com.carrotsearch:junit-benchmarks:0.7.2 [info] | | +-junit:junit:4.13.2 [info] | | | +-org.hamcrest:hamcrest-core:1.3 [info] | | | [info] | | +-org.apache.commons:commons-lang3:3.17.0 (evicted by: 3.20.0) [info] | | +-org.apache.commons:commons-lang3:3.20.0 [info] | | +-org.apache.parquet:parquet-common:1.17.0 [info] | | | +-junit:junit:4.13.2 [info] | | | | +-org.hamcrest:hamcrest-core:1.3 ``` `junit:junit` also depends on `org.hamcrest:hamcrest-core:1.3`, which is causes an issue. If, we build spark using Maven with `-Psparkr`, the pom file for `hamcrest-core:1.3` will be downloaded but the corresponding jar file will not. ``` $ build/mvn -DskipTests -Psparkr package $ ls ~/.m2/repository/org/hamcrest/hamcrest-core/1.3/ _remote.repositories hamcrest-core-1.3.pom hamcrest-core-1.3.pom.lastUpdated hamcrest-core-1.3.pom.sha1 ``` If we then build Spark using sbt in this situation, it will fail. ``` $ build/sbt package ... $ build/sbt package ... [error] lmcoursier.internal.shaded.coursier.error.FetchError$DownloadingArtifacts: Error fetching artifacts: [error] file:/home/release/.m2/repository/org/hamcrest/hamcrest-core/1.3/hamcrest-core-1.3.jar: not found: /home/release/. m2/repository/org/hamcrest/hamcrest-core/1.3/hamcrest-core-1.3.jar ``` For the same reason, building Scala/Java API document will fail. ``` $ cd docs $ SKIP_RDOC=1 SKIP_PYTHONDOC=1 SKIP_SQLDOC=1 bundler exec jekyll build ... [error] lmcoursier.internal.shaded.coursier.error.FetchError$DownloadingArtifacts: Error fetching artifacts: [error] file:/home/release/.m2/repository/org/hamcrest/hamcrest-core/1.3/hamcrest-core-1.3.jar: not found: /home/release/. m2/repository/org/hamcrest/hamcrest-core/1.3/hamcrest-core-1.3.jar ``` ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Confirmed both of the following commands succeeded ``` $ build/mvn -DskipTests -Psparkr package $ build/sbt package $ cd docs && SKIP_RDOC=1 SKIP_PYTHONDOC=1 SKIP_SQLDOC=1 bundler exec jekyll build ``` ### Was this patch authored or co-authored using generative AI tooling? No. Closes #55645 from sarutak/exclude-old-junit. Authored-by: Kousuke Saruta <sarutak@apache.org> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org> (cherry picked from commit 74ac7f4) Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Retry of f37a0d9
Why are the changes needed?
Retry of f37a0d9
Does this PR introduce any user-facing change?
No
How was this patch tested?
Will monitor the build.
Was this patch authored or co-authored using generative AI tooling?
No.