From bfe1af9a720ed235937a0fdf665376ffff7cce54 Mon Sep 17 00:00:00 2001 From: yangjie01 Date: Thu, 15 Dec 2022 00:39:44 -0800 Subject: [PATCH] [SPARK-41522][BUILD] Pin `versions-maven-plugin` to 2.13.0 to recover `test-dependencies.sh` ### What changes were proposed in this pull request? This pr aims to pin `versions-maven-plugin` to 2.13.0 to recover `test-dependencies.sh` and make GA pass , this pr should revert after we know how to use version 2.14.0. ### Why are the changes needed? `dev/test-dependencies.sh` always use latest `versions-maven-plugin` version, and `versions-maven-plugin` 2.14.0 has not set the version of the sub-module. Run: ``` build/mvn -q versions:set -DnewVersion=spark-928034 -DgenerateBackupPoms=false ``` **2.14.0** ``` + git status On branch test-ci Changes not staged for commit: (use "git add ..." to update what will be committed) (use "git restore ..." to discard changes in working directory) modified: assembly/pom.xml modified: core/pom.xml modified: examples/pom.xml modified: graphx/pom.xml modified: hadoop-cloud/pom.xml modified: launcher/pom.xml modified: mllib-local/pom.xml modified: mllib/pom.xml modified: pom.xml modified: repl/pom.xml modified: streaming/pom.xml modified: tools/pom.xml ``` **2.13.0** ``` + git status On branch test-ci Changes not staged for commit: (use "git add ..." to update what will be committed) (use "git restore ..." to discard changes in working directory) modified: assembly/pom.xml modified: common/kvstore/pom.xml modified: common/network-common/pom.xml modified: common/network-shuffle/pom.xml modified: common/network-yarn/pom.xml modified: common/sketch/pom.xml modified: common/tags/pom.xml modified: common/unsafe/pom.xml modified: connector/avro/pom.xml modified: connector/connect/common/pom.xml modified: connector/connect/server/pom.xml modified: connector/docker-integration-tests/pom.xml modified: connector/kafka-0-10-assembly/pom.xml modified: connector/kafka-0-10-sql/pom.xml modified: connector/kafka-0-10-token-provider/pom.xml modified: connector/kafka-0-10/pom.xml modified: connector/kinesis-asl-assembly/pom.xml modified: connector/kinesis-asl/pom.xml modified: connector/protobuf/pom.xml modified: connector/spark-ganglia-lgpl/pom.xml modified: core/pom.xml modified: dev/test-dependencies.sh modified: examples/pom.xml modified: graphx/pom.xml modified: hadoop-cloud/pom.xml modified: launcher/pom.xml modified: mllib-local/pom.xml modified: mllib/pom.xml modified: pom.xml modified: repl/pom.xml modified: resource-managers/kubernetes/core/pom.xml modified: resource-managers/kubernetes/integration-tests/pom.xml modified: resource-managers/mesos/pom.xml modified: resource-managers/yarn/pom.xml modified: sql/catalyst/pom.xml modified: sql/core/pom.xml modified: sql/hive-thriftserver/pom.xml modified: sql/hive/pom.xml modified: streaming/pom.xml modified: tools/pom.xml ``` Therefore, the following compilation error will occur when using 2.14.0. ``` 2022-12-15T02:37:35.5536924Z [ERROR] [ERROR] Some problems were encountered while processing the POMs: 2022-12-15T02:37:35.5538469Z [FATAL] Non-resolvable parent POM for org.apache.spark:spark-sketch_2.12:3.4.0-SNAPSHOT: Could not find artifact org.apache.spark:spark-parent_2.12:pom:3.4.0-SNAPSHOT and 'parent.relativePath' points at wrong local POM line 22, column 11 ``` ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Pass GitHub Actions Closes #39067 from LuciferYang/test-ci. Authored-by: yangjie01 Signed-off-by: Dongjoon Hyun --- dev/test-dependencies.sh | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/dev/test-dependencies.sh b/dev/test-dependencies.sh index 2268a262d5f8f..442ae58100973 100755 --- a/dev/test-dependencies.sh +++ b/dev/test-dependencies.sh @@ -76,11 +76,13 @@ function reset_version { find "$HOME/.m2/" | grep "$TEMP_VERSION" | xargs rm -rf # Restore the original version number: - $MVN -q versions:set -DnewVersion=$OLD_VERSION -DgenerateBackupPoms=false > /dev/null + # SPARK-41522: pin versions-maven-plugin version to recover test-dependencies.sh and make GA task pass. + $MVN -q org.codehaus.mojo:versions-maven-plugin:2.13.0:set -DnewVersion=$OLD_VERSION -DgenerateBackupPoms=false > /dev/null } trap reset_version EXIT -$MVN -q versions:set -DnewVersion=$TEMP_VERSION -DgenerateBackupPoms=false > /dev/null +# SPARK-41522: pin versions-maven-plugin version to recover test-dependencies.sh and make GA task pass. +$MVN -q org.codehaus.mojo:versions-maven-plugin:2.13.0:set -DnewVersion=$TEMP_VERSION -DgenerateBackupPoms=false > /dev/null # Generate manifests for each Hadoop profile: for HADOOP_HIVE_PROFILE in "${HADOOP_HIVE_PROFILES[@]}"; do