[SPARK-31713][INFRA] Make test-dependencies.sh detect version string correctly #28532

dongjoon-hyun · 2020-05-14T21:46:25Z

What changes were proposed in this pull request?

This PR makes test-dependencies.sh detect the version string correctly by ignoring all the other lines.

Why are the changes needed?

Currently, all SBT jobs are broken like the following.

https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-branch-3.0-test-sbt-hadoop-3.2-hive-2.3/476/console

[error] running /home/jenkins/workspace/spark-branch-3.0-test-sbt-hadoop-3.2-hive-2.3/dev/test-dependencies.sh ; received return code 1
Build step 'Execute shell' marked build as failure

The reason is that the script detects the old version like Falling back to archive.apache.org to download Maven 3.1.0-SNAPSHOT when build/mvn did fallback.

Specifically, in the script, OLD_VERSION became Falling back to archive.apache.org to download Maven 3.1.0-SNAPSHOT instead of 3.1.0-SNAPSHOT if build/mvn did fallback. Then, pom.xml file is corrupted like the following at the end and the exit code become 1 instead of 0. It causes Jenkins jobs fails

-    <version>3.1.0-SNAPSHOT</version>
+    <version>Falling</version>

NO FALLBACK

$ build/mvn -q -Dexec.executable="echo" -Dexec.args='${project.version}' --non-recursive org.codehaus.mojo:exec-maven-plugin:1.6.0:exec
Using `mvn` from path: /Users/dongjoon/APACHE/spark-merge/build/apache-maven-3.6.3/bin/mvn
3.1.0-SNAPSHOT

FALLBACK

$ build/mvn -q -Dexec.executable="echo" -Dexec.args='${project.version}' --non-recursive org.codehaus.mojo:exec-maven-plugin:1.6.0:exec
Falling back to archive.apache.org to download Maven
Using `mvn` from path: /Users/dongjoon/APACHE/spark-merge/build/apache-maven-3.6.3/bin/mvn
3.1.0-SNAPSHOT

In the script

$ echo $(build/mvn -q -Dexec.executable="echo" -Dexec.args='${project.version}' --non-recursive org.codehaus.mojo:exec-maven-plugin:1.6.0:exec)
Using `mvn` from path: /Users/dongjoon/APACHE/spark-merge/build/apache-maven-3.6.3/bin/mvn
Falling back to archive.apache.org to download Maven 3.1.0-SNAPSHOT

This PR will prevent irrelevant logs like Falling back to archive.apache.org to download Maven.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Pass the PR Builder.

…correctly

dongjoon-hyun · 2020-05-14T21:49:41Z

Hi, @srowen . Could you review this PR? This PR will remove the weird test-dependency failures on the PR builder and Jenkins jobs.

dongjoon-hyun · 2020-05-14T21:50:57Z

dev/test-dependencies.sh

@@ -47,7 +47,7 @@ OLD_VERSION=$($MVN -q \
    -Dexec.executable="echo" \
    -Dexec.args='${project.version}' \
    --non-recursive \
-    org.codehaus.mojo:exec-maven-plugin:1.6.0:exec)
+    org.codehaus.mojo:exec-maven-plugin:1.6.0:exec | grep -e '[0-9]\.[0-9]\.[0-9]')


I used [0-9] because \d is accepted only on Mac OS.

This would still match 3.1.0-SNAPSHOT wouldn't it - was the idea not to match it?
You could exclude "SNAPSHOT" matches but that might be just a band-aid.

You might need [0-9]+ in each case to account for double-digit versions.

\d might work with -E but not worth it.

Thanks . Yes. It matches SNAPSHOT, too.

This is only grepping lines with version like string.

Oh, just to be clear, what are you trying to match or not match here?

In the script, $(...) merges two lines into one.

BTW, is the following working in grep on Mac?

You might need [0-9]+ in each case to account for double-digit versions.

It works but you need -E as that is 'extended' regex syntax, like many things. I think -E is standard across GNU / BSD grep.

OK if you tell me this works, then I'm missing something, but it seems like this grep does not exclude the line I think you are trying to exclude, unless you want to print only the matching version from that line, and that's -o I understand it's picking up "Falling" so I am assuming something needs to ignore this line or only extract the version. grep -v Falling would just ignore that line

Thanks. BTW, grep -v Falling is not robust enough. If someone adds another warning output someday, this situation will repeat again.

I'll update to use grep -E '[0-9]+\.[0-9]+\.[0-9]+'.

dongjoon-hyun · 2020-05-15T00:16:05Z

The falling back is happening in this PR's PR Builder like the following, but this PR builder passed. Please see Falling back to archive.apache.org to download Maven messages.

========================================================================
Running Apache RAT checks
========================================================================
Attempting to fetch rat
RAT checks passed.

========================================================================
Running build tests
========================================================================
exec: curl --silent --show-error -L https://downloads.lightbend.com/zinc/0.3.15/zinc-0.3.15.tgz
exec: curl --silent --show-error -L https://downloads.lightbend.com/scala/2.12.10/scala-2.12.10.tgz
exec: curl --silent --show-error -L https://archive.apache.org/dist/maven/maven-3/3.6.3/binaries/apache-maven-3.6.3-bin.tar.gz
Using `mvn` from path: /home/jenkins/workspace/SparkPullRequestBuilder/build/apache-maven-3.6.3/bin/mvn
Using `mvn` from path: /home/jenkins/workspace/SparkPullRequestBuilder/build/apache-maven-3.6.3/bin/mvn
Performing Maven install for hadoop-2.7-hive-1.2
Falling back to archive.apache.org to download Maven
Using `mvn` from path: /home/jenkins/workspace/SparkPullRequestBuilder/build/apache-maven-3.6.3/bin/mvn
Performing Maven validate for hadoop-2.7-hive-1.2
Falling back to archive.apache.org to download Maven
Using `mvn` from path: /home/jenkins/workspace/SparkPullRequestBuilder/build/apache-maven-3.6.3/bin/mvn
Generating dependency manifest for hadoop-2.7-hive-1.2
Using `mvn` from path: /home/jenkins/workspace/SparkPullRequestBuilder/build/apache-maven-3.6.3/bin/mvn
Performing Maven install for hadoop-2.7-hive-2.3
Falling back to archive.apache.org to download Maven
Using `mvn` from path: /home/jenkins/workspace/SparkPullRequestBuilder/build/apache-maven-3.6.3/bin/mvn
Performing Maven validate for hadoop-2.7-hive-2.3
Falling back to archive.apache.org to download Maven
Using `mvn` from path: /home/jenkins/workspace/SparkPullRequestBuilder/build/apache-maven-3.6.3/bin/mvn
Generating dependency manifest for hadoop-2.7-hive-2.3
Using `mvn` from path: /home/jenkins/workspace/SparkPullRequestBuilder/build/apache-maven-3.6.3/bin/mvn
Performing Maven install for hadoop-3.2-hive-2.3
Falling back to archive.apache.org to download Maven
Using `mvn` from path: /home/jenkins/workspace/SparkPullRequestBuilder/build/apache-maven-3.6.3/bin/mvn
Performing Maven validate for hadoop-3.2-hive-2.3
Falling back to archive.apache.org to download Maven
Using `mvn` from path: /home/jenkins/workspace/SparkPullRequestBuilder/build/apache-maven-3.6.3/bin/mvn
Generating dependency manifest for hadoop-3.2-hive-2.3
Using `mvn` from path: /home/jenkins/workspace/SparkPullRequestBuilder/build/apache-maven-3.6.3/bin/mvn
Using `mvn` from path: /home/jenkins/workspace/SparkPullRequestBuilder/build/apache-maven-3.6.3/bin/mvn

========================================================================
Building Spark
========================================================================
...

SparkQA · 2020-05-15T02:01:03Z

Test build #122636 has finished for PR 28532 at commit b96806a.

This patch fails build dependency tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2020-05-15T02:02:45Z

Oh.. In Jenkins server, the new pattern seems not working.

srowen · 2020-05-15T02:04:42Z

do you mean grep -E "^[0-9]+\.[0-9]+\.[0-9]+$"? that would force it to match the whole line and not accept anything else

dongjoon-hyun · 2020-05-15T02:05:16Z

Ah, it's different issue. There exists some corrupted ~/.m2/repository in some machines.

Using `mvn` from path: /home/jenkins/workspace/SparkPullRequestBuilder/build/apache-maven-3.6.3/bin/mvn
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-install-plugin:3.0.0-M1:install (default-cli) on project spark-parent_2.12: ArtifactInstallerException: 
Failed to install metadata org.apache.spark:spark-parent_2.12/maven-metadata.xml: 
Could not parse metadata /home/jenkins/.m2/repository/org/apache/spark/spark-parent_2.12/maven-metadata-local.xml: 
in epilog non whitespace content is not allowed but got >
(position: END_TAG seen ...</metadata>\n>... @13:2) -> [Help 1]

dongjoon-hyun · 2020-05-15T02:06:12Z

It seems that we need to clean up ./m2/repository at all Jenkins machine. During recent network outage, the directory might be corrupted.

dongjoon-hyun · 2020-05-15T02:07:10Z

Retest this please.

SparkQA · 2020-05-15T02:12:29Z

Test build #122635 has finished for PR 28532 at commit e9bdb59.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-05-15T02:17:15Z

Test build #122638 has finished for PR 28532 at commit b96806a.

This patch fails build dependency tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2020-05-15T02:17:59Z

amp-jenkins-worker-05 has the corrupted Maven repo. Last two runs were assigned into the same machine.

[SPARK-31713][INFRA] Make test-dependencies.sh detect version string correctly #28532 (comment)
[SPARK-31713][INFRA] Make test-dependencies.sh detect version string correctly #28532 (comment)

dongjoon-hyun · 2020-05-15T02:18:47Z

@srowen . If you don't mind, can we merge this? This PR passed already on amp-jenkins-worker-03 with a fallback situation (#28532 (comment)). We don't have a configuration to assign this PR into another Jenkins job machine.

srowen · 2020-05-15T02:26:17Z

I trust your judgment; it can't really make things worse either if most PR builds are failing.

HyukjinKwon

Okay, same logic. LGTM

dongjoon-hyun · 2020-05-15T02:28:06Z

Thank you, @srowen and @HyukjinKwon !

HyukjinKwon · 2020-05-15T02:28:20Z

Should we clean that specific correct corrupted one in .m2, @shaneknapp?

…correctly ### What changes were proposed in this pull request? This PR makes `test-dependencies.sh` detect the version string correctly by ignoring all the other lines. ### Why are the changes needed? Currently, all SBT jobs are broken like the following. - https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-branch-3.0-test-sbt-hadoop-3.2-hive-2.3/476/console ``` [error] running /home/jenkins/workspace/spark-branch-3.0-test-sbt-hadoop-3.2-hive-2.3/dev/test-dependencies.sh ; received return code 1 Build step 'Execute shell' marked build as failure ``` The reason is that the script detects the old version like `Falling back to archive.apache.org to download Maven 3.1.0-SNAPSHOT` when `build/mvn` did fallback. Specifically, in the script, `OLD_VERSION` became `Falling back to archive.apache.org to download Maven 3.1.0-SNAPSHOT` instead of `3.1.0-SNAPSHOT` if build/mvn did fallback. Then, `pom.xml` file is corrupted like the following at the end and the exit code become `1` instead of `0`. It causes Jenkins jobs fails ``` - <version>3.1.0-SNAPSHOT</version> + <version>Falling</version> ``` **NO FALLBACK** ``` $ build/mvn -q -Dexec.executable="echo" -Dexec.args='${project.version}' --non-recursive org.codehaus.mojo:exec-maven-plugin:1.6.0:exec Using `mvn` from path: /Users/dongjoon/APACHE/spark-merge/build/apache-maven-3.6.3/bin/mvn 3.1.0-SNAPSHOT ``` **FALLBACK** ``` $ build/mvn -q -Dexec.executable="echo" -Dexec.args='${project.version}' --non-recursive org.codehaus.mojo:exec-maven-plugin:1.6.0:exec Falling back to archive.apache.org to download Maven Using `mvn` from path: /Users/dongjoon/APACHE/spark-merge/build/apache-maven-3.6.3/bin/mvn 3.1.0-SNAPSHOT ``` **In the script** ``` $ echo $(build/mvn -q -Dexec.executable="echo" -Dexec.args='${project.version}' --non-recursive org.codehaus.mojo:exec-maven-plugin:1.6.0:exec) Using `mvn` from path: /Users/dongjoon/APACHE/spark-merge/build/apache-maven-3.6.3/bin/mvn Falling back to archive.apache.org to download Maven 3.1.0-SNAPSHOT ``` This PR will prevent irrelevant logs like `Falling back to archive.apache.org to download Maven`. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass the PR Builder. Closes #28532 from dongjoon-hyun/SPARK-31713. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> (cherry picked from commit cd5fbcf) Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

HyukjinKwon · 2020-05-15T02:29:44Z

Shall we file a JIRA and assign @shaneknapp? I think he asked this before :-).

dongjoon-hyun · 2020-05-15T02:30:19Z

I left a comment on the existing JIRA. But, new JIRA sounds good.

https://issues.apache.org/jira/browse/SPARK-31693?focusedCommentId=17107856&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17107856

[SPARK-31713][INFRA] Make test-dependencies.sh detect version string …

e9bdb59

…correctly

probot-autolabeler bot added the BUILD label May 14, 2020

dongjoon-hyun commented May 14, 2020

View reviewed changes

Address comment

b96806a

dongjoon-hyun mentioned this pull request May 15, 2020

[SPARK-31387] Handle unknown operation/session ID in HiveThriftServer2Listener #28155

Closed

HyukjinKwon approved these changes May 15, 2020

View reviewed changes

dongjoon-hyun closed this in cd5fbcf May 15, 2020

dongjoon-hyun deleted the SPARK-31713 branch May 15, 2020 02:30

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-31713][INFRA] Make test-dependencies.sh detect version string correctly #28532

[SPARK-31713][INFRA] Make test-dependencies.sh detect version string correctly #28532

dongjoon-hyun commented May 14, 2020 •

edited

dongjoon-hyun commented May 14, 2020 •

edited

dongjoon-hyun May 14, 2020 •

edited

srowen May 14, 2020

dongjoon-hyun May 14, 2020

dongjoon-hyun May 14, 2020

srowen May 14, 2020

dongjoon-hyun May 15, 2020

dongjoon-hyun May 15, 2020

srowen May 15, 2020

dongjoon-hyun May 15, 2020 •

edited

dongjoon-hyun May 15, 2020

dongjoon-hyun commented May 15, 2020 •

edited

SparkQA commented May 15, 2020

dongjoon-hyun commented May 15, 2020

srowen commented May 15, 2020

dongjoon-hyun commented May 15, 2020

dongjoon-hyun commented May 15, 2020 •

edited

dongjoon-hyun commented May 15, 2020

SparkQA commented May 15, 2020

SparkQA commented May 15, 2020

dongjoon-hyun commented May 15, 2020 •

edited

dongjoon-hyun commented May 15, 2020 •

edited

srowen commented May 15, 2020

HyukjinKwon left a comment

dongjoon-hyun commented May 15, 2020

HyukjinKwon commented May 15, 2020

HyukjinKwon commented May 15, 2020

dongjoon-hyun commented May 15, 2020

[SPARK-31713][INFRA] Make test-dependencies.sh detect version string correctly #28532

[SPARK-31713][INFRA] Make test-dependencies.sh detect version string correctly #28532

Conversation

dongjoon-hyun commented May 14, 2020 • edited

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

dongjoon-hyun commented May 14, 2020 • edited

dongjoon-hyun May 14, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dongjoon-hyun May 15, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dongjoon-hyun commented May 15, 2020 • edited

SparkQA commented May 15, 2020

dongjoon-hyun commented May 15, 2020

srowen commented May 15, 2020

dongjoon-hyun commented May 15, 2020

dongjoon-hyun commented May 15, 2020 • edited

dongjoon-hyun commented May 15, 2020

SparkQA commented May 15, 2020

SparkQA commented May 15, 2020

dongjoon-hyun commented May 15, 2020 • edited

dongjoon-hyun commented May 15, 2020 • edited

srowen commented May 15, 2020

HyukjinKwon left a comment

Choose a reason for hiding this comment

dongjoon-hyun commented May 15, 2020

HyukjinKwon commented May 15, 2020

HyukjinKwon commented May 15, 2020

dongjoon-hyun commented May 15, 2020

dongjoon-hyun commented May 14, 2020 •

edited

dongjoon-hyun commented May 14, 2020 •

edited

dongjoon-hyun May 14, 2020 •

edited

dongjoon-hyun May 15, 2020 •

edited

dongjoon-hyun commented May 15, 2020 •

edited

dongjoon-hyun commented May 15, 2020 •

edited

dongjoon-hyun commented May 15, 2020 •

edited

dongjoon-hyun commented May 15, 2020 •

edited