[SPARK-32253][INFRA] Show errors only for the sbt tests of github actions #29133

gengliangwang · 2020-07-16T14:58:25Z

What changes were proposed in this pull request?

Make the test result log of github action more readable by showing errors from SBT only.

Add "--error" flag to sbt in github action to set the log level as "ERROR"
Show only failed test cases in stderr output of github action. According to https://www.scalatest.org/user_guide/using_the_runner, with SBT option -eNCXEHLOPQMDF we can drop all the following events:

N - drop TestStarting events
C - drop TestSucceeded events
X - drop TestIgnored events
E - drop TestPending events
H - drop SuiteStarting events
L - drop SuiteCompleted events
O - drop InfoProvided events
P - drop ScopeOpened events
Q - drop ScopeClosed events
R - drop ScopePending events
M - drop MarkupProvided events

and enable the following two mode:

D - show all durations
F - show full stack traces

Why are the changes needed?

Currently, the output of github action is very long and we have to scroll down to find the failed test cases. Even more, the log may be truncated. In such a case, we will have to wait until all the jobs are completed and then download all the raw logs.

Does this PR introduce any user-facing change?

No

How was this patch tested?

Before changes, all the warnings in compiling are shown:

as well as all the passed and ignored test cases:

After changes, sbt test only shows the summary for a successful job:

If there is a test failure, a full stack track is shown as well as a test failure summary at the end of test log:

SparkQA · 2020-07-16T15:01:06Z

Test build #125982 has finished for PR 29133 at commit 3bcfbb1.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-07-16T15:08:24Z

Test build #125983 has finished for PR 29133 at commit 60db1af.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

gengliangwang · 2020-07-16T16:40:35Z

project/SparkBuild.scala

+    // Show only the failed test cases in github action to make the log more readable.
+    testOptions in Test += Tests.Argument(TestFrameworks.ScalaTest,
+      sys.env.get("GITHUB_ACTIONS").map { _ =>
+        Seq("-eNCXEHLOPQMDSF")


Check https://www.scalatest.org/user_guide/using_the_runner for the options here.

It seems that you explicitly enabled all available standard error options except W and U. If then, could you describe the reason why you choose some and exclude those two, please?

W - without color U - unformatted mode

Also, could you add the link into the comment at below line 1030 because this is non-trivial?

Here we drop all the following events:

N - drop TestStarting events C - drop TestSucceeded events X - drop TestIgnored events E - drop TestPending events H - drop SuiteStarting events L - drop SuiteCompleted events O - drop InfoProvided events P - drop ScopeOpened events Q - drop ScopeClosed events R - drop ScopePending events M - drop MarkupProvided events

and enable the following two mode:

D - show all durations F - show full stack traces

The following two modes will make the readability worse

W - without color U - unformatted mode

Could you put that explanation into the PR description, please? PR description will become a permanent commit log . So, it will help other people to understand your decision. Thanks, @gengliangwang .

@dongjoon-hyun Sure, I have updated the pr description.

SparkQA · 2020-07-16T19:16:21Z

Test build #125989 has finished for PR 29133 at commit 663e5a7.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-07-16T19:45:40Z

Test build #125991 has finished for PR 29133 at commit 3590262.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2020-07-17T03:10:54Z

@gengliangwang, it would be easier to review if we have before/after examples of the logs :-) cc @viirya as well

gengliangwang · 2020-07-17T03:34:03Z

@HyukjinKwon I have updated the PR description.
Meanwhile, I created a PR on my repo to see what the test failure log will look like: gengliangwang#6

Here is an example of failed log output: https://github.com/gengliangwang/spark/pull/6/checks?check_run_id=880362871

SparkQA · 2020-07-17T11:12:31Z

Test build #126044 has finished for PR 29133 at commit 2d39b8e.

This patch fails PySpark pip packaging tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2020-07-17T15:37:56Z

"This patch fails PySpark pip packaging tests." flakiness happens frequently in these day although I don't know the root cause.

viirya · 2020-07-18T05:28:59Z

I saw the screenshot of after this change. Can we also add one screenshot of before this change too?

gengliangwang · 2020-07-18T06:49:18Z

@viirya I have updated the PR description.

HyukjinKwon · 2020-07-19T01:32:26Z

@dongjoon-hyun, I am debugging the flakiness at #29117 to make a complete fix. I think it was caused by my recent fixes and Jenkins environment differences vs Github Actions. I will be able to fix at least in the next week.

HyukjinKwon · 2020-07-19T01:36:00Z

I am okay to try this out for now until we have a way to report the test results but I'll leave it to @dongjoon-hyun and @viirya.

viirya · 2020-07-19T01:41:24Z

Looks okay to me. We can keep improving this.

HyukjinKwon

Okay.. LGTM

HyukjinKwon · 2020-07-19T03:00:01Z

I filed new JIRA for test reporter (SPARK-32357).. let's see how it goes.

Merged to master.

HyukjinKwon · 2020-07-24T04:15:16Z

@gengliangwang, seems this hides the test results from JUnit tests. Can you take a look and see if it shows JUnit test failures as well? If you're busy for something else, we can revert it for now as well.

gengliangwang · 2020-07-24T04:29:41Z

@HyukjinKwon thanks for the reminder! I will see if there is any quick fix. If not, I will revert this very soon.

HyukjinKwon · 2020-07-24T04:45:30Z

Thank you @gengliangwang for investigating this.

…thub actions" ### What changes were proposed in this pull request? This reverts commit 026b0b9. ### Why are the changes needed? As HyukjinKwon pointed out in #29133 (comment), there is no JUnit test report after #29133. Let's revert #29133 for now and find a better solution to improve the log output later. ### Does this PR introduce _any_ user-facing change? No, dev-only. ### How was this patch tested? GitHub Actions build Closes #29219 from gengliangwang/revertErrorOnly. Authored-by: Gengliang Wang <gengliang.wang@databricks.com> Signed-off-by: Gengliang Wang <gengliang.wang@databricks.com>

show error only

3bcfbb1

probot-autolabeler bot added the BUILD label Jul 16, 2020

fix flag

60db1af

fix

663e5a7

gengliangwang changed the title ~~[SPARK-32253][INFRA] Make readability better in the test result logs~~ [WIP][SPARK-32253][INFRA] Make readability better in the test result logs Jul 16, 2020

update comment

3590262

gengliangwang requested review from HyukjinKwon and dongjoon-hyun July 16, 2020 16:38

gengliangwang commented Jul 16, 2020

View reviewed changes

gengliangwang requested a review from viirya July 16, 2020 16:41

gengliangwang changed the title ~~[WIP][SPARK-32253][INFRA] Make readability better in the test result logs~~ [WIP][SPARK-32253][INFRA] Show errors only for the sbt tests of github actions Jul 16, 2020

gengliangwang changed the title ~~[WIP][SPARK-32253][INFRA] Show errors only for the sbt tests of github actions~~ [SPARK-32253][INFRA] Show errors only for the sbt tests of github actions Jul 17, 2020

add comments and remove option 'S'

2d39b8e

HyukjinKwon approved these changes Jul 19, 2020

View reviewed changes

HyukjinKwon closed this in 026b0b9 Jul 19, 2020

gengliangwang mentioned this pull request Jul 24, 2020

Revert "[SPARK-32253][INFRA] Show errors only for the sbt tests of github actions" #29219

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-32253][INFRA] Show errors only for the sbt tests of github actions #29133

[SPARK-32253][INFRA] Show errors only for the sbt tests of github actions #29133

gengliangwang commented Jul 16, 2020 •

edited

SparkQA commented Jul 16, 2020

SparkQA commented Jul 16, 2020

gengliangwang Jul 16, 2020

dongjoon-hyun Jul 17, 2020 •

edited

dongjoon-hyun Jul 17, 2020

gengliangwang Jul 17, 2020

gengliangwang Jul 17, 2020

dongjoon-hyun Jul 17, 2020 •

edited

gengliangwang Jul 17, 2020

SparkQA commented Jul 16, 2020

SparkQA commented Jul 16, 2020

HyukjinKwon commented Jul 17, 2020

gengliangwang commented Jul 17, 2020 •

edited

SparkQA commented Jul 17, 2020

dongjoon-hyun commented Jul 17, 2020

viirya commented Jul 18, 2020

gengliangwang commented Jul 18, 2020

HyukjinKwon commented Jul 19, 2020

HyukjinKwon commented Jul 19, 2020

viirya commented Jul 19, 2020

HyukjinKwon left a comment

HyukjinKwon commented Jul 19, 2020

HyukjinKwon commented Jul 24, 2020

gengliangwang commented Jul 24, 2020

HyukjinKwon commented Jul 24, 2020

[SPARK-32253][INFRA] Show errors only for the sbt tests of github actions #29133

[SPARK-32253][INFRA] Show errors only for the sbt tests of github actions #29133

Conversation

gengliangwang commented Jul 16, 2020 • edited

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

SparkQA commented Jul 16, 2020

SparkQA commented Jul 16, 2020

gengliangwang Jul 16, 2020

Choose a reason for hiding this comment

dongjoon-hyun Jul 17, 2020 • edited

Choose a reason for hiding this comment

dongjoon-hyun Jul 17, 2020

Choose a reason for hiding this comment

gengliangwang Jul 17, 2020

Choose a reason for hiding this comment

gengliangwang Jul 17, 2020

Choose a reason for hiding this comment

dongjoon-hyun Jul 17, 2020 • edited

Choose a reason for hiding this comment

gengliangwang Jul 17, 2020

Choose a reason for hiding this comment

SparkQA commented Jul 16, 2020

SparkQA commented Jul 16, 2020

HyukjinKwon commented Jul 17, 2020

gengliangwang commented Jul 17, 2020 • edited

SparkQA commented Jul 17, 2020

dongjoon-hyun commented Jul 17, 2020

viirya commented Jul 18, 2020

gengliangwang commented Jul 18, 2020

HyukjinKwon commented Jul 19, 2020

HyukjinKwon commented Jul 19, 2020

viirya commented Jul 19, 2020

HyukjinKwon left a comment

Choose a reason for hiding this comment

HyukjinKwon commented Jul 19, 2020

HyukjinKwon commented Jul 24, 2020

gengliangwang commented Jul 24, 2020

HyukjinKwon commented Jul 24, 2020

gengliangwang commented Jul 16, 2020 •

edited

dongjoon-hyun Jul 17, 2020 •

edited

dongjoon-hyun Jul 17, 2020 •

edited

gengliangwang commented Jul 17, 2020 •

edited