[SPARK-32647][INFRA] Report SparkR test results with JUnit reporter #29456

HyukjinKwon · 2020-08-18T03:45:01Z

What changes were proposed in this pull request?

This PR proposes to generate JUnit XML test report in SparkR tests that can be leveraged in both Jenkins and GitHub Actions.

GitHub Actions

Jenkins

NOTE that while I am here, I am switching back the console reporter from "progress" to "summary". Currently non-ascii codes are broken in Jenkins console and switching it to "summary" can work around it.
"summary" is the default format used in testthat 1.x.

Why are the changes needed?

To check the test failures more easily.

Does this PR introduce any user-facing change?

No, dev-only

How was this patch tested?

It is tested in GitHub Actions at https://github.com/HyukjinKwon/spark/pull/23/checks?check_run_id=996586446
In case of Jenkins, https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127525/testReport/

SparkQA · 2020-08-18T04:29:48Z

Test build #127525 has finished for PR 29456 at commit 8fb651a.

This patch fails SparkR unit tests.
This patch merges cleanly.
This patch adds no public classes.

This reverts commit 8fb651a.

HyukjinKwon · 2020-08-18T05:18:15Z

cc @dongjoon-hyun, @viirya, @felixcheung, @shivaram, @gengliangwang FYI

HyukjinKwon · 2020-08-18T05:20:20Z

Note that PySpark, Scala and Java are already doing this. SparkR is only the one which currently does not leverage JUnit XML reports. This PR completes and consistently supports it in all languages with the consistent directory names (target/test-reports).

Jenkins and GitHub Actions are already configured to look up these directories to report in test results.

viirya · 2020-08-18T05:27:40Z

R/pkg/tests/run-all.R

+      dir.create("target/test-reports", showWarnings = FALSE)
+      reporter <- MultiReporter$new(list(
+        SummaryReporter$new(),
+        JunitReporter$new(file = "target/test-reports/test-results.xml")


Is it only supported by testthat 2.0+?

It looks supported in testthat 1.x but testthat 1.x support here is supposed to be removed soon at SPARK-30663. I didn't bother test as Jenkins and GitHub Actions use testthat 2+ anyway.

viirya · 2020-08-18T05:29:13Z

R/pkg/tests/run-all.R

+        SummaryReporter$new(),
+        JunitReporter$new(file = "target/test-reports/test-results.xml")


We need both SummaryReporter and JunitReporter?

Yup, otherwise, the console output is not shown. I manually tested.

It was previously ProgressReporter (from testthat 2+) by default. It shows the console output such as:

�� | 0 | SparkSQL functions �� | 1 | SparkSQL functions �� | 2 | SparkSQL functions ...

in Jenkins (ever since we switched the default system encoding if I am not mistaken).

Now it follows the default format SummaryReporter (in testthat 1.x). It shows the console output as below:

functions in client.R: ..... test functions in sparkR.R: .............................................. include R packages: .. JVM API: .. MLlib classification algorithms, except for tree-based algorithms: ...........................................................................

There are no differences in the report format when some tests are failed.

HyukjinKwon · 2020-08-18T05:39:58Z

oh, cc @zero323 too.

SparkQA · 2020-08-18T05:53:39Z

Test build #127533 has finished for PR 29456 at commit 5bca181.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2020-08-18T06:33:23Z

Thanks guys. Let me merge this after I cherry-pick Github Actions to other branches (at #29460)

SparkQA · 2020-08-18T07:05:02Z

Test build #127544 has finished for PR 29456 at commit 603268e.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-08-18T08:40:51Z

Test build #127557 has finished for PR 29456 at commit 1fe2806.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2020-08-18T10:34:50Z

Merged to master.

This PR proposes to generate JUnit XML test report in SparkR tests that can be leveraged in both Jenkins and GitHub Actions. **GitHub Actions** ![Screen Shot 2020-08-18 at 12 42 46 PM](https://user-images.githubusercontent.com/6477701/90467934-55b85b00-e150-11ea-863c-c8415e764ddb.png) **Jenkins** ![Screen Shot 2020-08-18 at 2 03 42 PM](https://user-images.githubusercontent.com/6477701/90472509-a5505400-e15b-11ea-9165-777ec9b96eaa.png) NOTE that while I am here, I am switching back the console reporter from "progress" to "summary". Currently non-ascii codes are broken in Jenkins console and switching it to "summary" can work around it. "summary" is the default format used in testthat 1.x. To check the test failures more easily. No, dev-only It is tested in GitHub Actions at https://github.com/HyukjinKwon/spark/pull/23/checks?check_run_id=996586446 In case of Jenkins, https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127525/testReport/ Closes #29456 from HyukjinKwon/sparkr-junit. Authored-by: HyukjinKwon <gurwls223@apache.org> Signed-off-by: HyukjinKwon <gurwls223@apache.org>

HyukjinKwon · 2020-08-19T06:13:25Z

I merged into branch-3.0 and branch-2.4 as well. cc @zhengruifeng and @ScrapCodes . this is test-only so should be fine.

This PR proposes to generate JUnit XML test report in SparkR tests that can be leveraged in both Jenkins and GitHub Actions. **GitHub Actions** ![Screen Shot 2020-08-18 at 12 42 46 PM](https://user-images.githubusercontent.com/6477701/90467934-55b85b00-e150-11ea-863c-c8415e764ddb.png) **Jenkins** ![Screen Shot 2020-08-18 at 2 03 42 PM](https://user-images.githubusercontent.com/6477701/90472509-a5505400-e15b-11ea-9165-777ec9b96eaa.png) NOTE that while I am here, I am switching back the console reporter from "progress" to "summary". Currently non-ascii codes are broken in Jenkins console and switching it to "summary" can work around it. "summary" is the default format used in testthat 1.x. To check the test failures more easily. No, dev-only It is tested in GitHub Actions at https://github.com/HyukjinKwon/spark/pull/23/checks?check_run_id=996586446 In case of Jenkins, https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127525/testReport/ Closes #29456 from HyukjinKwon/sparkr-junit. Authored-by: HyukjinKwon <gurwls223@apache.org> Signed-off-by: HyukjinKwon <gurwls223@apache.org>

felixcheung · 2020-08-23T19:00:38Z

LGTM

HyukjinKwon added 2 commits August 18, 2020 12:34

Use testthat's JUnit reporter

c040c77

Add test failures

8fb651a

probot-autolabeler bot added R SQL labels Aug 18, 2020

HyukjinKwon marked this pull request as draft August 18, 2020 03:45

HyukjinKwon changed the title ~~[WIP][SPARK-32647][INFRA] Report SparkR test results with JUnit reporter~~ [SPARK-32647][INFRA] Report SparkR test results with JUnit reporter Aug 18, 2020

Revert "Add test failures"

5bca181

This reverts commit 8fb651a.

HyukjinKwon marked this pull request as ready for review August 18, 2020 05:17

viirya reviewed Aug 18, 2020

View reviewed changes

viirya approved these changes Aug 18, 2020

View reviewed changes

gengliangwang approved these changes Aug 18, 2020

View reviewed changes

install xml2 in AppVeyor

603268e

probot-autolabeler bot added the INFRA label Aug 18, 2020

Empty committer to kick AppVeyor again

1fe2806

HyukjinKwon closed this in babb654 Aug 18, 2020

HyukjinKwon deleted the sparkr-junit branch December 7, 2020 02:06

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-32647][INFRA] Report SparkR test results with JUnit reporter #29456

[SPARK-32647][INFRA] Report SparkR test results with JUnit reporter #29456

HyukjinKwon commented Aug 18, 2020 •

edited

SparkQA commented Aug 18, 2020

HyukjinKwon commented Aug 18, 2020

HyukjinKwon commented Aug 18, 2020 •

edited

viirya Aug 18, 2020

HyukjinKwon Aug 18, 2020 •

edited

viirya Aug 18, 2020

HyukjinKwon Aug 18, 2020

HyukjinKwon commented Aug 18, 2020

SparkQA commented Aug 18, 2020

HyukjinKwon commented Aug 18, 2020

SparkQA commented Aug 18, 2020

SparkQA commented Aug 18, 2020

HyukjinKwon commented Aug 18, 2020

HyukjinKwon commented Aug 19, 2020 •

edited

felixcheung commented Aug 23, 2020 via email

		SummaryReporter$new(),
		JunitReporter$new(file = "target/test-reports/test-results.xml")

[SPARK-32647][INFRA] Report SparkR test results with JUnit reporter #29456

[SPARK-32647][INFRA] Report SparkR test results with JUnit reporter #29456

Conversation

HyukjinKwon commented Aug 18, 2020 • edited

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

SparkQA commented Aug 18, 2020

HyukjinKwon commented Aug 18, 2020

HyukjinKwon commented Aug 18, 2020 • edited

viirya Aug 18, 2020

Choose a reason for hiding this comment

HyukjinKwon Aug 18, 2020 • edited

Choose a reason for hiding this comment

viirya Aug 18, 2020

Choose a reason for hiding this comment

HyukjinKwon Aug 18, 2020

Choose a reason for hiding this comment

HyukjinKwon commented Aug 18, 2020

SparkQA commented Aug 18, 2020

HyukjinKwon commented Aug 18, 2020

SparkQA commented Aug 18, 2020

SparkQA commented Aug 18, 2020

HyukjinKwon commented Aug 18, 2020

HyukjinKwon commented Aug 19, 2020 • edited

felixcheung commented Aug 23, 2020 via email

HyukjinKwon commented Aug 18, 2020 •

edited

HyukjinKwon commented Aug 18, 2020 •

edited

HyukjinKwon Aug 18, 2020 •

edited

HyukjinKwon commented Aug 19, 2020 •

edited