Junit reporter #481

lbartnik · 2016-05-28T08:11:38Z

I've rewritten the JUnit reporter to match the new API. I understand xml2 and its own tests are missing, I'll work on that in the coming days. In the meantime, could you please let me know if there is anything missing/flawed as far as JUnit/testthat designs go?

I want to integrate testthat with JUnit/XML output but I'm not a JUnit expert.

Lukasz

Need to rethink the mapping onto jUnit structure. Currently test identifiers change if they fail.

Works for expect_that() but not expect_equals() etc.

fix broken substitution of info arg; a pseudo-test of info arg.

Conflicts: DESCRIPTION NAMESPACE

This reverts commit 96f65a3.

- change name to JunitReporter to make it searchable by "junit" - add a default output file name - fix outdated API calls

codecov-io · 2016-05-28T08:15:15Z

Current coverage is 79.88% (diff: 100%)

Merging #481 into master will increase coverage by 1.28%

@@             master       #481   diff @@
==========================================
  Files            56         57     +1   
  Lines          1668       1735    +67   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits           1311       1386    +75   
+ Misses          357        349     -8   
  Partials          0          0

Powered by Codecov. Last update 46d15da...c095ae3

lbartnik · 2016-05-30T21:08:47Z

Hi, @hadley given your comment I have taken a look at xml2 and it seems that this package does not have a convenient API to build a XML document the way you can do it with XML. Any hints on how to replace XML with xml2 in this case?

lbartnik · 2016-12-06T00:51:30Z

@dirkschumacher's advice has proven correct. AppVeyor doesn't choke on tests anymore :-)

hadley · 2016-12-15T03:36:24Z

Could you please merge/rebase to clean up the commit history? Then I'll do a more thorough review.

Also need to add to test-reporter.R for regression tests.

I'm planning for a release early next year so it would be great if we can (finally) get this in.

lbartnik · 2016-12-15T06:12:00Z

I've merged but I'm not sure if that's enough to "clean up the commit history". Can you elaborate what exactly would be your goal? I could copy the current diff into a new branch and that way you'd have a single commit with no history there - but I thought you would rather keep track of all actual contributors.

I've also added a few lines to test-reporter.R but now I'm not sure if the current design is correct. By default this reporter writes to a file ./report.xml and the standard output gets something much more simplified and human-readable. But maybe the XML should be displayed directly on stdout and the file should be an option that's not used by default?

hadley

You have 37 commits which seems rather a lot but if you don't want to squash the commits, I'll simply do it when I merge it.

I agree that you should be outputting the xml, not the summary reporter results.

hadley · 2016-12-15T06:42:40Z

R/reporter-junit.R

+}
+
+#' @importFrom xml2 xml_attr<- xml_add_child
+add_broken <- function (parent, type, message) {


I'd prefer that you not import these functions and instead use fully qualified calls.

Can I ask why?

hadley · 2016-12-15T06:43:52Z

R/reporter-junit.R

+#' This reporter includes detailed results about each test and summaries,
+#' written to a file (or stdout) in jUnit XML format. This can be read by
+#' the Jenkins Continuous Integration System to report on a dashboard etc.
+#' Requires the XML package.


hadley · 2016-12-15T06:44:04Z

R/reporter-junit.R

+#' the Jenkins Continuous Integration System to report on a dashboard etc.
+#' Requires the XML package.
+#'
+#' This works for \code{\link{expect_that}} but not for the wrappers like


I don't know... I think I must have left it since the original (previous) pull request.

hadley · 2016-12-15T06:44:52Z

R/reporter-junit.R

+#' References for the jUnit XML format:
+#' \url{http://llg.cubic.org/docs/junit/}
+#'
+#' Output drawn from the SummaryReporter is printed to the standard


One reporter should do one thing, so I agree that you should drop this and simply output xml

hadley · 2016-12-15T06:46:32Z

R/reporter-junit.R

+      self$suite_time <- self$suite_time + time
+
+      # XML node for test case
+      testcase <- xml_add_child(self$suite, "testcase")


Can eliminate add_attrs in favour of:

testcase <- xml_add_child(self$suite, "testcase", time = time, classname = paste0(classnameOK(context), '.', classnameOK(name))) )

(And please note my preferred indenting style)

By the way, I used the default RStudio indentation setting.

hadley · 2016-12-15T06:47:07Z

R/reporter-junit.R

+
+      # XML node for test case
+      testcase <- xml_add_child(self$suite, "testcase")
+      name <- if (is.null(test) || !nchar(test)) "(unnamed)" else test


Please use name <- test %||% "(unnamed)"

hadley · 2016-12-15T06:47:47Z

R/reporter-junit.R

+        add_broken(testcase, 'error', message)
+        self$errors <- self$errors + 1
+      }
+      else if (expectation_failure(result)) {


Please more else to previous line

hadley · 2016-12-15T06:48:48Z

R/reporter-junit.R

+
+#' @importFrom xml2 xml_attr<- xml_add_child
+add_broken <- function (parent, type, message) {
+  child <- xml_add_child(parent, type)


You can assign the attributes during creation

hadley · 2016-12-15T06:49:08Z

appveyor.yml

@@ -14,6 +14,7 @@ install:

 build_script:
  - travis-tool.sh install_deps
+  - travis-tool.sh install_r Rcpp


Why is this necessary?

AppVeyor's build would fail on Windows without this - Rcpp wouldn't be installed. I don't know why but this fixed the problem.

@krlmlr any ideas?

Rcpp is not being installed probably due to r-lib/devtools#1246 or possibly r-lib/devtools#1409. Use devel devtools in appveyor should fix it.

hadley

Getting close. Thanks!

hadley · 2016-12-19T17:35:48Z

NAMESPACE

@@ -131,6 +132,10 @@ importFrom(crayon,green)
 importFrom(crayon,red)
 importFrom(crayon,yellow)
 importFrom(magrittr,"%>%")
+importFrom(xml2,"xml_attr<-")


These can go away now

hadley · 2016-12-19T17:36:26Z

R/reporter-junit.R

+      self$cat(context, ": ")
+
+      self$suite <- xml2::xml_add_child(self$root, "testsuite")
+      add_attrs(self$suite,


Could now be inlined into call above, and add_attrs() deleted, right?

hadley · 2016-12-19T17:36:50Z

appveyor.yml

@@ -14,6 +14,7 @@ install:

 build_script:
  - travis-tool.sh install_deps
+  - travis-tool.sh install_r Rcpp


@krlmlr any ideas?

hadley · 2016-12-19T17:37:23Z

R/reporter-junit.R

+        add_broken(testcase, 'error', message)
+        self$errors <- self$errors + 1
+      } else
+      if (expectation_failure(result)) {


if should be on same line as else

Oh, I didn't get that. That's really hard to read. How come this is your preferred style?

hadley · 2016-12-19T17:37:58Z

R/reporter-junit.R

+
+      # add child XML node if not success
+      if (expectation_error(result)) {
+        self$cat_tight(single_letter_summary(result))


I don't think you should be printing these out at all

That's deleted in my branch...

hadley · 2016-12-19T17:38:21Z

R/reporter-junit.R

+      if (expectation_error(result)) {
+        self$cat_tight(single_letter_summary(result))
+        add_broken(testcase, 'error', message)
+        self$errors <- self$errors + 1


And I don't think you're using errors anywhere, so you can remove.

I do use them, in end_context

hadley · 2016-12-19T17:39:29Z

R/reporter-junit.R

+  gsub("[ \\.]", "_", text)
+}
+
+add_broken <- function (parent, type, message) {


Don't need assignment any more, and maybe should be add_failure_node()? Are you sure the type is both the node name and an attribute?

I followed the previous author here but type should be either the "exception class" (for error) or "type of the assertion" for failure). However, it doesn't seem we have an equivalent value here and XSD requires a type. Do you have a better idea what to put there?

hadley · 2016-12-19T17:39:51Z

R/reporter-junit.R

+      self$cat("\n")
+
+      # this causes a segfault write_xml(xmlDoc, self$file, format = )
+      cat(toString(self$doc), file = self$file)


This should just be printed to stdout like all the other reporters.

What if I'd like to output XML to a file? Do you expect people to capture.output() ?

lbartnik · 2016-12-19T18:36:09Z

OK, waiting for the next batch of comments... ;-)

hadley

Almost there. Just a few little things left.

I think redirecting the output to the right place is outside the scope of testthat. I think usually some other process will be calling R, and it will take care of saving the output to the right place.

hadley · 2016-12-19T20:57:49Z

R/helper-junitmock.R

@@ -0,0 +1,18 @@
+createJunitReporterMock <- function (file) {


Could you add a brief comment along the lines of "Fix components of JunitReporter that otherwise vary from run-to-run"

hadley · 2016-12-19T20:59:56Z

R/reporter-junit.R

+
+    end_reporter = function() {
+      if (inherits(self$file, "connection")) {
+        file <- tempfile()


Would you mind filing an issue for this on xml2 as well? (It's fine to leave this workaround as is, but long-term xml2 should also supporting writing to connections)

Submitted under: r-lib/xml2#157

hadley · 2016-12-19T21:00:05Z

R/reporter-junit.R

+      } else if (is.character(self$file)) {
+        xml2::write_xml(self$doc, self$file, format = TRUE)
+      }
+      else {


Should be on same line as previous.

hadley · 2016-12-19T21:01:06Z

tests/testthat/test-reporter-junit.R

+context("JUnitReporter")
+
+
+test_that("Junit reporter regression", {


Do you think this is needed as well as the saved xml file? It seems a bit redundant to me.

No, you're right.

hadley · 2016-12-19T22:05:36Z

R/helper-junitmock.R

@@ -0,0 +1,26 @@
+# Fix components of JunitReporter that otherwise vary from run-to-run


Oh I just noticed this in R/ not tests/testthat. Can you please move there? (And then we're done 🎉)

Oh, shoot... Sorry for that.

lbartnik · 2016-12-19T22:06:37Z

It seems I don't need the tempfile() workaround but then XML won't be pretty-formatted. r-lib/xml2#157

hadley · 2016-12-19T22:55:13Z

Sorry, I forgot to ask for a bullet in NEWS.md — can you please add?

lbartnik · 2016-12-19T22:59:41Z

Sure. Is that OK?

hadley · 2016-12-19T23:30:34Z

Perfect - thanks!

floybix and others added 26 commits September 7, 2011 22:57

First cut at JUnitReporter.

1461799

Need to rethink the mapping onto jUnit structure. Currently test identifiers change if they fail.

Use expect_that() call as test name/identifier.

e2999cf

Works for expect_that() but not expect_equals() etc.

newJUnitReporter() wrapper function;

674ec89

fix broken substitution of info arg; a pseudo-test of info arg.

code cleanup and extended function help

f079343

merge hadley's master back in

9ee71a8

update JUnitReporter to use Reference Classes

25e16f0

oops, commit modified files, I am used to using bzr not git

5e329a9

merge in upstream changes

998d097

Merge remote-tracking branch 'floybix/master' #1

406fecd

Conflicts: DESCRIPTION NAMESPACE

re #1 Changed DESCRIPTION to make the thing build

07f0d13

re #1 Export JUnitReporter

132c49d

Removed bad example

3bf68de

Fixed documentation

a60e836

Fixes #2 changed bad calls to colourise

e77343e

#3 Result failure message in outcoming XML

96f65a3

Revert "#3 Result failure message in outcoming XML"

6bc6872

This reverts commit 96f65a3.

#3 Fix failure XML node contents

16086e7

#3 Add messages to XML

d79d08d

#6 Use srcref to get test location

5ee04aa

#4 Remove deparse of expect_that

272495a

Close #4 Remove unclear test

a0d2e16

Merge remote-tracking branch 'hadley/master' into junit-reporter

cc95e33

rename junit file

c3eb3fa

rewrite JUnitReporter in R6

dbc3aa4

update reference

2d3154a

fix basic runtime issues

c8afb8b

- change name to JunitReporter to make it searchable by "junit" - add a default output file name - fix outdated API calls

lbartnik mentioned this pull request May 28, 2016

jUnit reporter #260

Closed

fix minor issues

859b4c3

hadley mentioned this pull request Dec 15, 2016

Exit code once a test fails #515

Closed

lbartnik added 2 commits December 14, 2016 20:09

Merge branch 'hadley/master' into junit-reporter

72396b9

add regression tests

2455a42

hadley reviewed Dec 15, 2016

View reviewed changes

address comment by @hadley

9709727

hadley reviewed Dec 19, 2016

View reviewed changes

lbartnikmsft added 4 commits December 19, 2016 10:08

fix unit tests

8fbd66c

remove add_broken and add_attrs

c163713

fix else if

9b27aa8

Merge remote-tracking branch 'hadley/master' into junit-reporter

3820731

lbartnikmsft added 3 commits December 19, 2016 10:39

fix DESCRIPTION

8d451b3

switch to Markdown in Roxygen comments

2dd4402

fix tests after merge

0ff35b9

hadley reviewed Dec 19, 2016

View reviewed changes

lbartnikmsft added 2 commits December 19, 2016 13:34

address the last few comments

507eb4f

remove a redundant test

8e24a79

hadley reviewed Dec 19, 2016

View reviewed changes

lbartnikmsft added 2 commits December 19, 2016 14:07

move helper into tests/testthat

df9025f

fix DESCRIPTION

73fa6ab

a new entry in NEWS.md

d9e487b

hadley merged commit 3b2f225 into r-lib:master Dec 19, 2016

nealrichardson mentioned this pull request Aug 7, 2017

Write TAP report to a file specified via options() #612

Closed

		@@ -0,0 +1,18 @@
		createJunitReporterMock <- function (file) {

		context("JUnitReporter")


		test_that("Junit reporter regression", {

		@@ -0,0 +1,26 @@
		# Fix components of JunitReporter that otherwise vary from run-to-run

Junit reporter #481

Junit reporter #481

Conversation

lbartnik commented May 28, 2016

codecov-io commented May 28, 2016 • edited Loading

Current coverage is 79.88% (diff: 100%)

lbartnik commented May 30, 2016

lbartnik commented Dec 6, 2016

hadley commented Dec 15, 2016

lbartnik commented Dec 15, 2016

hadley left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hadley left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lbartnik Dec 19, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lbartnik commented Dec 19, 2016

hadley left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lbartnik commented Dec 19, 2016

hadley commented Dec 19, 2016

lbartnik commented Dec 19, 2016

hadley commented Dec 19, 2016

codecov-io commented May 28, 2016 •

edited

Loading

lbartnik Dec 19, 2016 •

edited

Loading