run_tests does not return results if any tests did not pass

[BillDunlap]

If there are any test failures, testthat::run_tests (hence test_package) throws
an error and returns nothing, which makes it hard to see what went wrong.
Below I use trace() to extract the 'res' object. (One could also make a reporter
object with reporter=rep <- ListReporter$new() in the call to test_package()
and look at rep$get_results().) Am I missing something about how to use
the reporter? The help files for the reporters have no examples.

packageVersion("testthat")
[1] ‘1.0.2.9000’
trace(testthat:::run_tests, quote(run_tests_envir <<- environment()))
Tracing function "run_tests" in package "testthat (not-exported)"
[1] "run_tests"
res <- testthat::test_package("dplyr", reporter="list")
Tracing run_tests(package, test_path, filter, reporter, ...) on entry
Auto-disconnecting SQLiteConnection
Error: Test failures
In addition: Warning message:
call dbDisconnect() when finished working with a connection
res
Error: object 'res' not found
res_df <- as.data.frame(run_tests_envir$res)
dim(res_df)
[1] 946 11
subset(res_df, error)
file context test nb failed skipped error warning user system real
911 test-tbl-cube.R tbl_cube filter 0 0 FALSE TRUE 0 0 0 0
912 test-tbl-cube.R tbl_cube summarise works with single group 0 0 FALSE TRUE 0 0 0 0
913 test-tbl-cube.R tbl_cube can coerce to data_frame 0 0 FALSE TRUE 0 0 0 0
914 test-tbl-cube.R tbl_cube can coerce to table 0 0 FALSE TRUE 0 0 0 0
915 test-tbl-cube.R tbl_cube group_vars() returns variables 0 0 FALSE TRUE 0 0 0 0
916 test-tbl.R tbl tbl_nongroup_vars() excludes group variables 0 0 FALSE TRUE 0 0 0 0

[hadley]

I never thought through that scenario. I can export another function that doesn't automatically throw on failure.

[hadley]

Maybe it's fine for this to be an argument - it's not exactly changing the output type of the function (although it is changing the potential side effects)

[hadley]

Maybe stop_on_failure = TRUE, and we might also want stop_on_warning = FALSE