New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

run_tests does not return results if any tests did not pass #609

Closed
BillDunlap opened this Issue Jul 20, 2017 · 3 comments

Comments

Projects
None yet
2 participants
@BillDunlap

BillDunlap commented Jul 20, 2017

If there are any test failures, testthat::run_tests (hence test_package) throws
an error and returns nothing, which makes it hard to see what went wrong.
Below I use trace() to extract the 'res' object. (One could also make a reporter
object with reporter=rep <- ListReporter$new() in the call to test_package()
and look at rep$get_results().) Am I missing something about how to use
the reporter? The help files for the reporters have no examples.

packageVersion("testthat")
[1] ‘1.0.2.9000’
trace(testthat:::run_tests, quote(run_tests_envir <<- environment()))
Tracing function "run_tests" in package "testthat (not-exported)"
[1] "run_tests"
res <- testthat::test_package("dplyr", reporter="list")
Tracing run_tests(package, test_path, filter, reporter, ...) on entry
Auto-disconnecting SQLiteConnection
Error: Test failures
In addition: Warning message:
call dbDisconnect() when finished working with a connection
res
Error: object 'res' not found
res_df <- as.data.frame(run_tests_envir$res)
dim(res_df)
[1] 946 11
subset(res_df, error)
file context test nb failed skipped error warning user system real
911 test-tbl-cube.R tbl_cube filter 0 0 FALSE TRUE 0 0 0 0
912 test-tbl-cube.R tbl_cube summarise works with single group 0 0 FALSE TRUE 0 0 0 0
913 test-tbl-cube.R tbl_cube can coerce to data_frame 0 0 FALSE TRUE 0 0 0 0
914 test-tbl-cube.R tbl_cube can coerce to table 0 0 FALSE TRUE 0 0 0 0
915 test-tbl-cube.R tbl_cube group_vars() returns variables 0 0 FALSE TRUE 0 0 0 0
916 test-tbl.R tbl tbl_nongroup_vars() excludes group variables 0 0 FALSE TRUE 0 0 0 0

@hadley

This comment has been minimized.

Member

hadley commented Oct 2, 2017

I never thought through that scenario. I can export another function that doesn't automatically throw on failure.

@hadley

This comment has been minimized.

Member

hadley commented Oct 3, 2017

Maybe it's fine for this to be an argument - it's not exactly changing the output type of the function (although it is changing the potential side effects)

@hadley

This comment has been minimized.

Member

hadley commented Oct 3, 2017

Maybe stop_on_failure = TRUE, and we might also want stop_on_warning = FALSE

@hadley hadley closed this in d73d1d1 Oct 3, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment