Adding `wrap` Parameter to `test_dir`, `test_check`, and `run_tests`

[brodieG]

@krlmlr, Since 37cc0d0 it is no longer possible to run code with test_dir that involves withCallingHandlers error handling. This is because the default source action now sets wrap=TRUE, which causes all sourced code to be run within test_code.

Since wrap is not exposed in test_dir there is no way to disable this behavior when using test_dir (or test_check).

A simple example of the problem (here we just use test_file since test_file has the argument so we can show both cases):

f <- tempfile()
cat(
  "
  withCallingHandlers(
    stop('This should be handled'),
    error=function(e) cat('handled\n')
  )
  test_that('truth', {expect_true(TRUE)})
  ",
  file=f
)
test_file(f)              # fails
test_file(f, wrap=FALSE)  # works

For reference, more details on the running withCallingHandlers inside try block issue.

I'm happy to submit a PR that addresses this if you'd like.

[krlmlr]

Thanks. A straightforward fix would be to expose the wrap argument. @hadley: Would you support that?

[hadley]

I'd rather leave this for now, and think more about the handler support in rlang.

[brodieG]

@hadley note that 37cc0d0 breaks existing use cases and without adding the wrap parameter to test_dir etal (as was done for test_file) there is no way to restore the original behavior. Seems odd to add this functionality and a wrap parameter to test_file to control it, but not to test_dir or test_check.

[hadley]

This seems ok to me - handled is printed in both cases

library(testthat)
f <- tempfile()
cat(
  "
  withCallingHandlers(
    stop('This should be handled'),
    error=function(e) cat('handled\n')
  )
  test_that('truth', {expect_true(TRUE)})
  ",
  file=f
)
test_file(f)              # fails
#> handled
#> 1
#> Failed -------------------------------------------------------------------------------------------------------------------
#> 1. Error: (unknown) (@file11e33799dbdb7#2) -------------------------------------------------------------------------------
#> This should be handled
#> 1: withCallingHandlers(stop("This should be handled"), error = function(e) cat("handled\n")) at /tmp/RtmpoLCXK7/file11e33799dbdb7:2
#> 2: stop("This should be handled")
#> 
#> DONE =====================================================================================================================
#> Don't worry, you'll get it.
test_file(f, wrap=FALSE)  # works
#> handled
#> Error in withCallingHandlers(stop("This should be handled"), error = function(e) cat("handled\n")): This should be handled
#> 
#> DONE =====================================================================================================================

[hadley]

If that's not correct, it would be useful for you to write up what you do expect, and why.

[brodieG]

I'm sorry I was not very clear about the exact nature of the problem:

• In the first run, testthat declares that the tests in our file failed with 1 error.
• In the second run the error is handled by withCallingHandlers alone, but since it is handled AND wrap=FALSE, testthat does not declare a test failure (although there is some screen output)

To try to clarify a little, I run with the check reporter:

> test_file(f, reporter=CheckReporter)
handled
1. Error: (unknown) (@file6e7c7be4641e#2) --------------------------------------
This should be handled
1: withCallingHandlers(stop("This should be handled"), error = function(e) cat("handled\n")) at /tmp/Rtmpsuh4RO/file6e7c7be4641e:2
2: stop("This should be handled")

testthat results ================================================================
OK: 0 SKIPPED: 0 FAILED: 1
1. Error: (unknown) (@file6e7c7be4641e#2)

Error: testthat unit tests failed
> test_file(f, wrap=FALSE, reporter=CheckReporter)
handled
Error in withCallingHandlers(stop("This should be handled"), error = function(e) cat("handled\n")) :
  This should be handled
testthat results ================================================================
OK: 0 SKIPPED: 0 FAILED: 0

Notice how in the second run the tests are not considered to fail.

The problem is that with withCallingHandlers there is no way to fully capture the triggered error and prevent outer handling calls from wresting control from the inner ones. If I had used tryCatch then this would not be a problem; however, if I do that I cannot resume evaluation after non fatal conditions as I can with withCallingHandlers.

I explore the withCallingHandlers issue on SO in depth, if you're interested.

This matters to me because I built the tests for my own unit testing framework with testthat. My unit testing framework uses withCallingHandlers internally, so it itself cannot be run within a test_that block because evaluation would be interrupted. My testing pattern is to run my functions outside of test_that blocks, and then check the results stored in variables within test_that blocks. This worked fine before the change in paradigm discussed in this issue. Now my tests start failing because as soon a condition is issued inside my tests it is captured by the handlers added in test_file(..., wrap=TRUE).

[lionel-]

The handle_fatal() handler could rethrow errors that come from outside of a test_that() block. However this feels wrong, I think it makes sense to catch errors here.

It seems the best solution would be to put your code within test_that() blocks. You can isolate your handling code from testthat's handlers by evaluating within a ToplevelExec() block. This is a C function but we'll expose it at R level in rlang.

[brodieG]

Thanks @lionel, I'll explore the use of ToplevelExec() for my use case. I will note that even if that works is probably useful to have a way of running expressions outside of handlers in test_dir et al., especially since that already exists for test_file (i.e. wrap=FALSE) and that is the status quo with the current version. Maybe an alternative could be a with_no_handlers block, although that seems like a lot of work compared to just adding the param already in test_file to the sister functions.

[lionel-]

It's just that the wrap argument feels a bit ill defined and complicates the API. So if it might be better to find solutions that don't require this argument (which doesn't exist on the current CRAN version IIUC). Rethrowing errors seems to preserve your use case. skip conditions could be resignaled as well.

[brodieG]

Note that I'm NOT looking for an error to be re-thrown, in fact, I'm looking for the exact opposite. I want withCallingHandlers to be able to fully handle conditions, which cannot be done if wrapped in a tryCatch:

f <- function()
  withCallingHandlers({
      warning("A");
      cat("/\\/\\ WOOHOO /\\/\\ \n")
    },
    warning = function(w) {}
  )

f()
## /\/\ WOOHOO /\/\                                              #   <-------- Notice this
## Warning message:
## In withCallingHandlers({ : A
tryCatch(f(), warning=function(w) warning(w))
## Warning message:
## In withCallingHandlers({ : A

A stand alone withCallingHandlers will resume execution after the warning, whereas when you wrap it in a tryCatch with a warning handler control is never returned to the interior evaluation. IMO this is a flaw in R's condition handling mechanism, which hopefully can be mitigated with ToplevelExec().

Previously I could just run some code that used withCallingHandlers outside of the test_that blocks. Now this is no longer possible with test_dir, test_check, etc because they auto-wrap things in test blocks. It is possible with test_file because it has the wrap argument that I can set to FALSE.

[lionel-]

Yes sorry I got confused. If we rethrow errors from the tryCatch the stack is of course unwound.

IMO this is a flaw in R's condition handling mechanism

I don't think it's flawed, it's just that it doesn't make sense to mark an exceptional condition (an error) as handled if the code that threw the error is not designed for this kind of recovery (i.e. it doesn't set a restart point).

[brodieG]

But what about warnings? In my use case I'm evaluating arbitrary code inside withCallingHandlers. I need the code to keep evaluating normally after it issues a warning as it would if it were run on the command line. As far as I know there is no way to do this if withCallingHandlers is called within a tryCatch block that also handles warnings.

Keep in mind I'm evaluating arbitrary code. I don't know if it will contain warnings or not, e.g.:

withCallingHandlers(eval(exp), warning=...)

Where exp is supplied by the user.

[lionel-]

All warnings have restart points, which makes sense because they are not exceptional conditions. So you can invoke the muffleWarning restart from your handler to mark it as handled. Your local handler will be the last one called by R.

[lionel-]

Unfortunately base::signalCondition() doesn't set up restart points automatically. Unlike rlang::cnd_signal() which sets up a restart by default.