Testthat reporting warnings that do not appear in code

[jeroen]

Minimal reprex

from @gaborcsardi

test_that("x", {
  op <- options(warn = -1)  
  warning("not see this")
  options(op)
  expect_true(TRUE)
})

Real world case

If you run this in R, there is no warning:

out <- available.packages(repos = 'https://cloud.r-project.org', type = 'mac.binary.el-capitan')

However if you run this code anywhere in testthat:

This warning should never appear because R sets options(warn = -1L) internally.

[gaborcsardi]

Even better reprex:

code <- quote(test_that("x", {
  op <- options(warn = -1)  
  warning("not see this")
  options(op)
  expect_true(TRUE)
}))

cat(deparse(code), sep = "\n", file = tmp <- tempfile())

testthat::test_file(tmp)
#> ✔ |  OK F W S | Context
#> 
⠏ |   0       | filebd63127caaed
✔ |   1   1   | filebd63127caaed
#> ───────────────────────────────────────────────────────────────────────────
#> filebd63127caaed:3: warning: x
#> not see this
#> ───────────────────────────────────────────────────────────────────────────
#> 
#> ══ Results ════════════════════════════════════════════════════════════════
#> Duration: 0.2 s
#> 
#> OK:       1
#> Failed:   0
#> Warnings: 1
#> Skipped:  0

^{Created on 2019-11-04 by the reprex package (v0.3.0)}

[gaborcsardi]

But there is no warning coming out here:

withCallingHandlers(
  testthat::test_that("x", {
    op <- options(warn = -1)  
    warning("not see this")
    options(op)
    testthat::expect_true(TRUE)
  }),
  warning = function(w) { print(w); browser() }
)

^{Created on 2019-11-04 by the reprex package (v0.3.0)}

[hadley]

Probably needs special handling around here: https://github.com/r-lib/testthat/blob/windows-check/R/test-that.R#L120-L123

[gaborcsardi]

But how does the warning get to testthat at all? I guess I am missing the fine bits of the options(warn = -1) implementation...

[jeroen]

I am guessing the default R condition handler for warnings checks with options(warn) on how to proceed, but testthat overrides this with a custom condition handler that doesn't do this?

[gaborcsardi]

How can you define a default condition handler? I am clearly missing something about the condition system. :(