max_fail is checked after `test_that` blocks, not after each expectation

[vspinu]

I see that the default to 10 max_failures is no longer respected. The max_fail seems to no longer be used in the code https://github.com/r-lib/testthat/search?q=max_fail&unscoped_q=max_fail

[vspinu]

Actually it's used, but not quite as expected. It's the github search facility which didn't find all the occurrences.

My original issue is that the number of test failures is checked only in test_that blocks.

For example put this into the test_tmp.R file:

test_that("Example", {
  for (i in 1:100) {
    expect_true(FALSE)
  }
})

and run test_file("./test-tmp.R"). This will run all the 100 expectations.

Details

> sessionInfo()
R Under development (unstable) (2019-08-06 r76924)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 18.04.3 LTS

Matrix products: default
BLAS:   /home/vspinu/bin/R-latest/lib/libRblas.so
LAPACK: /home/vspinu/bin/R-latest/lib/libRlapack.so

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C               LC_TIME=en_US.UTF-8       
 [4] LC_COLLATE=en_US.UTF-8     LC_MONETARY=de_DE.UTF-8    LC_MESSAGES=C             
 [7] LC_PAPER=de_DE.UTF-8       LC_NAME=C                  LC_ADDRESS=C              
[10] LC_TELEPHONE=C             LC_MEASUREMENT=de_DE.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  utils     datasets  grDevices methods   base     

other attached packages:
[1] testthat_2.2.1

loaded via a namespace (and not attached):
[1] compiler_3.7.0 magrittr_1.5   R6_2.4.0       tools_3.7.0    rlang_0.4.1   

[hadley]

Looking at the code, I can't even figure out why this works at all

....

Aaah, it's the is_full() method that test_file() checks after each file. I don't think anything has changed here for a long time — the check only happens at the end of the file, which means that even after max_failure fails in one file, the rest of the tests are still run.