Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

max_fail is checked after test_that blocks, not after each expectation #967

Closed
vspinu opened this issue Nov 19, 2019 · 2 comments · Fixed by #1069
Closed

max_fail is checked after test_that blocks, not after each expectation #967

vspinu opened this issue Nov 19, 2019 · 2 comments · Fixed by #1069
Labels
bug an unexpected problem or unintended behavior reporter 📝

Comments

@vspinu
Copy link
Contributor

vspinu commented Nov 19, 2019

I see that the default to 10 max_failures is no longer respected. The max_fail seems to no longer be used in the code https://github.com/r-lib/testthat/search?q=max_fail&unscoped_q=max_fail

@vspinu
Copy link
Contributor Author

vspinu commented Nov 19, 2019

Actually it's used, but not quite as expected. It's the github search facility which didn't find all the occurrences.

My original issue is that the number of test failures is checked only in test_that blocks.

For example put this into the test_tmp.R file:

test_that("Example", {
  for (i in 1:100) {
    expect_true(FALSE)
  }
})

and run test_file("./test-tmp.R"). This will run all the 100 expectations.

> sessionInfo()
R Under development (unstable) (2019-08-06 r76924)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 18.04.3 LTS

Matrix products: default
BLAS:   /home/vspinu/bin/R-latest/lib/libRblas.so
LAPACK: /home/vspinu/bin/R-latest/lib/libRlapack.so

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C               LC_TIME=en_US.UTF-8       
 [4] LC_COLLATE=en_US.UTF-8     LC_MONETARY=de_DE.UTF-8    LC_MESSAGES=C             
 [7] LC_PAPER=de_DE.UTF-8       LC_NAME=C                  LC_ADDRESS=C              
[10] LC_TELEPHONE=C             LC_MEASUREMENT=de_DE.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  utils     datasets  grDevices methods   base     

other attached packages:
[1] testthat_2.2.1

loaded via a namespace (and not attached):
[1] compiler_3.7.0 magrittr_1.5   R6_2.4.0       tools_3.7.0    rlang_0.4.1   

@vspinu vspinu changed the title max_fail is no longer respected max_fail is checked after test_that blocks, not after each expectation Nov 19, 2019
@hadley hadley added feature a feature request or enhancement reporter 📝 labels Apr 13, 2020
@hadley
Copy link
Member

hadley commented Jun 30, 2020

Looking at the code, I can't even figure out why this works at all

....

Aaah, it's the is_full() method that test_file() checks after each file. I don't think anything has changed here for a long time — the check only happens at the end of the file, which means that even after max_failure fails in one file, the rest of the tests are still run.

@hadley hadley added bug an unexpected problem or unintended behavior and removed feature a feature request or enhancement labels Jun 30, 2020
hadley added a commit that referenced this issue Jun 30, 2020
hadley added a commit that referenced this issue Jul 1, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug an unexpected problem or unintended behavior reporter 📝
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants