Building the report when a test fails #3
Comments
|
the script is using |
|
fo the time being the i would suggest moving to |
|
i opened a new branch for this if you want to PR to it. |
|
@llrs this works now on the skip branch, you can test it out on a package you have. cc @dpastoor, @jimhester. |
|
@yonicd thanks for the new branch. I tried and it comments a single line (the one producing the data to be tested) but not the failing test (or the The error I get on the console: > covrpage()
processing file: _covrpage.Rmd
|.... | 6%
inline R code fragments
|........ | 12%
label: unnamed-chunk-1 (with options)
List of 1
$ include: logi FALSE
|........... | 18%
ordinary text without R code
|............... | 24%
label: unnamed-chunk-2 (with options)
List of 1
$ include: logi FALSE
|................... | 29%
ordinary text without R code
|....................... | 35%
label: unnamed-chunk-3 (with options)
List of 1
$ include: logi FALSE
|........................... | 41%
ordinary text without R code
|............................... | 47%
label: unnamed-chunk-4 (with options)
List of 1
$ include: logi FALSE
|.................................. | 53%
ordinary text without R code
|...................................... | 59%
label: unnamed-chunk-5 (with options)
List of 1
$ echo: logi FALSE
Quitting from lines 52-77 (_covrpage.Rmd)
Error: Failure in `/tmp/RtmpNDFpvL/R_LIBS5d7c37dbca90/GSEAdv/GSEAdv-tests/testthat.Rout.fail`On that file, after starting R and loading the package the output ends with: >
> test_check("GSEAdv")
�[31m──�[39m �[31m1. Error: keepGPP works (@test_keeps.R#9) �[39m �[31m──────────────────────────────────────────────────────────────────────────────────────────�[39m
object 'out' not found
1: expect_equal(ncol(out), 5L) at testthat/test_keeps.R:9
2: quasi_label(enquo(object), label)
3: eval_bare(get_expr(quo), get_env(quo))
4: ncol(out)
══ testthat results ═══════════════════════════════════════════════════════════════════════════════════════════════════════════════════
OK: 183 SKIPPED: 1 FAILED: 1
1. Error: keepGPP works (@test_keeps.R#9)
Error: testthat unit tests failed
Execution haltedThe relevant lines on that file are here: test_that("keepGPP works", {
# out <- keepGPP(Info)
expect_equal(ncol(out), 5L)
})The commented line was introduced by covrpage, but then the expect is failing, as it can't find the out object |
|
The intent was to have the expectation fail, you are missing there a global variable ‘out’ it is crashing for a different reason I think. Try replacing 5L with 6L. |
|
I don't understand the comment. Why I'm missing there a global variable 'out'? Initially it wasn't commented in the test_that call, and after running the covrpage it got commented. (Doing I on purpose changed the expectation in order to fail the test, if I change it back to |
|
You misunderstood me. I meant to keep it a vaild expectation test, i did not know it was 6 before you changed it. My suggestion is to keep the out object in the environment but have it try to check against something wrong on purpose. Removing out make little sense practically in a normal test file. Usually you would have out be created but with wrong dimensions and catching that error. |
|
That is what I did initially, and still the covrpage function failed to generate the report (despite using the skip branch) |
|
Ok. Can you paste the Rout.fail lines so I can see why it is failing. |
|
It is what I originally posted on the last block of code on my first comment after you pinged me. >
> test_check("GSEAdv")
�[31m──�[39m �[31m1. Error: keepGPP works (@test_keeps.R#9) �[39m �[31m──────────────────────────────────────────────────────────────────────────────────────────�[39m
object 'out' not found
1: expect_equal(ncol(out), 5L) at testthat/test_keeps.R:9
2: quasi_label(enquo(object), label)
3: eval_bare(get_expr(quo), get_env(quo))
4: ncol(out)
══ testthat results ═══════════════════════════════════════════════════════════════════════════════════════════════════════════════════
OK: 183 SKIPPED: 1 FAILED: 1
1. Error: keepGPP works (@test_keeps.R#9)
Error: testthat unit tests failed
Execution halted |
|
That is saying out is missing. Can you rerun wo commenting out that line in the test? |
|
test_that("keepGPP works", { |
|
Yes, it says that it is missing because it gets commented when I run covrpage. When I cleaned the session and redid the covrpage again, I got some other test commented and then a failing test in another archive. These branch comments my test at its will, and some of them are test that pass. |
|
Ah, ok. So the way this works is to run testthat locate failed tests The tests are mapped out using covrpage::map_testthat() Try seeing that the lines are being mapped correctly to that test. It seemed to work ok on my end. |
|
ok. try again (repull). I noticed that i wrote now the mapping should work for both types of inputs. thanks for being the tester for this. |
|
I installed from the latest version of the branch, but I got the same error: ══ testthat results ════════════════════════════════════════════════════════════════════════
OK: 185 SKIPPED: 1 FAILED: 1
1. Failure: keepGPP works (@test_keeps.R#9)
Error: testthat unit tests failed
Execution halted |
|
errr... what is the GH url of the repo are you testing? testthat should be failing that is the point, but covr should be running ok, ignoring that test. |
|
I am using : |
|
my mistake. i didnt return the mapping directory to the right place. I also added some functionality that the function will cleanup itself all the commenting, even if it crashing during the covr build. try again (hopefully the last time) |
|
No, I get the same error "yonicd/covrpage@111cb6b", it only adds a new line at the end of a test I have |
|
Are you testing covrpage or your package? |
|
i have tried it on a few repos i checked out that i can manipulate the tests. for example @jonocarroll hosts ggeasy. I changed line 43 in test_labs to throw an error. I run the new function in the root repo directory. It is acting as expected. > covrpage::coverage_skip(test_path = 'tests/testthat')
✔ | OK F W S | Context
✖ | 3 1 | attr labs
──────────────────────────────────────────────────────────────────────────────────────────────
test-labs.R:43: failure: regular labs pass new labels through easy_labs
`easy_res` not equal to `hard_res`.
Component “labels”: Component “title”: 1 string mismatch
──────────────────────────────────────────────────────────────────────────────────────────────
✔ | 9 | remove legend [0.2 s]
✔ | 9 | remove axes [0.2 s]
══ Results ═══════════════════════════════════════════════════════════════════════════════════
Duration: 0.5 s
OK: 21
Failed: 1
Warnings: 0
Skipped: 0
No-one is perfect!
ggeasy Coverage: 84.21%
R/axis.R: 55.88%
R/labs.R: 95.00%
R/legend.R: 100.00% |
|
Oh, sorry I was trying the ══ Terminating early ══════════════════════════════════════════════════
Too many failuresbut on the linked file I have has the same error: ══ testthat results ═════════════════════════════════════════════════════════════════
OK: 149 SKIPPED: 9 FAILED: 1
1. Failure: keepGPP works (@test_keeps.R#9)
Error: testthat unit tests failed
Execution halted |
|
no worries, i added a small caveat to do the tests work in a regular context without force failing them? |
|
Hey, The function seems to fail if you simply do |
|
I changed to default path of |
|
@yonicd Yes, When I change back to the right value the Many thanks for your long and continued effort to solve this issue! |
|
what test path are you setting in the coverage_skip call and what directory are you running it from? |
|
I am using I get ══ Terminating early ═══════════════════════════════════════════════════════════════════════════════════════════════════════════════════
Too many failures
GSEAdv Coverage: 77.33%
R/completness.R: 0.00%
R/expand.R: 0.00%
R/summary.R: 0.00%
R/pathwaysPerGene.R: 14.29%
R/calc-nPathways.R: 33.33%
R/simulateGSC.R: 36.30%
R/check-GSC.R: 59.52%
R/keep.R: 72.00%
R/simulate2GSC.R: 78.38%
R/fromSizeGenes_sizePathways.R: 78.57%
R/as_GeneSetCollection.R: 83.33%
R/sizesPer.R: 86.11%
R/AllGenerics.R: 90.24%
R/genesPerGene.R: 90.32%
R/cond.R: 90.91%
R/fromSizePathways.R: 92.50%
R/add.R: 92.59%
R/pathway.R: 94.64%
R/remove.R: 95.65%
R/nested.R: 97.14%
R/duplicated.R: 100.00%
R/estimate_sizes.R: 100.00%
R/fromSizeGenes.R: 100.00%
R/gene.R: 100.00%
R/genesPerPathway.R: 100.00%
R/modify.R: 100.00%
R/n_functions.R: 100.00%
R/sizes.R: 100.00%
R/utilities.R: 100.00%
R/zzz.R: 100.00%When the test is set to pass, and it fails when a test fails. |
|
is your package on a github repo that i can clone and test? |
|
Yes, it is llrs/GSEAdv. Although the specific test file is a new one I created and I haven't pushed yet (I can't in my current network), and as it is on github it doesn't pass the Let me know if I have to do something. |
|
ok. i'll check back in on Friday to test the commit. I have had a few other people test out the current build of coverage_skip and it seems to work for them. |
|
@yonicd I updated the package in llrs/GSEAdv, the file I change to check the function covrpage is the test on By the way, I would like to propose this package to check Bioconductor's packages in their severs. But they only accept packages available on CRAN. Do you plan to submit it on CRAN? |
|
i will check why it is failing on that repo. I am planning on submitting to CRAN when I get a chance. thanks! |
> biocLite("GSEAdv")
BioC_mirror: https://bioconductor.org
Using Bioconductor 3.7 (BiocInstaller 1.30.0), R 3.5.1 (2018-07-02).
Installing package(s) ‘GSEAdv’
Old packages: 'digest', 'tinytex', 'batchtools', 'broom', 'callr', 'dbplyr',
'devtools', 'energy', 'fansi', 'foreign', 'fs', 'ggplotify', 'ggtern', 'haven',
'huxtable', 'later', 'leaflet', 'officer', 'openssl', 'pander', 'pkgconfig',
'plotly', 'processx', 'RcppArmadillo', 'reticulate', 'rlang', 'roxygen2', 'scales',
'sinew', 'streamR', 'subprocess', 'survival', 'sys', 'texPreview', 'usethis',
'XML', 'xtable'
Update all/some/none? [a/s/n]:
n
Warning message:
package ‘GSEAdv’ is not available (for R version 3.5.1) SessionInfo> devtools::session_info()
Session info ------------------------------------------------------------------------
setting value
version R version 3.5.1 (2018-07-02)
system x86_64, darwin15.6.0
ui RStudio (1.2.942)
language (EN)
collate en_US.UTF-8
tz America/New_York
date 2018-09-01
Packages ----------------------------------------------------------------------------
package * version date source
base * 3.5.1 2018-07-05 local
BiocInstaller * 1.30.0 2018-05-01 Bioconductor
compiler 3.5.1 2018-07-05 local
datasets * 3.5.1 2018-07-05 local
devtools 1.13.5 2018-02-18 CRAN (R 3.5.0)
digest 0.6.15 2018-01-28 CRAN (R 3.5.0)
graphics * 3.5.1 2018-07-05 local
grDevices * 3.5.1 2018-07-05 local
memoise 1.1.0 2017-04-21 CRAN (R 3.5.0)
methods * 3.5.1 2018-07-05 local
packrat 0.4.9-3 2018-06-01 CRAN (R 3.5.0)
stats * 3.5.1 2018-07-05 local
tools 3.5.1 2018-07-05 local
utils * 3.5.1 2018-07-05 local
withr 2.1.2 2018-03-15 CRAN (R 3.5.0) |
|
Yes, the GSEAdv package is not yet on Bioconductor, that's why that code block was commented. |
|
seems ok on my end. test_keeps.Rcontext("keep.R")
test_that("keepGPP works", {
})
test_that("keepGPP works", {
out <- keepGPP(Info)
expect_equal(ncol(out), 5L)
})covrpage::coverage_skip()> covrpage::coverage_skip()
✔ | OK F W S | Context
✖ | 0 3 | Testing adding method
──────────────────────────────────────────────────────────────────────────────────────
test_add.R:4: error: gene as character
unused arguments (gene = "2", pathway = "1430728")
1: .handleSimpleError(function (e)
{
handled <<- TRUE
test_error <<- e
options(expressions = expressions_opt_new)
on.exit(options(expressions = expressions_opt), add = TRUE)
e$expectation_calls <- frame_calls(11, 2)
test_error <<- e
register_expectation(e)
e$handled <- TRUE
test_error <<- e
}, "unused arguments (gene = \"2\", pathway = \"1430728\")", quote(add(Info, gene = "2",
pathway = "1430728"))) at tests/testthat/test_add.R:4
2: eval(code, test_env)
test_add.R:15: error: pathway as character
unused arguments (gene = c("2", "5", "7"), pathway = "156581")
1: .handleSimpleError(function (e)
{
handled <<- TRUE
test_error <<- e
options(expressions = expressions_opt_new)
on.exit(options(expressions = expressions_opt), add = TRUE)
e$expectation_calls <- frame_calls(11, 2)
test_error <<- e
register_expectation(e)
e$handled <- TRUE
test_error <<- e
}, "unused arguments (gene = c(\"2\", \"5\", \"7\"), pathway = \"156581\")", quote(add(Info,
gene = c("2", "5", "7"), pathway = "156581"))) at tests/testthat/test_add.R:15
2: eval(code, test_env)
test_add.R:31: error: pathway as character
unused arguments (gene = "2", pathway = c("156581", "211"))
1: expect_warning(gsc <- add(Info, gene = "2", pathway = c("156581", "211")), "Removing") at tests/testthat/test_add.R:31
2: quasi_capture(enquo(object), capture_warnings, label = label)
3: capture(act$val <- eval_bare(get_expr(quo), get_env(quo)))
4: withCallingHandlers(code, warning = function(condition) {
out$push(condition)
invokeRestart("muffleWarning")
})
5: eval_bare(get_expr(quo), get_env(quo))
──────────────────────────────────────────────────────────────────────────────────────
✔ | 8 | Testing list to GeneSetCollection method
✔ | 1 | calc.nPathways
✔ | 1 | Testing check-GSC methods
✔ | 1 | Testing utilities method
✔ | 16 | Testing cond* method
✔ | 22 | Testing drop method [0.1 s]
✔ | 6 | Testing duplications
✔ | 2 | estimate.n*
✔ | 3 | Testing fromSizeGenes_sizePathways
✔ | 2 | Testing fromSizeGenes method
✔ | 2 | Testing fromSizePathways method
✔ | 7 | Testing gene method
✔ | 6 | Testing genePerGene
✔ | 5 | genesPerPathway
✔ | 2 | Estimating numbers
✖ | 0 1 1 | keep.R
──────────────────────────────────────────────────────────────────────────────────────
test_keeps.R:3: skip: keepGPP works
Empty test
test_keeps.R:9: failure: keepGPP works
ncol(out) not equal to 5.
1/1 mismatches
[1] 6 - 5 == 1
──────────────────────────────────────────────────────────────────────────────────────
✔ | 12 | Testing modify method
✔ | 8 | Testing nested method
✔ | 7 | Testing pathway method
✔ | 13 | Testing simulations to create GeneSetCollections [0.3 s]
✔ | 22 | Testing sizePathways and sizeGenes
✔ | 1 | Testing summary methodnd sizeGenes
✔ | 21 | Testing utilities method
══ Results ═══════════════════════════════════════════════════════════════════════════
Duration: 1.1 s
OK: 174
Failed: 4
Warnings: 0
Skipped: 1
GSEAdv Coverage: 78.44%
R/expand.R: 0.00%
R/summary.R: 0.00%
R/pathwaysPerGene.R: 14.29%
R/calc-nPathways.R: 33.33%
R/simulateGSC.R: 39.26%
R/check-GSC.R: 71.43%
R/keep.R: 72.00%
R/simulate2GSC.R: 78.38%
R/fromSizeGenes_sizePathways.R: 78.57%
R/as_GeneSetCollection.R: 83.33%
R/sizesPer.R: 86.11%
R/AllGenerics.R: 86.67%
R/genesPerGene.R: 90.32%
R/cond.R: 90.91%
R/fromSizePathways.R: 92.50%
R/add.R: 92.59%
R/pathway.R: 94.64%
R/nested.R: 97.14%
R/remove.R: 97.67%
R/completness.R: 100.00%
R/duplicated.R: 100.00%
R/estimate_sizes.R: 100.00%
R/fromSizeGenes.R: 100.00%
R/gene.R: 100.00%
R/genesPerPathway.R: 100.00%
R/modify.R: 100.00%
R/n_functions.R: 100.00%
R/sizes.R: 100.00%
R/utilities.R: 100.00%
R/zzz.R: 100.00%covrpage.md outputTests and Coverage02 September, 2018 08:52:33 This output is created by CoverageCoverage summary is created using the
Unit TestsUnit Test summary is created using the
Show Detailed Test Results
Session Info
|
|
fork of your repo: https://github.com/yonicd/GSEAdv |
|
Yes, on that commit there isn't any test failing. |
|
There are failing tests in that commit |
|
The covrpage report says that there are some errors, but the CI reports that all the test are passing, and the covr report is generated. |
|
Not sure I follow. covrpage (which is local): 3 errors + 1 fail how are you running local tests? this is getting farther from the original issue. maybe for the sake of length i'll close this one and you can open a fresh one with the updated problem. |
|
I was running the tests with |
I was trying the package in another package where I still have some failing test (work in progress..). Could it be possible to generate the page even if a test failed?
I think it uses
test_check(in case of using testthat), which has the argument ofstop_on_failureset to TRUE by default. Changingstop_on_failureto FALSE would solve the issue, but maybe there is a reason behind this.The text was updated successfully, but these errors were encountered: