Incorrect package coverage

[mjakubczak]

I use covr::package_coverage() to get the coverage of this package. Everything worked properly, the coverage was ~88% and the output was correct:

> covr::package_coverage()
FutureManager Coverage: 88.16%
R/main.R: 84.62%
R/utils.R: 87.23%
R/methods.R: 93.75%
R/button.R: 93.88%
R/progress.R: 100.00%

I added some new unit tests in test-integration2.R file and the coverage dropped to ~14% with a weird output:

> covr::package_coverage()
FutureManager Coverage: 14.02%
R/button.R: 0.00%
R/main.R: 0.00%
R/progress.R: 0.00%
R/utils.R: 63.83%
R/methods.R: 93.75%

Once in a while (I could not find stable reproduction steps) it says:

> covr::package_coverage()
FutureManager Coverage: 94.39%
R/button.R: 91.84%
R/main.R: 92.86%
R/methods.R: 93.75%
R/progress.R: 100.00%
R/utils.R: 100.00%

And this output looks like it should, but it's a rare result. Any suggestions what may be the reason? It's definitely something in test-integration2.R, because when I disable this file the coverage is ~88% again.

[jimhester]

There is an open issue about coverage for shiny applications (#277) I would generally not expect it to work out of the box currently.

If you want to try and investigate it further you can use covr::package_coverage(quiet = FALSE, clean = FALSE) and then inspect your R sessions temp directory (tempdir()) for the output files.

[llrs]

I have a similar problem on ropensci/rtweet, which doesn't use shiny. The local report doesn't increase the coverage of some functions that I have just added test for them (using R 3.6.3/4.0.1, testthat 3.0, RStudio 1.4.1103), but on the github action the report is much higher than on the local computer. Tried installing the package and covr::report it again, but didn't solve the issue.

[StatisMike]

The same problem seems to be happening in shiny.reglog. Tests are written using shinytest2 package. Coverage seems to be inconsistent. Until I introduced additional test file test-mongo_app.R everything seemed fine. After its introduction, even after commenting it out - coverage is still inconsistent.

Interestingly, introducing app$stop() at the end of end scripts seemed to make the problem more consistent. After its removal, I've gotten one correct coverage output, afterwards github action depending on codecov returned wrong result again.

[StatisMike]

@jimhester Do you happen to know how to read trace files retained with covr::package_coverage(quiet = FALSE, clean = FALSE)

[jimhester]

Yes, covr writes the trace files on an on.exit handler, so R needs to be given time to quit rather than being issued a sigkill.

In https://github.com/rstudio/shinytest2/blob/08c4060609d5ee9b52278521a08cbf4e0593ffb4/R/app-driver-stop.R#L30-L34 it seems there is a brief wait, but likely it is not long enough. You really need to avoid SIGKILL completely to have reproducible results. I would suggest opening a shinytest2 issue about it if you are interested in this feature.