devtools::check fails after use_catch() without (undocumented) fix

[hsloot]

I got the somewhat unexpected result after the last update of testthat to 3.0.0 that R CMD CHECK failed for my package which uses Catch due to the new test methods for compiled code and the following line in particular:

testthat/R/test-compiled-code.R

Line 34 in 45a9c70

check_installed("xml2", "run_cpp_tests()")

While this can be fixed by adding Suggests: xml2 to the DESCRIPTION file of my own package, this is undocumented and can only be discovered by getting the error or looking up the code directly. I suggest to either document this dependency in run_cpp_tests and adding a message to use_catch or placing xml2 again in the imports of testthat. As a user of the Catch-feature, the first approach seems a little unnatural to me as I am not using the xml2 package directly and it seems to be more of an implementation detail of testthat, but I understand that xml2 might be an unnecessary dependency for most testthat users.

Here is a reprex using the usethis workflow:

tmp <- tempdir()
setwd(tmp)
usethis::create_package("testcatch", open=FALSE)
#> ✓ Creating 'testcatch/'
#> ✓ Setting active project to '/private/var/folders/29/_qzzdhnn75780wtf8805n85w0000gn/T/RtmpRYD114/testcatch'
#> ✓ Creating 'R/'
#> ✓ Writing 'DESCRIPTION'
#> Package: testcatch
#> Title: What the Package Does (One Line, Title Case)
#> Version: 0.0.0.9000
#> Authors@R (parsed):
#>     * First Last <first.last@example.com> [aut, cre] (YOUR-ORCID-ID)
#> Description: What the package does (one paragraph).
#> License: `use_mit_license()`, `use_gpl3_license()` or friends to pick a
#>     license
#> Encoding: UTF-8
#> LazyData: true
#> Roxygen: list(markdown = TRUE)
#> RoxygenNote: 7.1.1
#> ✓ Writing 'NAMESPACE'
#> ✓ Setting active project to '<no active project>'
setwd("testcatch")

usethis::use_rcpp()
#> ✓ Setting active project to '/private/var/folders/29/_qzzdhnn75780wtf8805n85w0000gn/T/RtmpRYD114/testcatch'
#> ✓ Creating 'src/'
#> ✓ Adding '*.o', '*.so', '*.dll' to 'src/.gitignore'
#> ● Copy and paste the following lines into '/private/var/folders/29/_qzzdhnn75780wtf8805n85w0000gn/T/RtmpRYD114/testcatch/R/testcatch-package.R':
#>   ## usethis namespace: start
#>   #' @useDynLib testcatch, .registration = TRUE
#>   ## usethis namespace: end
#>   NULL
#> ✓ Adding 'Rcpp' to LinkingTo field in DESCRIPTION
#> ✓ Adding 'Rcpp' to Imports field in DESCRIPTION
#> ● Copy and paste the following lines into '/private/var/folders/29/_qzzdhnn75780wtf8805n85w0000gn/T/RtmpRYD114/testcatch/R/testcatch-package.R':
#>   ## usethis namespace: start
#>   #' @importFrom Rcpp sourceCpp
#>   ## usethis namespace: end
#>   NULL
#> ✓ Writing 'src/code.cpp'
usethis::use_testthat()
#> ✓ Adding 'testthat' to Suggests field in DESCRIPTION
#> ✓ Creating 'tests/testthat/'
#> ✓ Writing 'tests/testthat.R'
#> ● Call `use_test()` to initialize a basic test file and open it for editing.
testthat::use_catch()
#> > Added C++ unit testing infrastructure.
#> > Please ensure you have 'LinkingTo: testthat' in your DESCRIPTION.
#> > Please ensure you have 'useDynLib(testcatch, .registration = TRUE)' in your NAMESPACE.
writeLines("## usethis namespace: start\n#' @useDynLib testcatch, .registration = TRUE\n#' @importFrom Rcpp sourceCpp\n## usethis namespace: end\nNULL", con = "R/testcatch-package.R")
usethis::use_package("testthat", type = "LinkingTo")
#> ✓ Adding 'testthat' to LinkingTo field in DESCRIPTION
#> ● Possible includes are:
#>   #include <testthat.h>

devtools::check(document=TRUE)
#> Updating testcatch documentation
#> Loading testcatch
#> Re-compiling testcatch
#>   ─  installing *source* package ‘testcatch’ ... (470ms)
#>      ** using staged installation
#>      ** libs
#>      clang++ -std=gnu++11 -I"/usr/local/Cellar/r/4.0.3/lib/R/include" -DNDEBUG  -I'/usr/local/lib/R/4.0/site-library/Rcpp/include' -I'/usr/local/lib/R/4.0/site-library/testthat/include' -I/usr/local/opt/gettext/include -I/usr/local/opt/readline/include -I/usr/local/include  -O3 -Wall -pedantic -pipe -Weffc++ -fPIC  -g -O2  -c code.cpp -o code.o
#>      clang++ -std=gnu++11 -I"/usr/local/Cellar/r/4.0.3/lib/R/include" -DNDEBUG  -I'/usr/local/lib/R/4.0/site-library/Rcpp/include' -I'/usr/local/lib/R/4.0/site-library/testthat/include' -I/usr/local/opt/gettext/include -I/usr/local/opt/readline/include -I/usr/local/include  -O3 -Wall -pedantic -pipe -Weffc++ -fPIC  -g -O2  -c test-example.cpp -o test-example.o
#>      clang++ -std=gnu++11 -I"/usr/local/Cellar/r/4.0.3/lib/R/include" -DNDEBUG  -I'/usr/local/lib/R/4.0/site-library/Rcpp/include' -I'/usr/local/lib/R/4.0/site-library/testthat/include' -I/usr/local/opt/gettext/include -I/usr/local/opt/readline/include -I/usr/local/include  -O3 -Wall -pedantic -pipe -Weffc++ -fPIC  -g -O2  -c test-runner.cpp -o test-runner.o
#>      clang++ -std=gnu++11 -dynamiclib -Wl,-headerpad_max_install_names -undefined dynamic_lookup -single_module -multiply_defined suppress -L/usr/local/Cellar/r/4.0.3/lib/R/lib -L/usr/local/opt/gettext/lib -L/usr/local/opt/readline/lib -L/usr/local/lib -o testcatch.so code.o test-example.o test-runner.o -L/usr/local/Cellar/r/4.0.3/lib/R/lib -lR -lintl -Wl,-framework -Wl,CoreFoundation
#>      installing to /private/var/folders/29/_qzzdhnn75780wtf8805n85w0000gn/T/RtmpRYD114/devtools_install_32da458875bd/00LOCK-testcatch/00new/testcatch/libs
#>      ** checking absolute paths in shared objects and dynamic libraries
#>   ─  DONE (testcatch) (529ms)
#> Writing NAMESPACE
#> ── Building ─────────────────────────────────────────────────────── testcatch ──
#> Setting env vars:
#> ● CFLAGS    : -Wall -pedantic
#> ● CXXFLAGS  : -Wall -pedantic
#> ● CXX11FLAGS: -Wall -pedantic
#> ────────────────────────────────────────────────────────────────────────────────
#>      checking for file ‘/private/var/folders/29/_qzzdhnn75780wtf8805n85w0000gn/T/RtmpRYD114/testcatch/DESCRIPTION’ ...  ✓  checking for file ‘/private/var/folders/29/_qzzdhnn75780wtf8805n85w0000gn/T/RtmpRYD114/testcatch/DESCRIPTION’ (616ms)
#>   ─  preparing ‘testcatch’:
#>      checking DESCRIPTION meta-information ...  ✓  checking DESCRIPTION meta-information
#>   ─  cleaning src
#>   ─  checking for LF line-endings in source and make files and shell scripts
#>   ─  checking for empty or unneeded directories
#>      Removed empty directory ‘testcatch/man’
#>   ─  building ‘testcatch_0.0.0.9000.tar.gz’
#>      
#> ── Checking ─────────────────────────────────────────────────────── testcatch ──
#> Setting env vars:
#> ● _R_CHECK_CRAN_INCOMING_USE_ASPELL_: TRUE
#> ● _R_CHECK_CRAN_INCOMING_REMOTE_    : FALSE
#> ● _R_CHECK_CRAN_INCOMING_           : FALSE
#> ● _R_CHECK_FORCE_SUGGESTS_          : FALSE
#> ● NOT_CRAN                          : true
#> ── R CMD check ─────────────────────────────────────────────────────────────────
#> * using log directory ‘/private/var/folders/29/_qzzdhnn75780wtf8805n85w0000gn/T/RtmpRYD114/testcatch.Rcheck’
#> * using R version 4.0.3 (2020-10-10)
#> * using platform: x86_64-apple-darwin19.6.0 (64-bit)
#> * using session charset: UTF-8
#> * using options ‘--no-manual --as-cran’
#> * checking for file ‘testcatch/DESCRIPTION’ ... OK
#> * this is package ‘testcatch’ version ‘0.0.0.9000’
#> * package encoding: UTF-8
#> * checking package namespace information ... OK
#> * checking package dependencies ... OK
#> * checking if this is a source package ... OK
#> * checking if there is a namespace ... OK
#> * checking for executable files ... OK
#> * checking for hidden files and directories ... OK
#> * checking for portable file names ... OK
#> * checking for sufficient/correct file permissions ... OK
#> * checking serialization versions ... OK
#> * checking whether package ‘testcatch’ can be installed ... OK
#> * checking installed package size ... OK
#> * checking package directory ... OK
#> * checking for future file timestamps ... NOTE
#> unable to verify current time
#> * checking DESCRIPTION meta-information ... WARNING
#> Non-standard license specification:
#>   `use_mit_license()`, `use_gpl3_license()` or friends to pick a
#>   license
#> Standardizable: FALSE
#> * checking top-level files ... OK
#> * checking for left-over files ... OK
#> * checking index information ... OK
#> * checking package subdirectories ... OK
#> * checking R files for non-ASCII characters ... OK
#> * checking R files for syntax errors ... OK
#> * checking whether the package can be loaded ... OK
#> * checking whether the package can be loaded with stated dependencies ... OK
#> * checking whether the package can be unloaded cleanly ... OK
#> * checking whether the namespace can be loaded with stated dependencies ... OK
#> * checking whether the namespace can be unloaded cleanly ... OK
#> * checking loading without being on the library search path ... OK
#> * checking dependencies in R code ... OK
#> * checking S3 generic/method consistency ... OK
#> * checking replacement functions ... OK
#> * checking foreign function calls ... OK
#> * checking R code for possible problems ... OK
#> * checking for missing documentation entries ... OK
#> * checking line endings in C/C++/Fortran sources/headers ... OK
#> * checking pragmas in C/C++ headers and code ... OK
#> * checking compilation flags used ... NOTE
#> Compilation used the following non-portable flag(s):
#>   ‘-Weffc++’
#> * checking compiled code ... NOTE
#> File ‘testcatch/libs/testcatch.so’:
#>   Found no calls to: ‘R_registerRoutines’, ‘R_useDynamicSymbols’
#> 
#> It is good practice to register native routines and to disable symbol
#> search.
#> 
#> See ‘Writing portable packages’ in the ‘Writing R Extensions’ manual.
#> * checking examples ... NONE
#> * checking for unstated dependencies in ‘tests’ ... OK
#> * checking tests ...
#>   Running ‘testthat.R’
#>  ERROR
#> Running the tests in ‘tests/testthat.R’ failed.
#> Last 13 lines of output:
#>   > test_check("testcatch")
#>   ── �[38;5;214mError�[39m (test-cpp.R:1:1): (code run outside of `test_that()`) ─────────────────
#>   Error: The xml2 package must be installed in order to use `run_cpp_tests()`
#>   �[34mℹ�[39m Do you need to run `install.packages('xml2')`?
#>   �[1mBacktrace:�[22m
#>   �[90m    �[39m█
#>   �[90m 1. �[39m└─testthat::run_cpp_tests("testcatch") �[90mtest-cpp.R:1:0�[39m
#>   �[90m 2. �[39m  └─testthat:::check_installed("xml2", "run_cpp_tests()")
#>   
#>   ══ testthat results  ═══════════════════════════════════════════════════════════
#>   �[38;5;214mError�[39m (test-cpp.R:1:1): (code run outside of `test_that()`)
#>   
#>   [ �[38;5;214mFAIL�[39m 1 | �[35mWARN�[39m 0 | �[34mSKIP�[39m 0 | �[32mPASS�[39m 0 ]
#>   Error: Test failures
#>   Execution halted
#> * checking for non-standard things in the check directory ... OK
#> * checking for detritus in the temp directory ... OK
#> * DONE
#> 
#> Status: 1 ERROR, 1 WARNING, 3 NOTEs
#> See
#>   ‘/private/var/folders/29/_qzzdhnn75780wtf8805n85w0000gn/T/RtmpRYD114/testcatch.Rcheck/00check.log’
#> for details.
#> 
#> ── R CMD check results ─────────────────────────────── testcatch 0.0.0.9000 ────
#> Duration: 49s
#> 
#> > checking tests ...
#>   See below...
#> 
#> > checking DESCRIPTION meta-information ... WARNING
#>   Non-standard license specification:
#>     `use_mit_license()`, `use_gpl3_license()` or friends to pick a
#>     license
#>   Standardizable: FALSE
#> 
#> > checking for future file timestamps ... NOTE
#>   unable to verify current time
#> 
#> > checking compilation flags used ... NOTE
#>   Compilation used the following non-portable flag(s):
#>     ‘-Weffc++’
#> 
#> > checking compiled code ... NOTE
#>   File ‘testcatch/libs/testcatch.so’:
#>     Found no calls to: ‘R_registerRoutines’, ‘R_useDynamicSymbols’
#>   
#>   It is good practice to register native routines and to disable symbol
#>   search.
#>   
#>   See ‘Writing portable packages’ in the ‘Writing R Extensions’ manual.
#> 
#> ── Test failures ───────────────────────────────────────────────── testthat ────
#> 
#> > library(testthat)
#> > library(testcatch)
#> > 
#> > test_check("testcatch")
#> ── �[38;5;214mError�[39m (test-cpp.R:1:1): (code run outside of `test_that()`) ─────────────────
#> Error: The xml2 package must be installed in order to use `run_cpp_tests()`
#> �[34mℹ�[39m Do you need to run `install.packages('xml2')`?
#> �[1mBacktrace:�[22m
#> �[90m    �[39m█
#> �[90m 1. �[39m└─testthat::run_cpp_tests("testcatch") �[90mtest-cpp.R:1:0�[39m
#> �[90m 2. �[39m  └─testthat:::check_installed("xml2", "run_cpp_tests()")
#> 
#> ══ testthat results  ═══════════════════════════════════════════════════════════
#> �[38;5;214mError�[39m (test-cpp.R:1:1): (code run outside of `test_that()`)
#> 
#> [ �[38;5;214mFAIL�[39m 1 | �[35mWARN�[39m 0 | �[34mSKIP�[39m 0 | �[32mPASS�[39m 0 ]
#> Error: Test failures
#> Execution halted
#> 
#> 1 error x | 1 warning x | 3 notes x
#> Error: R CMD check found ERRORs

^{Created on 2020-11-16 by the reprex package (v0.3.0)}

Session info

devtools::session_info()
#> ─ Session info ───────────────────────────────────────────────────────────────
#>  setting  value                       
#>  version  R version 4.0.3 (2020-10-10)
#>  os       macOS Catalina 10.15.7      
#>  system   x86_64, darwin19.6.0        
#>  ui       unknown                     
#>  language (EN)                        
#>  collate  de_DE.UTF-8                 
#>  ctype    de_DE.UTF-8                 
#>  tz       Europe/Berlin               
#>  date     2020-11-16                  
#> 
#> ─ Packages ───────────────────────────────────────────────────────────────────
#>  ! package     * version    date       lib source        
#>    assertthat    0.2.1      2019-03-21 [1] CRAN (R 4.0.0)
#>    backports     1.2.0      2020-11-02 [1] CRAN (R 4.0.3)
#>    callr         3.5.1      2020-10-13 [1] CRAN (R 4.0.3)
#>    cli           2.1.0      2020-10-12 [1] CRAN (R 4.0.3)
#>    crayon        1.3.4      2017-09-16 [1] CRAN (R 4.0.0)
#>    desc          1.2.0      2018-05-01 [1] CRAN (R 4.0.0)
#>    devtools      2.3.2      2020-09-18 [1] CRAN (R 4.0.2)
#>    digest        0.6.27     2020-10-24 [1] CRAN (R 4.0.3)
#>    ellipsis      0.3.1      2020-05-15 [1] CRAN (R 4.0.0)
#>    evaluate      0.14       2019-05-28 [1] CRAN (R 4.0.0)
#>    fansi         0.4.1      2020-01-08 [1] CRAN (R 4.0.0)
#>    fs            1.5.0      2020-07-31 [1] CRAN (R 4.0.2)
#>    git2r         0.27.1     2020-05-03 [1] CRAN (R 4.0.0)
#>    glue          1.4.2      2020-08-27 [1] CRAN (R 4.0.2)
#>    highr         0.8        2019-03-20 [1] CRAN (R 4.0.0)
#>    htmltools     0.5.0      2020-06-16 [1] CRAN (R 4.0.2)
#>    knitr         1.30       2020-09-22 [1] CRAN (R 4.0.3)
#>    lifecycle     0.2.0      2020-03-06 [1] CRAN (R 4.0.0)
#>    magrittr      1.5        2014-11-22 [1] CRAN (R 4.0.0)
#>    memoise       1.1.0      2017-04-21 [1] CRAN (R 4.0.0)
#>    pillar        1.4.6      2020-07-10 [1] CRAN (R 4.0.2)
#>    pkgbuild      1.1.0      2020-07-13 [1] CRAN (R 4.0.2)
#>    pkgconfig     2.0.3      2019-09-22 [1] CRAN (R 4.0.0)
#>    pkgload       1.1.0      2020-05-29 [1] CRAN (R 4.0.1)
#>    prettyunits   1.1.1      2020-01-24 [1] CRAN (R 4.0.0)
#>    processx      3.4.4      2020-09-03 [1] CRAN (R 4.0.2)
#>    ps            1.4.0      2020-10-07 [1] CRAN (R 4.0.3)
#>    purrr         0.3.4      2020-04-17 [1] CRAN (R 4.0.0)
#>    R6            2.5.0      2020-10-28 [1] CRAN (R 4.0.3)
#>    rcmdcheck     1.3.3      2019-05-07 [1] CRAN (R 4.0.0)
#>    Rcpp          1.0.5      2020-07-06 [1] CRAN (R 4.0.2)
#>    remotes       2.2.0      2020-07-21 [1] CRAN (R 4.0.2)
#>    rlang         0.4.8      2020-10-08 [1] CRAN (R 4.0.3)
#>    rmarkdown     2.5        2020-10-21 [1] CRAN (R 4.0.3)
#>    roxygen2      7.1.1      2020-06-27 [1] CRAN (R 4.0.2)
#>    rprojroot     1.3-2      2018-01-03 [1] CRAN (R 4.0.0)
#>    rstudioapi    0.13       2020-11-12 [1] CRAN (R 4.0.3)
#>    sessioninfo   1.1.1      2018-11-05 [1] CRAN (R 4.0.0)
#>    stringi       1.5.3      2020-09-09 [1] CRAN (R 4.0.2)
#>    stringr       1.4.0      2019-02-10 [1] CRAN (R 4.0.0)
#>  R testcatch   * 0.0.0.9000 <NA>       [?] <NA>          
#>    testthat      3.0.0      2020-10-31 [1] CRAN (R 4.0.3)
#>    tibble        3.0.4      2020-10-12 [1] CRAN (R 4.0.3)
#>    usethis       1.6.3      2020-09-17 [1] CRAN (R 4.0.2)
#>    vctrs         0.3.4      2020-08-29 [1] CRAN (R 4.0.2)
#>    whisker       0.4        2019-08-28 [1] CRAN (R 4.0.0)
#>    withr         2.3.0      2020-09-22 [1] CRAN (R 4.0.3)
#>    xfun          0.19       2020-10-30 [1] CRAN (R 4.0.3)
#>    xml2          1.3.2      2020-04-23 [1] CRAN (R 4.0.3)
#>    xopen         1.0.0      2018-09-17 [1] CRAN (R 4.0.0)
#>    yaml          2.2.1      2020-02-01 [1] CRAN (R 4.0.0)
#> 
#> [1] /usr/local/lib/R/4.0/site-library
#> [2] /usr/local/Cellar/r/4.0.3/lib/R/library
#> 
#>  R ── Package was removed from disk.

[hadley]

@jimhester do you think adding to docs is adequate?

[jimhester]

Yeah I think adding documentation would be fine.