Vignette on using testthat outside the context of a package

[jennybc]

I taught "writing functions" today and showed a bit of testthat. Which overalls feels like a good thing.

But I feel there's little guidance on how to use testthat on functions that are not (yet) in a package.

Do you make a .R file that sources the target functions and proceeds to execute lots of expectations, possibly gathered into tests? How do you trigger testing? It's tempting to tell someone that their need to do this suggests the function and its tests should be in a package. Once you've given that advice though, you still need to do something today. But exactly what?

Another use case extracted from a student question: An organization has lots of existing scripts. Not necessarily using functions, not necessarily very modular, etc etc. It is what it is. How can you use testthat to start getting more control and visibility into the output of these scripts? Especially, is the output changing as a result of people tinkering with the scripts or upgrading R or upgrading packages? I suspect there's a workflow here using reference objects and reference files.

[hadley]

I think our current advice is to make a package. In the near future we might be able to recommend making an analysis package instead. Either way, it's just matter of running a helper function and then putting your R scripts in R/

[cboettig]

@jennybc This is potentially related to the approach of testrmd? I think having just a .Rmd file is the natural step before having a package, so I like the approach of writing tests into the .Rmd. (Once one is creating more complicated directory structure with multiple test scripts or something I think one may as well use some from of package structure, even if just a more lightweight analysis package a la checkers)

[czeildi]

Coming from industry I feel there is place for tests in a poject which is more complex than an rmd file but not a package. Although I certainly should make more packages I do not think everything should be a package.

Although I know that technically you need very little for a project to be a package for me a package has clear purpose, more limited scope than a bigger research/analysis project and main intended use is after installation.

On the other hand the quality assurance brought by testing is needed very early on. Some projects will be abandoned before reaching a package phase others could be split into several small packages but it is not clear from the beginning. I also don't necessarily want to think about what functions to export, when to increase versions etc.

Regarding advice: imo most of it applies equally to packages and not packages:

• test_dir and auto_test are both great and can be used in any project
• generally one expectation per test
• one context per file
• write test at least after something failed
• write unit tests: fast and no external resource needed
• for some functions in my_funs.R create a test file named test-my_funs.R and put these in two separate folders
• if there are tests they should be green before every commit but at least push

This last point what I missed guidance about the most, how to process test summary results automatically, like for a CI: single function call to determine whether all tests has passed. Do you think such thing has a place in testthat?

I am curious about the mentioned concept of an analysis package, is there any resource I can read?

[jennybc]

For context, @hadley, there was a conversation in rOpenSci Slack re: how to use testthat outside of a package and I encouraged the participants to weigh in here.

[srvanderplas]

I would also echo the desire to use testthat outside of a package context. Many of my analyses are not in packages (unnecessary overhead) but could greatly benefit from testing.

[cboettig]

Re testing for 'analysis packages' and non-packages, there's so much tooling around R packages that I think using that approach actually reduces overhead. For instance, r-travis will automatically install dependencies if it sees them listed in a DESCRIPTION file.

For example, here's a template I use to have travis test that any .Rmd files in a student's repo can be knit successfully: https://github.com/cboettig/compendium .

[hadley]

I think it's suboptimal to think about packages as adding overhead - they add conventions, which with appropriate tooling, reduces overhead. At some level, all a package means is to put your R code in a directory called R/ and then add some metadata in a DESCRIPTION file.

[krlmlr]

One major hassle is that subdirectories of R/ are not supported. This is impractical both for packages and analysis projects.

Sourcing the files manually works for both devtools::load_all() and building + installing the package, I have whipped up a small demonstration project: https://github.com/krlmlr/pkg.subdir.test . We'd still need to add roxygen2 support.

[rolweber]

My task is to assemble an environment with many R packages from Anaconda, and some from CRAN. I would like to run tests that check for the presence of certain packages, and later maybe some very basic tests that selected packages work well together. That's to detect when something is dropped accidentally or some incompatibility is introduced through package updates in the future.

I have that setup established for Python, now I want to achieve the same for R. But being new to R, I don't even know how to run testthat tests without devtools or RStudio. There is no code of my own to put in a package... do I have to create an empty dummy package, just to put the tests in there?

[jnolis]

As another data point for this, my team has a situation where we have a small git repository that uses R with plumber to create an API endpoint that runs a R keras model. Our team is used to doing testing for Java and we need to figure out how to get the automatic testing to work with R. testthat seems like a good solution, however turning the repository into an R package feels like a lot of overhead. This is especially true with all the non-R stuff that has to live in the repo (the dockerfile, the jenkinsfile, the linux stuff to get plumber to work with https etc).

[rolweber]

@jnolis: It's actually quite simple to use the API, once you know how to do it:

library('testthat')

testthat::test_dir('tests/testthat/')

This runs all the tests in directory tests/testthat/, relative to the current working directory.
Testcases in that directory are based on this example:
https://github.com/r-lib/testthat/blob/master/tests/testthat/test-compare.R

I'm calling an R script run-tests.r containing just the two statements above, like this:

Rscript --verbose run-tests.r

Hope this helps.

[MeyerGrace]

I was looking for information on this because I have an analysis script which is getting complicated and has the potential to keep growing. I thought that this was a good first step to ensuring reproducible results and making me be purposeful in the file structure.

I found a nice blog on this ( https://katherinemwood.github.io/post/testthat/) which I took as a guide and implemented but I think it would be nice to have a recommended "light touch" framework.

Why I'm not making a package now:

• it might not be a useful analysis in the future
• I don't make packages much so would have to re-learn and tinker
• my company has no internal packages so it adds overhead to share/audit at this moment.

[hadley]

I'm going to close this issue, because I am not convinced that creating a package "adds overhead". All it requires is that you put your R files in one directory, your test files in another, and have a file that describes what packages are needed to run your code.

Nevertheless, if you believe strongly that a package is too much overhead, I think it is still possible to use testthat by calling test_file() etc yourself, and I encourage you to document whatever pattern you come up with elsewhere.

[rolweber]

It's definitely possible to use testthat by calling test_file() and friends, see my snippet in a comment above. The problem is that it's awfully hard to find documentation or examples on how to call those. Everything I found was about testing a package. I had to dig into the source code to come up with that snippet. But with this issue and the comments around, I hope others will have an easier start.

[jnolis]

To build on that, things like having helper functions that have to live in specific folders to be loaded by test_file may be unintuitive for people who aren't used to making packages (like me).

Right now all our my team's docker containers have a test.R file that sources some scripts, creates some helper functions, then runs tests all within the file. If the tests fail the build stops. It seems like I should be able to do something more elegant by using a function like test_file, but it's hard for me to tell what the trade offs are, or if it's even possible without a large restructuring of a code base (that still needs to work with docker).

If I'm having a lot of trouble trying to understand this, I suspect there are lots of other people who would benefit too!

[ldanai]

I found this useful for setting up unit testing for an analysis project

library(usethis)
library(testthat)

# Create test files and directories
use_directory("./tests/")
file_create("./tests/test-my-analysis-1.R")
file_create("./tests/test-my-analysis-2.R")

# run a test file
test_file("./tests/test-my-analysis-1.R")

# run all test files in a directory (filenames must start with "test-", otherwise it won't work)
test_dir("./tests/")

[jzadra]

I've had success running test_file() on standalone R scripts, however when I run it on an Rmd I get Error in parse(con, n = -1, srcfile = srcfile, encoding = "UTF-8") : attempt to use zero-length variable name

Should test_file() work on an Rmd? When I take the code from a .R file (where test_file() works)and put it into a single chunk in an Rmd, it still fails with that error.

As an aside, I use testthat for quality assurance and to check that my assumptions of existing data are still met when additional data is added over time and the script is re-run.