Skip to content
Choose a tag to compare

testthat 2.0.0

@hadley hadley released this
· 744 commits to master since this release
Choose a tag to compare

Breaking API changes

  • "Can't mock functions in base packages": You can no longer use with_mock()
    to mock functions in base packages, because this no longer works in
    R-devel due to changes with the byte code compiler. I recommend using
    mockery or mockr instead.

  • The order of arguments to expect_equivalent() and expect_error() has
    changed slightly as both now pass ... on another function. This reveals
    itself with a number of different errors, like:

    • 'what' must be a character vector
    • 'check.attributes' must be logical
    • 'tolerance' should be numeric
    • argument is not interpretable as logical
    • threw an error with unexpected class
    • argument "quo" is missing, with no default
    • argument is missing, with no default

    If you see one of these errors, check the number, order, and names of
    arguments to the expectation.

  • "Failure: (unknown)". The last release mistakenly failed to test
    bare expectations not wrapped inside test_that(). If you see "(unknown)"
    in a failure message, this is a failing expectation that you previously
    weren't seeing. As well as fixing the failure, please also wrap inside
    a test_that() with an informative name.

  • "Error: the argument has already been evaluated": the way in which
    expectations now need create labels has changed, which caused a couple
    of failures with unusual usage when combined with Reduce, lapply(),
    and Map(). Avoid these functions in favour of for loops. I also recommend
    reading the section below on quasiquotation support in order to create more
    informative failure messages.


New and improved expectations

  • expect_condition() works like expect_error() but captures any
    condition, not just error conditions (#621).

  • expect_error() gains a class argument that allows you to make an
    assertion about the class of the error object (#530).

  • expect_reference() checks if two names point to the same object (#622).

  • expect_setequal() compares two sets (stored in vectors), ignoring
    duplicates and differences in order (#528).

New and improved skips

  • skip_if() makes it easy to skip a test when a condition is true (#571).
    For example, use skip_if(getRversion() <= 3.1) to skip a test in older
    R versions.

  • skip_if_translated() skips tests if you're running in an locale
    where translations are likely to occur (#565). Use this to avoid
    spurious failures when checking the text of error messages in non-English

  • skip_if_not_installed() gains new minimum_version argument (#487, #499).

Known good values

We have identified a useful family of expectations that compares the results of an expression to a known good value stored in a file. They are designed to be use in conjunction with git so that you can see what precisely has changed, and revert it if needed.

  • expect_known_output() replaces expect_output_file(), which has
    been soft-deprecated. It now defaults to update = TRUE and warn, rather
    than failing on the first run. It gains a print argument to automatically
    print the input (#627). It also sets the width option to 80 to ensure
    consistent output across environments (#514)

  • expect_known_value() replaces expect_equal_to_reference(), which
    has been soft-deprecated. It gains an update argument defaulting to TRUE.
    This changes behaviour from the previous version, and soft-deprecated
    expect_equal_to_reference() gets update = FALSE.

  • expect_known_failure() stored and compares the failure message from
    an expectation. It's a useful regression test when developing informative
    failure messges for your own expectations.

Quasiquotation support

All expectations can now use unquoting (#626). This makes it much easier to generate informative failure messages when running tests in a for loop.

For example take this test:

f <- function(i) if (i > 3) i * 9 else i * 10

for (i in 1:5) {
  expect_equal(f(i), i * 10)

When it fails, you'll see the message Error: `f(i)` not equal to `i * 10` .
That's hard to diagnose because you don't know which iteration caused the problem!

for (i in 1:5) {
  expect_equal(f(!!i), !!(i * 10))

If you unquote the values using !!, you get the failure message `f(4L)` not equal to 40.. This is much easier to diagnose! See ?quasi_label() for more details.

(Note that this is not tidy evaluation per se, but is closely related. At this time you can not unquote quosures.)

New features

Setup and teardown

  • New setup() and teardown() functions allow you to run at the start and
    end of each test file. This is useful if you want to pair cleanup code
    with the code that messes up state (#536).

  • Two new prefixes are recognised in the test/ directory. Files starting
    with setup are run before tests (but unlike helpers are not run in
    devtools::load_all()). Files starting with teardown are run after all
    tests are completed (#589).

Other new features

  • All files are now read and written as UTF-8 (#510, #605).

  • is_testing() allows you to tell if your code is being run inside a
    testing environment (#631). Rather than taking a run-time dependency on testthat
    you may want to inline the function into your own package:

    is_testing <- function() {
      identical(Sys.getenv("TESTTHAT"), "true")

    It's frequently useful to combine with interactive().

New default reporter

A new default reporter, ReporterProgress, produces more aesthetically pleasing output and makes the most important information available upfront (#529). You can return to the previous default by setting option(testthat.default_reporter = "summary").


  • Output colours have been tweaked to be consistent with clang:
    warnings are now in magenta, and skips in blue.

  • New default_reporter() and check_reporter() which returns the default
    reporters for interactive and check environments (#504).

  • New DebugReporter that calls a better version of recover() in case of
    failures, errors, or warnings (#360, #470).

  • New JunitReporter generates reports in JUnit compatible format.
    (#481, @lbartnik; #640, @nealrichardson; #575)

  • New LocationReporter which just prints the location of every expectation.
    This is useful for locating segfaults and C/C++ breakpoints (#551).

  • SummaryReporter recieved a number of smaller tweaks

    • Aborts testing as soon the limit given by the option
      testthat.summary.max_reports (default 10) is reached (#520).

    • New option testthat.summary.omit_dots = TRUE hides the progress dots
      speeding up tests by a small amount (#502).

    • Bring back random praise and encouragement which I accidentally dropped

  • New option testthat.default_check_reporter, defaults to "check".
    Continuous Integration system can set this option before evaluating
    package test sources in order to direct test result details to known

  • All reporters now accept a file argument on initialization. If provided,
    reporters will write the test results to that path. This output destination
    can also be controlled with the option testthat.output_file
    (#635, @nealrichardson).

Deprecated functions

  • is_null() and matches() have been deprecated because they conflict
    with other functions in the tidyverse (#523).

Minor improvements and bug fixes

  • Updated Catch to 1.9.6. testthat now understands and makes use of the package
    routine registration mechanism required by CRAN with R >= 3.4.0.

  • Better reporting for deeply nested failures, limiting the stack trace to the
    first and last 10 entries (#474).

  • Bare expectations notify the reporter once again. This is achieved by running
    all tests inside test_code() by default (#427, #498). This behaviour can be
    overridden by setting wrap = FALSE in test_dir() and friends (#586).

  • auto_test() and auto_test_package() provide hash parameter to enable
    switching to faster, time-stamp-based modification detection
    (#598, @katrinleinweber). auto_test_package() works correctly on windows

  • capture_output_lines() is now exported (#504).

  • compare.character() works correctly for vectors of length > 5 (#513, @brodieG)

  • compare.default() gains a max_diffs argument and defaults to printing
    out only the first 9 differences (#538).

  • compare.numeric() respects check.attributes() so expect_equivalent()
    correctly ignores attributes of numeric vectors (#485).

  • Output expectations (expect_output(), expect_message(),
    expect_warning(), and expect_silent()) all invisibly return the first
    argument to be consistent with the other expectations (#615).

  • expect_length() works with any object that has a length method, not
    just vectors (#564, @nealrichardson)

  • expect_match() now accepts explicit perl and fixed arguments, and adapts
    the failure message to the value of fixed. This also affects other expectations
    that forward to expect_match(), like expect_output(), expect_message(),
    expect_warning(), and expect_error().

  • expect_match() escapes special regular expression characters when printing
    (#522, @jimhester).

  • expect_message(), expect_warning() and expect_error() produce clearer
    failure messages.

  • find_test_scripts() only looks for \.[rR] in the extension
    (#492, @brodieG)

  • test_dir(), test_package(), test_check() unset the R_TESTS env var

  • test_examples() now works with installed packages as well as source
    packages (@jimhester, #532).

  • test_dir(), test_package(), and test_check() gain stop_on_failure
    and stop_on_waring arguments that control whether or not an error
    is signalled if any tests fail or generate warnings (#609, #619).

  • test_file() now triggers a gc() after tests are run. This helps
    to ensure that finalisers are run earlier (#535).

  • test_path() now generates correct path when called from within
    tools::testInstalledPackage() (#542).

  • test_path() no longer assumes that the path exists (#448).

  • test_that() calls without any expectations generate a default skip()

  • test_dir() gains load_helpers argument (#505).

  • show_failures() simply prints a failure if it occurs. This makes it easier
    to show failures in examples.

  • with_mock() disallows mocking of functions in base packages, because this
    doesn't work with the current development version of R (#553).