@hadley hadley released this Apr 25, 2016 · 304 commits to master since this release

Assets 2

Breaking changes

The expectation() function now expects an expectation type (one of "success", "failure", "error", "skip", "warning") as first argument. If you're creating your own expectations, you'll need to use expect() instead (#437).

New expectations

The expectation system got a thorough overhaul (#217). This primarily makes it easier to add new expectations in the future, but also included a thorough review of the documentation, ensuring that related expectations are documented together, and have evocative names.

One useful change is that most expectations invisibly return the input object. This makes it possible to chain together expectations with magrittr:

factor("a") %>% 
  expect_type("integer") %>% 
  expect_s3_class("factor") %>% 
  expect_length(1)

(And to make this style even easier, testthat now re-exports the pipe, #412).

The exception to this rule are the expectations that evaluate (i.e.
for messages, warnings, errors, output etc), which invisibly return NULL. These functions are now more consistent: using NA will cause a failure if there is a errors/warnings/mesages/output (i.e. they're not missing), and will NULL fail if there aren't any errors/warnings/mesages/output. This previously didn't work for expect_output() (#323), and the error messages were confusing with expect_error(..., NA) (#342, @nealrichardson + @krlmlr, #317).

Another change is that expect_output() now requires you to explicitly print the output if you want to test a print method: expect_output("a", "a") will fail, expect_output(print("a"), "a") will succeed.

There are six new expectations:

  • expect_type() checks the type of the object (#316),
    expect_s3_class() tests that an object is S3 with given class,
    expect_s4_class() tests that an object is S4 with given class (#373).
    I recommend using these more specific expectations instead of the
    more general expect_is().
  • expect_length() checks that an object has expected length.
  • expect_success() and expect_failure() are new expectations designed
    specifically for testing other expectations (#368).

A number of older features have been deprecated:

  • expect_more_than() and expect_less_than() have been deprecated. Please
    use expect_gt() and expect_lt() instead.
  • takes_less_than() has been deprecated.
  • not() has been deprecated. Please use the explicit individual forms
    expect_error(..., NA) , expect_warning(.., NA) and so on.

Expectations are conditions

Now all expectations are also conditions, and R's condition system is used to signal failures and successes (#360, @krlmlr). All known conditions (currently, "error", "warning", "message", "failure", and "success") are converted to expectations using the new as.expectation(). This allows third-party test packages (such as assertthat, testit, ensurer, checkmate, assertive) to seamlessly establish testthat compatibility by issuing custom error conditions (e.g., structure(list(message = "Error message"), class = c("customError", "error", "condition"))) and then implementing as.expectation.customError(). The assertthat package contains an example.

Reporters

The reporters system class has been considerably refactored to make existing reporters simpler and to make it easier to write new reporters. There are two main changes:

  • Reporters classes are now R6 classes instead of Reference Classes.
  • Each callbacks receive the full context:
    • add_results() is passed context and test as well as the expectation.
    • test_start() and test_end() both get the context and test.
    • context_start() and context_end() get the context.
  • Warnings are now captured and reported in most reporters.
  • The reporter output goes to the original standard output and is not affected by sink() and expect_output() (#420, @krlmlr).
  • The default summary reporter lists all warnings (#310), and all skipped
    tests (@krlmlr, #343). New option testthat.summary.max_reports limits
    the number of reports printed by the summary reporter. The default is 15
    (@krlmlr, #354).
  • MinimalReporter correct labels errors with E and failures with F (#311).
  • New FailReporter to stop in case of failures or errors after all tests
    (#308, @krlmlr).

Other

  • New functions capture_output(), capture_message(), and
    capture_warnings() selectively capture function output. These are
    used in expect_output(), expect_message() and expect_warning()
    to allow other types out output to percolate up (#410).
  • try_again() allows you to retry code multiple times until it succeeds
    (#240).
  • test_file(), test_check(), and test_package() now attach testthat so
    all testing functions are available.
  • source_test_helpers() gets a useful default path: the testthat tests
    directory. It defaults to the test_env() to be consistent with the
    other source functions (#415).
  • test_file() now loads helpers in the test directory before running
    the tests (#350).
  • test_path() makes it possible to create paths to files in tests/testthat
    that work interactively and when called from tests (#345).
  • Add skip_if_not() helper.
  • Add skip_on_bioc() helper (@thomasp85).
  • make_expectation() uses expect_equal().
  • setup_test_dir() has been removed. If you used it previously, instead use
    source_test_helpers() and find_test_scripts().
  • source_file() exports the function testthat uses to load files from disk.
  • test_that() returns a logical that indicates if all tests were successful
    (#360, @krlmlr).
  • find_reporter() (and also all high-level testing functions) support a vector
    of reporters. For more than one reporter, a MultiReporter is created
    (#307, @krlmlr).
  • with_reporter() is used internally and gains new argument
    start_end_reporter = TRUE (@krlmlr, 355).
  • set_reporter() returns old reporter invisibly (#358, @krlmlr).
  • Comparing integers to non-numbers doesn't raise errors anymore, and falls
    back to string comparison if objects have different lengths. Complex numbers
    are compared using the same routine (#309, @krlmlr).
  • compare.numeric() and compare.chacter() recieved another overhaul. This
    should improve behaviour of edge cases, and provides a strong foundation for
    further work. Added compare.POSIXt() for better reporting of datetime
    differences.
  • expect_identical() and is_identical_to() now use compare() for more
    detailed output of differences (#319, @krlmlr).
  • Added Catch v1.2.1 for unit testing of C++ code.
    See ?use_catch() for more details. (@kevinushey)