Skip to content

Commit

Permalink
breaking up README into vignettes
Browse files Browse the repository at this point in the history
  • Loading branch information
dgkf committed Mar 27, 2024
1 parent 50fd719 commit 5227e6d
Show file tree
Hide file tree
Showing 13 changed files with 263 additions and 262 deletions.
2 changes: 2 additions & 0 deletions .Rbuildignore
Original file line number Diff line number Diff line change
Expand Up @@ -2,3 +2,5 @@
codecov\.yml
LICENSE\.md
pkgdown
docs

1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1 +1,2 @@
docs
inst/doc
8 changes: 6 additions & 2 deletions DESCRIPTION
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
Package: testex
Title: Add tests and assertions in-line in examples
Title: Add Tests to Examples
Version: 0.1.0
Authors@R:
c(
Expand All @@ -14,6 +14,7 @@ Description:
facilitating easier test writing in Rd files. However, a more familiar
interface is provided using 'roxygen2' tags. Tools are also provided for
facilitating package configuration and use with 'testthat'.
URL: https://github.com/dgkf/testex, https://dgkf.github.io/testex
License: MIT + file LICENSE
Depends:
R (>= 3.2.0)
Expand All @@ -24,8 +25,11 @@ Suggests:
withr,
callr,
roxygen2,
spelling
spelling,
knitr,
rmarkdown
Encoding: UTF-8
Roxygen: list(markdown = TRUE)
RoxygenNote: 7.3.1
Language: en-US
VignetteBuilder: knitr
4 changes: 3 additions & 1 deletion NEWS.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
# testex (dev)
# testex (development)

# testex 0.1.0

* Initial CRAN submission
29 changes: 16 additions & 13 deletions R/testthat.R
Original file line number Diff line number Diff line change
Expand Up @@ -24,19 +24,22 @@
#' 1 + 2
#'
#' # within `testthat_block`, test code refers to previous result with `.`
#' testthat_block(test_that("addition holds up", {
#' expect_equal(., 3)
#' })
#'
#' # `with_srcref` to spoof the source of the code that caused the failure
#' test_that("test failure is spoofed to report error at abc.R:1:0", {
#' with_srcref("abc.R:1:3", expect_equal(1, 2))
#' })
#'
#' # `expect_no_error`
#' test_that("expectation runs without error", {
#' expect_no_error(stop("whoops!"))
#' })}
#' testthat_block({
#' test_that("addition holds up", {
#' expect_equal(., 3)
#' })
#'
#' # `with_srcref` to spoof the source of the code that caused the failure
#' test_that("test failure is spoofed to report error at abc.R:1:0", {
#' with_srcref("abc.R:1:3", expect_equal(1, 2))
#' })
#'
#' # `expect_no_error`
#' test_that("expectation runs without error", {
#' expect_no_error(stop("whoops!"))
#' })
#' })
#' }
#'
#' @name testex-testthat
NULL
Expand Down
233 changes: 16 additions & 217 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,235 +37,34 @@ context for testing your examples. And if you aren't using `testthat`, then
you'll find that your tests are being run with your examples when you run
`R CMD check`

## Details
## `roxygen2` tags

`testex` provides two new `roxygen2` tags: `@test` and `@testthat`.
### `@test`

- `@test` will check that the result of the last example is identical to
your test. You can use the example output in a function using a `.`.

```r
#' @examples
#' sum(1:10)
#' @test 55
#' @test is.numeric(.)
```

- `@testthat` is similar, but has the added benefit of automatically inserting
a `.` into `testthat::expect_*` functions.

```r
#' @examples
#' sum(1:10)
#' @testthat expect_equal(55)
#' @testthat expect_vector(numeric())
```

## Setup

To make the most of `testex`, there are a few configuration steps you might
consider. These are made simple by using:

```r
testex::use_testex()
```

which will :

- [x] Add `packages = "testex"` to the `Roxygen` field in `DESCRIPTION`,
allowing `roxygen2` to make use of the `testex` tags.
- [x] Add `testex` as a `Suggests` dependency
- [x] Add settings to the `Config/testex/options` field in `DESCRIPTION`,
enabling example tests during `R CMD check` by default.
- [x] Add a `test-testex.R` test file if you're using `testthat`,
enabling example tests during `testthat` test evaluation.

### Running tests with `testthat`

Running tests using `testthat` is simple. Just use

```r
testex::use_testex_as_testthat()
```

This will add a simple, one-line file to your `tests/testthat` directory
containing

```r
testex::test_examples_as_testthat()
```

By adding this single line to a `testthat` test file (such as
`tests/testthat/test-testex.R`), your example tests will be included as part
of your test suite.

When run this way, `testex` tests are embedded with additional metadata
including the original file location of the examples so that `testthat` is
able to provide more informative error messages.

### Disabling example checks during `R CMD check`

By default, your tests will run when your run examples using `R CMD check`.
However, `R CMD check` will stop on the first error and truncates error output,
which can be inconvenient for debugging. If you'd prefer not to run tests
during checking, you can add the following line to your `DESCRIPTION`.

```
Config/testex/options: list(check = FALSE)
```

## Goals

R offers some pretty outstanding tools for presenting example code alongside a
package. `testex` aims to make it a bit more powerful by giving you a convenient
shorthand for writing tests and easier ability to customize how they are
executed.

* Example tests that integrate nicely with popular tools. For those that like to
write Rd files by hand and those that like to generate documentation via
`roxygen2`.

* Choose where and how your examples are tested - whether you prefer to test
through `R CMD check` or other mechanisms.

### Use Cases

Since R packages may mix and match tools (`roxygen2`, `testthat`, etc.) to suit
their development needs, a mechanism of testing examples should also be flexible
enough to accommodate any development style. Priority workflows include

* Handwritten Rd documentation
Probably the least common use-case, but a priority none-the-less since it
stress tests the goal of delivering a workflow-agnostic testing pattern.

* `roxygen2`-generated Rd documentation
It's hard to find a recent package that doesn't use `roxygen2` for generating
documentation. As such, any tools should integrate nicely.

* `testthat` testing
Choose whether you want to treat example tests as tests during `R CMD check`s
of examples or during your additional testing steps. Use `testthat` to
execute example tests and integrate the results with your test suite.

All paths lead back to `Rd` files as a common ground. Whatever tool you prefer
to use, using this central format as the basis for testing means that `testex`
can accommodate your workflow.

## Under the hood

Before we jump into what `testex` offers, let's take a look at the underlying
tools that it builds on top of.

### Base building-blocks

If you'd like to add testing code to examples in your documentation, you can use
the `\testonly` block to add code which only executes when testing examples.
Pretty neat!

```latex
\examples{
identity("hello, world")
\testonly{
stopifnot(.Last.value == "hello, world")
}
}
```

Here we use `.Last.value` to grab the result of our last example and test it
against an expected value. Though, as you might expect, you can't easily add
_another_ test because `.Last.value` will have changed. This is where `testex`
comes in to help simplify things.

```latex
\examples{
identity("hello, world")
\testonly{testex::testex(
is.character(.),
. == "hello, world")
)}
}
```

Already `testex` is doing a bit of work to make our lives easier. The
`.Last.value` is propagated to each of the tests and we can use the convenient
shorthand `.` to refer to the value we want to test.

### Use a `roxygen2` tag!

If you're already using `roxygen2`, then things get even easier! `roxygen2`
can make use of new tags provided by `testex`:
will check that the result of the last example is identical to your test. You
can use the example output in a function using a `.`.

```r
#' Hello, World!
#'
#' @examples
#'
#' hello("World")
#' @test "Hello, World!"
#'
#' hello("darkness my old friend")
#' @test grepl("darkness", .)
#'
#' @export
hello <- function(who) {
paste0("Hello, ", who, "!")
}
#' sum(1:10)
#' @test 55
#' @test is.numeric(.)
```

After running `roxygen2::roxygenize()`, you can take a peak at the `Rd` files
and see how the code has been translated to `testex` tests.
### `@testthat`

### Leverage `testthat` expectations

A convenience tag is also provide for those that prefer the `testthat` style of
testing. `testthat` provides a wealth of expectation functions, which can be used
in conjunction with `testex` to write more familiar tests.
is similar, but has the added benefit of automatically inserting a `.` into
`testthat::expect_*` functions.

```r
#' Hello, World!
#'
#' @examples
#'
#' hello("World")
#' @testthat expect_equal("Hello, World!")
#'
#' hello("testthat my old friend")
#' @testthat expect_match("testthat")
#'
#' @export
hello <- function(who) {
paste0("Hello, ", who, "!")
}
#' sum(1:10)
#' @testthat expect_equal(55)
#' @testthat expect_vector(numeric())
```

The `@testthat` tag will automatically insert the `.Last.value` from the
previous example into the first argument of each expectation. Multiple
consecutive `@testthat` expectations will all test the previous example output.

## Planned Features

| | |
|---|---|
| Example result propagation using `testex::testex()`| :ballot_box_with_check: |
| `DESCRPTION` `Config/testex/options` to disable execution during `R CMD check` | :ballot_box_with_check: |
| `roxygen2` tag `@test` | :ballot_box_with_check: |
| `roxygen2` tag `@testthat` | :ballot_box_with_check: |
| Aggregation with `testthat` test results | :ballot_box_with_check: |
| Add stable Rd-file API* | :thought_balloon: |
| Other ideas? Request a feature! | :thought_balloon: |
| Have a better name for the package? I'm all ears! | :ear: |

## Prior Art

I stumbled across the awesome [`roxytest`](https://github.com/mikldk/roxytest)
package when searching around for a solution like this. That package takes a bit
of a different approach, embedding test _writing_ alongside examples. The test
tags that are offered in that package are used to generate `tinytest` and
`testthat` tests from in-line tests but otherwise tests are effectively
independent from the code that goes on to live in an example.

For me, this didn't quite solve the problem of ensuring that package examples
continue to function as expected. I wanted to explore whether the code in the
examples could be tested _directly_ with more extensive integration with the
existing `examples` tools to decouple the testing from `roxygen2` (or any
testing suite) specifically.
- [`roxytest`](https://github.com/mikldk/roxytest)
A slightly different approach. Allows tests to be written in-line, but
generates test files used directly by a testing framework.
2 changes: 1 addition & 1 deletion inst/pkg.example/R/fn.R
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ fn_roxygen <- function(x) {
#' Test Function
#'
#' This example introduces `testthat`-style tests using in-line `@testthat`
#' roxygen tags.
#' `roxygen2` tags.
#'
#' @param x A thing
#'
Expand Down
29 changes: 16 additions & 13 deletions man/testex-testthat.Rd

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

6 changes: 6 additions & 0 deletions pkgdown/_pkgdown.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,15 @@ url: ~
template:
bootstrap: 5
theme: arrow-light #atom-one-light

bslib:
fg: "#404040"
bg: "#F4F4F4"
navbar-dark-active-color: var(--bs-fg)
navbar-dark-color: var(--bs-fg)
navbar-dark-brand-color: var(--bs-fg)
navbar-dark-brand-hover-color: white
navbar-dark-hover-color: white
primary: "#9A1FF4"

navbar:
Expand Down
Loading

0 comments on commit 5227e6d

Please sign in to comment.