A couple of autograding tools (nbgrader and otter-grader), which can autograde R code using software unit tests in Jupyter notebooks and R Markdown, require errors to be thrown in an interactive environment. If errors are not thrown, then the autograding software incorrectly marks student code as correct.
For example, we might as a question like:
In R, calculate sin(pi /4) and name it answer.
The students would then provide some code like (answer below is intentionally incorrect):
And then we would test it to give them marks if it is correct (and in this case we wouldn't because the answer is wrong).
test_that statements worked well up until the move to version 3.0, at that point the reporter changed and test_that tests run interactively no longer threw an error. Below I contrast the two behaviours:
Behaviour of {testthat} 2.3.2 (which is what is needed for the autograders):
library(testthat)
test_that("trigonometric functions match identities", {
expect_equal(sin(pi / 4), answer)
})
print("test kept running")
output:
Error: Test failed: 'trigonometric functions match identities'
* <text>:2: sin(pi/4) not equal to `answer`.
1/1 mismatches
[1] 0.707 - 0.866 == -0.159
Traceback:
1. test_that("trigonometric functions match identities", {
. expect_equal(sin(pi/4), answer)
. })
2. test_code(desc, code, env = parent.frame())
3. get_reporter()$end_test(context = get_reporter()$.context, test = test)
4. stop(message, call. = FALSE)
Behaviour of {testthat} 3.0.0:
library(testthat)
test_that("trigonometric functions match identities", {
expect_equal(sin(pi / 4), answer)
})
print("test kept running")
output:
── Failure (<text>:2:3): trigonometric functions match identities ──────────────
sin(pi/4) not equal to `answer`.
1/1 mismatches
[1] 0.707 - 0.866 == -0.159
[1] "test kept running"
I have tried changing the reporter to MultiReporter (CheckReporter and FailReporter to give the students info and force a failure), which works, but only if a new MultiReporter is created each time the tests are called.
This works if run in one code chunk/code cell:
library(testthat)
reporter <- MultiReporter$new(list(
CheckReporter$new(),
FailReporter$new()
))
with_reporter(reporter, {
test_that("trigonometric functions match identities", {
expect_equal(sin(pi / 4), answer)
})
})
print("test kept running")
Output after running the cell many times (as a student might as they work through the problem:
══ Failed tests ══════════════════════════════════════════════════════════════════════════════════════════════════════════════════════
── Failure (<text>:10:5): trigonometric functions match identities ─────────────────────
sin(pi/4) not equal to `answer`.
1/1 mismatches
[1] 0.707 - 0.866 == -0.159
[ FAIL 1 | WARN 0 | SKIP 0 | PASS 0 ]
Error: Failures detected.
Traceback:
1. with_reporter(reporter, {
. test_that("trigonometric functions match identities", {
. expect_equal(sin(pi/4), answer)
. })
. })
2. reporter$end_reporter()
3. o_apply(self$reporters, "end_reporter")
4. lapply(objects, f)
5. FUN(X[[i]], ...)
6. x$end_reporter(...)
7. stop("Failures detected.", call. = FALSE)
But if we create the MultiReporter once, in a code cell/code chunk and then use it in another code cell/code chunk (so we do not keep repeating ourselves), the past fails and passes remain. Below I show running the same test in another code cell/code chunk 3 times incorrectly and then finally once correctly:
with_reporter(reporter, {
test_that("trigonometric functions match identities", {
expect_equal(sin(pi / 4), answer)
})
})
print("test kept running")
output:
══ Failed tests ══════════════════════════════════════════════════════════════════════════════════════════════════════════════════════
── Failure (<text>:10:5): trigonometric functions match identities ─────────────────────
sin(pi/4) not equal to `answer`.
1/1 mismatches
[1] 0.707 - 0.866 == -0.159
── Failure (<text>:3:5): trigonometric functions match identities ─────────────────────
sin(pi/4) not equal to `answer`.
1/1 mismatches
[1] 0.707 - 0.866 == -0.159
── Failure (<text>:3:5): trigonometric functions match identities ─────────────────────
sin(pi/4) not equal to `answer`.
1/1 mismatches
[1] 0.707 - 0.866 == -0.159
[ FAIL 3 | WARN 0 | SKIP 0 | PASS 1 ]
Error: Failures detected.
Traceback:
1. with_reporter(reporter, {
. test_that("trigonometric functions match identities", {
. expect_equal(sin(pi/4), answer)
. })
. })
2. reporter$end_reporter()
3. o_apply(self$reporters, "end_reporter")
4. lapply(objects, f)
5. FUN(X[[i]], ...)
6. x$end_reporter(...)
7. stop("Failures detected.", call. = FALSE)
A couple of autograding tools (nbgrader and otter-grader), which can autograde R code using software unit tests in Jupyter notebooks and R Markdown, require errors to be thrown in an interactive environment. If errors are not thrown, then the autograding software incorrectly marks student code as correct.
For example, we might as a question like:
In R, calculate sin(pi /4) and name it
answer.The students would then provide some code like (answer below is intentionally incorrect):
And then we would test it to give them marks if it is correct (and in this case we wouldn't because the answer is wrong).
test_thatstatements worked well up until the move to version 3.0, at that point the reporter changed andtest_thattests run interactively no longer threw an error. Below I contrast the two behaviours:Behaviour of {testthat} 2.3.2 (which is what is needed for the autograders):
output:
Behaviour of {testthat} 3.0.0:
output:
I have tried changing the reporter to
MultiReporter(CheckReporterandFailReporterto give the students info and force a failure), which works, but only if a newMultiReporteris created each time the tests are called.This works if run in one code chunk/code cell:
Output after running the cell many times (as a student might as they work through the problem:
But if we create the MultiReporter once, in a code cell/code chunk and then use it in another code cell/code chunk (so we do not keep repeating ourselves), the past fails and passes remain. Below I show running the same test in another code cell/code chunk 3 times incorrectly and then finally once correctly:
output: