Use conditions for reporting results #360

krlmlr · 2016-02-05T15:27:22Z

Also: test_that() returns TRUE if successful (no failed expectations, errors, or skips) for programmatic use of testthat.

Bare expectations now simply stop if a test fails. The stop occurs upon the first event, unlike the StopReporter which stops after the first failed test.

for bare expectations

krlmlr · 2016-02-05T15:45:06Z

R/expect-that.r

+                    class = c("test_result", "condition"))
+
+  withRestarts(
+    if (results$passed) signalCondition(cond) else stop(cond),


Could also call message() instead of signalCondition() here.

krlmlr · 2016-02-05T16:41:07Z

@hadley @jimhester: PTAL. Part of #215 (comment).

- Return test results as named array of logical. Requires r-lib/testthat#360.

- New tweak `union`, for specifying a nonstandard way of combining queries. - All union queries now use full qualification for each column (required for `bigrquery`). - Return test results as named array of logical. Requires r-lib/testthat#360. - Use fork of `testthat`.

krlmlr · 2016-02-10T21:52:59Z

R/test-that.r

+
+  withRestarts(
+    {
+      signalCondition(cond)


Call stop() here.

as recommended in adv-r

Now a failing expectation always raises an error, even if the StopReporter isn't active. The update of this test reflects this change in behavior.

- New feature: tweaks - New argument `tweaks` to `make_context()` (#49). - New `tweaks()`, essentially constructs a named list of tweaks but with predefined and documented argument names. - `constructor_name`, respected by the `constructor.*` tests. - `strict_identifier`, if `TRUE` all identifier must be syntactic names even if quoted. The quoting test is now split, and a part is ignored conditional to this tweak. The `roundtrip_quotes` tests also respects this tweak. - `omit_blob_tests` for DBMS that don't have a BLOB data type. - `current_needs_parens` -- some SQL dialects (e.g., BigQuery) require parentheses for the functions `current_date`, `current_time` and `current_timestamp`. - `union`, for specifying a nonstandard way of combining queries. All union queries now name each column in each subquery (required for `bigrquery`). - New tests - `dbGetInfo(Result)` (r-dbi/DBI#55). - `dbListFields()` (#26). - New "package_name" test in `test_getting_started()`. - Improved tests - Stress test now installs package in temporary library (before loading `DBI`) using `R CMD INSTALL` before loading DBI (r-dbi/RSQLite#128, #48). - Row count is now tested for equality but not identity, so that backends can return a numeric value > 2^31 at their discretion. - Call `dbRemoveTable()` instead of issuing `DROP` requests, the latter might be unsupported. - Use subqueries in queries that use `WHERE`. - Test that `dbClearResult()` on a closed result set raises a warning. - Expect a warning instead of an error for double disconnect (#50). - Move connection test that requires `dbFetch()` to `test_result()`. - Split "can_connect_and_disconnect" test. - Expect `DBI` to be in `Imports`, not in `Depends`. - Removed tests - Remove test for `dbGetException()` (r-dbi/DBI#51). - Bug fixes - Fix broken tests for quoting. - Self-testing - Test `RPostgres`, `RMySQL`, `RSQLite` and `RKazam` as part of the Travis-CI tests (#52). - Travis CI now installs rstats-db/DBI, updated namespace imports (`dbiCheckCompliance()`, `dbListResults()`). - Use fork of `testthat`. - Utilities - Return test results as named array of logical. Requires r-lib/testthat#360, gracefully degrades with the CRAN version. - Internal - Refactored the `get_info_()` tests to use a vector of names. - Use versioned dependency for DBI - Use unqualified calls to `dbBind()` again - Same as 1.0-8.

hadley · 2016-02-18T19:22:59Z

R/test-that.r

@@ -73,6 +82,28 @@ test_code <- function(description, code, env) {
  invisible(ok)
 }

+report_results <- function(results) {


I think this would be better called report_expectation(), but I'm not sure report() is quite the right verb (but I'm not sure what would be better)

Or should this all be wrapped up into expectation()?

Do you mean that expectation() should create and raise the condition object? This means that we need to be very clever in expect_that(): Catch the condition (which is created in the condition(object) call), patch it, and re-raise it. To me, this looks like too much effort for too little gain. But I agree that expectation() should return a proper condition object right away. I'll update the PR soon.

Maybe we should have expectation() (noun) to create the object, and expect() (verb) raise the condition?

We already have expect() which accepts a logical and a message; we'll need as.expectation.logical() and as.expectation.expectation() to avoid changing too much unrelated code, but otherwise this is doable. How does that sound?

Then we'll add as.expectation.error(), as.expectation.skip() (as replacement for current expecation_error() and expectation_skipped()) and later as.expectation.warning() (#310).

krlmlr · 2016-02-19T08:56:37Z

Are you planning further refactorings? Then it would be great to merge this beforehand, otherwise I'll work further on this branch.

hadley · 2016-02-19T09:09:04Z

I don't have anything else major planned - I think most of the other issues need this PR completed first.

krlmlr · 2016-02-19T09:25:33Z

Extracted expectation_ok(), got rid of success_msg in my code, added two TODO markers.

I think test_code() can be simplified considerably, so that there's basically just one handler function taking care of everything. The reporter should be able to decide if tryCatch() catches the errors or not (#356) :

tryCatch(stop(), error = NULL)
tryCatch(stop(), error = function(e) NULL)

I'll merge the other PR before adding NEWS to this PR.

hadley · 2016-02-19T09:32:43Z

R/expectation.r

+  update_expectation(exp, srcref)
+}
+
+update_expectation <- function(exp, srcref, info = NULL, label = NULL) {


It feels like we should be able to eventually eliminate this function, wrapping it into the constructor

Difficult without finally deprecating info and label arguments (#218).

Yeah, lets look again once we've eliminated those args and refactored the expectations

krlmlr · 2016-02-19T09:45:40Z

Done. One more thing: expectation() is exported, you have changed its interface. Perhaps it's safer to keep success_msg (with a warning), or introduce a ... .

hadley · 2016-02-19T10:09:16Z

Yeah, I'm not too worried about that. I think the next release will be testthat 1.0.0 so it's ok to make a few breaking changes (and I doubt this one will affect many people)

hadley · 2016-02-19T10:09:43Z

And looks good to merge - feel free whenever you think it's ready

@krlmlr

* Refactored internal testing logic. Now all expectations are also conditions, and R's condition system is used to signal failures and successes (#360, @krlmlr). * `test_that()` returns a `logical` that indicates if all tests were successful (#360, @krlmlr).

Kirill Müller added 7 commits February 5, 2016 14:55

test_that() return ok value from test_code()

7339f0d

update ok in enclosing environment

5448986

add test (currently failing for case of test failure)

92aeb39

results are handled using conditions and restarts

6953645

use report_results() everywhere

68249e8

tests for special and new-style expectations

5e105d6

use signalCondition if tests passed

3d6dd33

for bare expectations

krlmlr changed the title ~~test_that() returns if tests passed~~ test_that() returns TRUE if successful (no failed expectations, errors, or skips) Feb 5, 2016

Kirill Müller added 2 commits February 5, 2016 16:30

typo

f1018a8

add test for error behavior with bare expectations

46795a6

krlmlr reviewed Feb 5, 2016
View reviewed changes

Kirill Müller added 6 commits February 5, 2016 16:48

remove commented code

12e8d57

also test skip()

00d4d90

check return value in test_test_that()

8481fd1

document (unrelated)

0deb2e0

only stop if condition not handled and test didn't pass

77d51dc

simplify bare tests

e450fbc

krlmlr force-pushed the feature/return-value branch from 73cb031 to e450fbc Compare February 5, 2016 16:26

move report_results

58ab45c

krlmlr pushed a commit to r-dbi/DBItest that referenced this pull request Feb 5, 2016

Merge branch 'feature/return-test-results'

28e1b23

- Return test results as named array of logical. Requires r-lib/testthat#360.

krlmlr pushed a commit to r-dbi/DBItest that referenced this pull request Feb 10, 2016

add workaround for r-lib/testthat#360.

4bfad96

krlmlr reviewed Feb 10, 2016
View reviewed changes

R/test-that.r

withRestarts(

{

signalCondition(cond)

Copy link

Member Author

krlmlr Feb 10, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Call stop() here.

krlmlr changed the title ~~test_that() returns TRUE if successful (no failed expectations, errors, or skips)~~ Use conditions for reporting results Feb 12, 2016

Kirill Müller added 2 commits February 12, 2016 09:56

use signalCondition() to signal success, and stop() to signal failure

ca3b8b8

as recommended in adv-r

update tests

8424b89

Now a failing expectation always raises an error, even if the StopReporter isn't active. The update of this test reflects this change in behavior.

krlmlr mentioned this pull request Feb 18, 2016

FR: DebugReporter #356

Closed

hadley reviewed Feb 18, 2016
View reviewed changes

Kirill Müller added 7 commits February 19, 2016 09:38

Merge commit 'origin/master~9' into feature/return-value

ed95622

Merge commit 'origin/master~5' into feature/return-value

29372d8

Merge commit 'origin/master~4' into feature/return-value

67c91c9

Merge commit 'origin/master~2' into feature/return-value

20ac9ea

Merge commit 'origin/master~1' into feature/return-value

72ba350

Merge remote-tracking branch 'origin/master' into feature/return-value

c3e6137

Merge remote-tracking branch 'origin/master' into feature/return-value

26b3c41

get rid of success_msg again

2b5c28c

extract expectation_ok()

5b45d70

Kirill Müller added 2 commits February 19, 2016 10:27

Merge remote-tracking branch 'origin/master' into feature/return-value

9d62f77

NEWS

e6b0cd6

hadley reviewed Feb 19, 2016
View reviewed changes

Kirill Müller added 3 commits February 19, 2016 10:40

move test

cfd0489

usage of expectation_ok()

116e60d

oops

d1fb341

krlmlr merged commit 0719724 into r-lib:master Feb 19, 2016

krlmlr deleted the feature/return-value branch February 19, 2016 10:39

krlmlr mentioned this pull request May 7, 2016

New DebugReporter #470

Merged

r2evans mentioned this pull request Aug 10, 2017

undoc'd string length limitation on NSE replacement for 'info' #614

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use conditions for reporting results #360

Use conditions for reporting results #360

krlmlr commented Feb 5, 2016

krlmlr Feb 5, 2016

krlmlr commented Feb 5, 2016

krlmlr Feb 10, 2016

hadley Feb 18, 2016

hadley Feb 18, 2016

krlmlr Feb 18, 2016

hadley Feb 18, 2016

krlmlr Feb 18, 2016

krlmlr Feb 18, 2016

krlmlr commented Feb 19, 2016

hadley commented Feb 19, 2016

krlmlr commented Feb 19, 2016

hadley Feb 19, 2016

krlmlr Feb 19, 2016

hadley Feb 19, 2016

krlmlr commented Feb 19, 2016

hadley commented Feb 19, 2016

hadley commented Feb 19, 2016

Use conditions for reporting results #360

Use conditions for reporting results #360

Conversation

krlmlr commented Feb 5, 2016

Choose a reason for hiding this comment

krlmlr commented Feb 5, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

krlmlr commented Feb 19, 2016

hadley commented Feb 19, 2016

krlmlr commented Feb 19, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

krlmlr commented Feb 19, 2016

hadley commented Feb 19, 2016

hadley commented Feb 19, 2016