Add an option to retry package testing

[craigcitro]

This would be especially useful in contexts like travis.

[jennybc]

While we are dreaming, it would also be nice if one could limit the retry to the specific tests or contexts that failed.

[gaborcsardi]

@craigcitro Can't you just retry from the shell?

[craigcitro]

@gaborcsardi sure -- but it'd be even better to ask testthat to do it, rather than have to teach every possible test runner, right?

[hadley]

Could you give a bit more context?

[jennybc]

I know this is @craigcitro's issue ... but what I meant when I chimed in was this:

1 out of every x times, a test fails for googlesheets on travis, just because of some glitch w/ the Drive or Sheets API. My dream was to have something like retry(n, time), kind of like skip_on_CRAN(), so test_that() would automatically retry n times with interval time before declaring failure.

[hadley]

Hmm I think @craigcitro probably wants something that retries all the tests, but an expectation level thing also sounds useful

[craigcitro]

the original context was indeed "just retry everything" -- which is an ugly way to do things, but i thought might be the most likely to get implemented as v0. the goal was to use retries to paper over any flakiness due to external factors (other APIs, network woes, hardware wonkiness, etc).

that said, i think this would be far superior as something you could enable for a group of tests, in particular to avoid ignoring tests that are flaky but shouldn't be. (in my head, at least, "just rerun all the tests" is the same sort of band-aid solution as using sleep() when trying to work around race conditions.) then users could make their CI systems happy without creating new problems.

and, as mentioned above, i like the idea of this being part of testthat so that it's agnostic to CI system.

maybe those were all my issues? i have so many, it's hard to keep track. 😁