New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add an option to retry package testing #240

Closed
craigcitro opened this Issue Mar 27, 2015 · 7 comments

Comments

Projects
None yet
4 participants
@craigcitro
Contributor

craigcitro commented Mar 27, 2015

This would be especially useful in contexts like travis.

@jennybc

This comment has been minimized.

Member

jennybc commented Mar 28, 2015

While we are dreaming, it would also be nice if one could limit the retry to the specific tests or contexts that failed.

@gaborcsardi

This comment has been minimized.

Member

gaborcsardi commented Mar 28, 2015

@craigcitro Can't you just retry from the shell?

@craigcitro

This comment has been minimized.

Contributor

craigcitro commented Mar 28, 2015

@gaborcsardi sure -- but it'd be even better to ask testthat to do it, rather than have to teach every possible test runner, right?

@hadley

This comment has been minimized.

Member

hadley commented Sep 24, 2015

Could you give a bit more context?

@jennybc

This comment has been minimized.

Member

jennybc commented Sep 24, 2015

I know this is @craigcitro's issue ... but what I meant when I chimed in was this:

1 out of every x times, a test fails for googlesheets on travis, just because of some glitch w/ the Drive or Sheets API. My dream was to have something like retry(n, time), kind of like skip_on_CRAN(), so test_that() would automatically retry n times with interval time before declaring failure.

@hadley

This comment has been minimized.

Member

hadley commented Sep 24, 2015

Hmm I think @craigcitro probably wants something that retries all the tests, but an expectation level thing also sounds useful

@craigcitro

This comment has been minimized.

Contributor

craigcitro commented Sep 25, 2015

the original context was indeed "just retry everything" -- which is an ugly way to do things, but i thought might be the most likely to get implemented as v0. the goal was to use retries to paper over any flakiness due to external factors (other APIs, network woes, hardware wonkiness, etc).

that said, i think this would be far superior as something you could enable for a group of tests, in particular to avoid ignoring tests that are flaky but shouldn't be. (in my head, at least, "just rerun all the tests" is the same sort of band-aid solution as using sleep() when trying to work around race conditions.) then users could make their CI systems happy without creating new problems.

and, as mentioned above, i like the idea of this being part of testthat so that it's agnostic to CI system.

maybe those were all my issues? i have so many, it's hard to keep track. 😁

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment