Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Automatic retry of failed test #2517

Closed
fastlanebot opened this issue Jan 5, 2016 · 18 comments

Comments

Projects
None yet
1 participant
@fastlanebot
Copy link

commented Jan 5, 2016

Original issue by @lilidotshi - Imported from fastlane/snapshot#372

It would be nice to have a retry option for failed tests. Sometimes tests fail because of random simulator/UITest issues (network, UITest framework bugs, etc) and not issues with the code itself. If it happens even 1 in 200 times (which is likely given simulator * languages), the HTML doesn't get generated. It would be nice to specify retry x number of times, as often I've experienced that re-running the test succeeds.

@fastlanebot

This comment has been minimized.

Copy link
Author

commented Jan 5, 2016

@dpettit commented

I agree, I find that one of the tests fails about 10% of the time, but all work fine on a 2nd run. Even if the snapshot command itself returned a value (t=all tests ran successfully, f=at least one test failed) after running all tests, I could just re-run snapshot myself whenever any of the tests failed.

@fastlanebot

This comment has been minimized.

Copy link
Author

commented Jan 16, 2016

@donaldpiret commented

+1 on this as well. Having major trouble getting through the entire test suite (all devices + all locales) without a single error.

@fastlanebot

This comment has been minimized.

Copy link
Author

commented Jan 16, 2016

@barbelith commented

+1

@fastlanebot

This comment has been minimized.

Copy link
Author

commented Jan 26, 2016

@muZZkat commented

+10000, the pre-1.0 snapshot did this and it was very useful

@fastlanebot

This comment has been minimized.

Copy link
Author

commented Feb 8, 2016

@buscarini commented

+1

@fastlanebot

This comment has been minimized.

Copy link
Author

commented Feb 8, 2016

@lyptt commented

+1

@fastlanebot

This comment has been minimized.

Copy link
Author

commented Feb 8, 2016

@KrauseFx commented

Just to clarify, snapshot should just retry a test run when it fails, assuming that it might succeed the second time? And it should be marked as failed if it fails twice?

@fastlanebot

This comment has been minimized.

Copy link
Author

commented Feb 9, 2016

@lilidotshi commented

Can it be configurable?

On Mon, Feb 8, 2016 at 5:35 PM Felix Krause notifications@github.com
wrote:

Just to clarify, snapshot should just retry a test run when it fails,
assuming that it might succeed the second time? And it should be marked as
failed if it fails twice?


Reply to this email directly or view it on GitHub
fastlane-old/snapshot#372 (comment).

@fastlanebot

This comment has been minimized.

Copy link
Author

commented Feb 9, 2016

@dpettit commented

I really like the idea of retrying a failed test, because they are often
successful the 2nd time through. Configurable would be even better (could
we just set our own max number of retries before marking it as failed).
Returning a programmatically accessible value would provide us with the
option of trying again at a different time, but that might not be such a
big deal if we could configure it how we wanted it.

On Monday, February 8, 2016, lilidotshi notifications@github.com wrote:

Can it be configurable?

On Mon, Feb 8, 2016 at 5:35 PM Felix Krause <notifications@github.com
javascript:_e(%7B%7D,'cvml','notifications@github.com');>
wrote:

Just to clarify, snapshot should just retry a test run when it fails,
assuming that it might succeed the second time? And it should be marked
as
failed if it fails twice?


Reply to this email directly or view it on GitHub
<fastlane-old/snapshot#372 (comment)
.


Reply to this email directly or view it on GitHub
fastlane-old/snapshot#372 (comment).

@fastlanebot

This comment has been minimized.

Copy link
Author

commented Feb 9, 2016

@dpettit commented

I really like the idea of retrying a failed test, because they are often
successful the 2nd time through. Configurable would be even better (could
we just set our own max number of retries before marking it as failed).
Returning a programmatically accessible value would provide us with the
option of trying again at a different time, but that might not be such a
big deal if we could configure it how we wanted it.

On Monday, February 8, 2016, Felix Krause notifications@github.com wrote:

Just to clarify, snapshot should just retry a test run when it fails,
assuming that it might succeed the second time? And it should be marked as
failed if it fails twice?


Reply to this email directly or view it on GitHub
fastlane-old/snapshot#372 (comment).

@fastlanebot

This comment has been minimized.

Copy link
Author

commented Feb 9, 2016

@KrauseFx commented

I just pushed fastlane-old/snapshot#431, and yes, you can configure how many times snapshot should retry running the tests. This will be available in the next release 👍

@fastlanebot

This comment has been minimized.

Copy link
Author

commented Feb 9, 2016

@dpettit commented

👍👍👍

On Tuesday, February 9, 2016, Felix Krause notifications@github.com wrote:

I just pushed #431 fastlane-old/snapshot#431, and
yes, you can configure how many times snapshot should retry running the
tests. This will be available in the next release [image: 👍]


Reply to this email directly or view it on GitHub
fastlane-old/snapshot#372 (comment).

@fastlanebot fastlanebot closed this Feb 9, 2016

@fastlanebot

This comment has been minimized.

Copy link
Author

commented Feb 9, 2016

@KrauseFx commented

Alright, new version is live: https://github.com/fastlane/snapshot/releases/tag/1.7.0 Please update and let me know how you like it 👍

@fastlanebot

This comment has been minimized.

Copy link
Author

commented Feb 11, 2016

@buscarini commented

awesome!!

@fastlanebot

This comment has been minimized.

Copy link
Author

commented Feb 11, 2016

@vCrespoP commented

Thank you mister ;)

@fastlanebot

This comment has been minimized.

Copy link
Author

commented Feb 11, 2016

@dpettit commented

Soooooooo nice! Thank you!

On Thursday, February 11, 2016, vCrespoP notifications@github.com wrote:

Thank you mister ;)


Reply to this email directly or view it on GitHub
fastlane-old/snapshot#372 (comment).

@fastlanebot

This comment has been minimized.

Copy link
Author

commented Feb 11, 2016

@lilidotshi commented

100% success rate now with three retries.
On Thu, Feb 11, 2016 at 8:37 AM dpettit notifications@github.com wrote:

Soooooooo nice! Thank you!

On Thursday, February 11, 2016, vCrespoP notifications@github.com wrote:

Thank you mister ;)


Reply to this email directly or view it on GitHub
<fastlane-old/snapshot#372 (comment)
.


Reply to this email directly or view it on GitHub
fastlane-old/snapshot#372 (comment).

@fastlanebot

This comment has been minimized.

Copy link
Author

commented Feb 12, 2016

@KrauseFx commented

Thanks everyone for confirming, I'm so happy this solves the issue 😃

@fastlane fastlane locked and limited conversation to collaborators Sep 26, 2016

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.