Skip to content
This repository has been archived by the owner. It is now read-only.

Automatic retry of failed test #372

Closed
lilidotshi opened this issue Jan 5, 2016 · 19 comments
Closed

Automatic retry of failed test #372

lilidotshi opened this issue Jan 5, 2016 · 19 comments

Comments

@lilidotshi
Copy link

@lilidotshi lilidotshi commented Jan 5, 2016

It would be nice to have a retry option for failed tests. Sometimes tests fail because of random simulator/UITest issues (network, UITest framework bugs, etc) and not issues with the code itself. If it happens even 1 in 200 times (which is likely given simulator * languages), the HTML doesn't get generated. It would be nice to specify retry x number of times, as often I've experienced that re-running the test succeeds.

@dpettit
Copy link

@dpettit dpettit commented Jan 5, 2016

I agree, I find that one of the tests fails about 10% of the time, but all work fine on a 2nd run. Even if the snapshot command itself returned a value (t=all tests ran successfully, f=at least one test failed) after running all tests, I could just re-run snapshot myself whenever any of the tests failed.

@donaldpiret
Copy link

@donaldpiret donaldpiret commented Jan 16, 2016

+1 on this as well. Having major trouble getting through the entire test suite (all devices + all locales) without a single error.

@pedropalmero
Copy link

@pedropalmero pedropalmero commented Jan 16, 2016

+1

@muZZkat
Copy link

@muZZkat muZZkat commented Jan 26, 2016

+10000, the pre-1.0 snapshot did this and it was very useful

@buscarini
Copy link

@buscarini buscarini commented Feb 8, 2016

+1

1 similar comment
@lyptt
Copy link

@lyptt lyptt commented Feb 8, 2016

+1

@KrauseFx
Copy link
Collaborator

@KrauseFx KrauseFx commented Feb 8, 2016

Just to clarify, snapshot should just retry a test run when it fails, assuming that it might succeed the second time? And it should be marked as failed if it fails twice?

@lilidotshi
Copy link
Author

@lilidotshi lilidotshi commented Feb 9, 2016

Can it be configurable?

On Mon, Feb 8, 2016 at 5:35 PM Felix Krause notifications@github.com
wrote:

Just to clarify, snapshot should just retry a test run when it fails,
assuming that it might succeed the second time? And it should be marked as
failed if it fails twice?


Reply to this email directly or view it on GitHub
#372 (comment).

@dpettit
Copy link

@dpettit dpettit commented Feb 9, 2016

I really like the idea of retrying a failed test, because they are often
successful the 2nd time through. Configurable would be even better (could
we just set our own max number of retries before marking it as failed).
Returning a programmatically accessible value would provide us with the
option of trying again at a different time, but that might not be such a
big deal if we could configure it how we wanted it.

On Monday, February 8, 2016, lilidotshi notifications@github.com wrote:

Can it be configurable?

On Mon, Feb 8, 2016 at 5:35 PM Felix Krause <notifications@github.com
javascript:_e(%7B%7D,'cvml','notifications@github.com');>
wrote:

Just to clarify, snapshot should just retry a test run when it fails,
assuming that it might succeed the second time? And it should be marked
as
failed if it fails twice?


Reply to this email directly or view it on GitHub
<#372 (comment)
.


Reply to this email directly or view it on GitHub
#372 (comment).

@dpettit
Copy link

@dpettit dpettit commented Feb 9, 2016

I really like the idea of retrying a failed test, because they are often
successful the 2nd time through. Configurable would be even better (could
we just set our own max number of retries before marking it as failed).
Returning a programmatically accessible value would provide us with the
option of trying again at a different time, but that might not be such a
big deal if we could configure it how we wanted it.

On Monday, February 8, 2016, Felix Krause notifications@github.com wrote:

Just to clarify, snapshot should just retry a test run when it fails,
assuming that it might succeed the second time? And it should be marked as
failed if it fails twice?


Reply to this email directly or view it on GitHub
#372 (comment).

@KrauseFx
Copy link
Collaborator

@KrauseFx KrauseFx commented Feb 9, 2016

I just pushed #431, and yes, you can configure how many times snapshot should retry running the tests. This will be available in the next release 👍

@dpettit
Copy link

@dpettit dpettit commented Feb 9, 2016

👍👍👍

On Tuesday, February 9, 2016, Felix Krause notifications@github.com wrote:

I just pushed #431 #431, and
yes, you can configure how many times snapshot should retry running the
tests. This will be available in the next release [image: 👍]


Reply to this email directly or view it on GitHub
#372 (comment).

@KrauseFx KrauseFx closed this in #431 Feb 9, 2016
@KrauseFx
Copy link
Collaborator

@KrauseFx KrauseFx commented Feb 9, 2016

Alright, new version is live: https://github.com/fastlane/snapshot/releases/tag/1.7.0 Please update and let me know how you like it 👍

@buscarini
Copy link

@buscarini buscarini commented Feb 11, 2016

awesome!!

@vCrespoP
Copy link

@vCrespoP vCrespoP commented Feb 11, 2016

Thank you mister ;)

@dpettit
Copy link

@dpettit dpettit commented Feb 11, 2016

Soooooooo nice! Thank you!

On Thursday, February 11, 2016, vCrespoP notifications@github.com wrote:

Thank you mister ;)


Reply to this email directly or view it on GitHub
#372 (comment).

@lilidotshi
Copy link
Author

@lilidotshi lilidotshi commented Feb 11, 2016

100% success rate now with three retries.
On Thu, Feb 11, 2016 at 8:37 AM dpettit notifications@github.com wrote:

Soooooooo nice! Thank you!

On Thursday, February 11, 2016, vCrespoP notifications@github.com wrote:

Thank you mister ;)


Reply to this email directly or view it on GitHub
<#372 (comment)
.


Reply to this email directly or view it on GitHub
#372 (comment).

@KrauseFx
Copy link
Collaborator

@KrauseFx KrauseFx commented Feb 12, 2016

Thanks everyone for confirming, I'm so happy this solves the issue 😃

@fastlanebot
Copy link

@fastlanebot fastlanebot commented Mar 7, 2016

This issue was migrated to fastlane/fastlane#2517. Please post all further comments there.

fastlane is now a mono repo, you can read more about the change in our blog post. All tools are now available in the fastlane main repo 🚀

@fastlane-old fastlane-old locked and limited conversation to collaborators Mar 11, 2016
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
10 participants