Automatic retry of failed test

[lilidotshi]

It would be nice to have a retry option for failed tests. Sometimes tests fail because of random simulator/UITest issues (network, UITest framework bugs, etc) and not issues with the code itself. If it happens even 1 in 200 times (which is likely given simulator * languages), the HTML doesn't get generated. It would be nice to specify retry x number of times, as often I've experienced that re-running the test succeeds.

[dpettit]

I agree, I find that one of the tests fails about 10% of the time, but all work fine on a 2nd run. Even if the snapshot command itself returned a value (t=all tests ran successfully, f=at least one test failed) after running all tests, I could just re-run snapshot myself whenever any of the tests failed.

[donaldpiret]

+1 on this as well. Having major trouble getting through the entire test suite (all devices + all locales) without a single error.

[pedropalmero]

+1

[muZZkat]

+10000, the pre-1.0 snapshot did this and it was very useful

[buscarini]

+1

[ghost]

+1

[KrauseFx]

Just to clarify, snapshot should just retry a test run when it fails, assuming that it might succeed the second time? And it should be marked as failed if it fails twice?

[lilidotshi]

Can it be configurable?

On Mon, Feb 8, 2016 at 5:35 PM Felix Krause notifications@github.com
wrote:

Just to clarify, snapshot should just retry a test run when it fails,
assuming that it might succeed the second time? And it should be marked as
failed if it fails twice?

—
Reply to this email directly or view it on GitHub
#372 (comment).

[dpettit]

I really like the idea of retrying a failed test, because they are often
successful the 2nd time through. Configurable would be even better (could
we just set our own max number of retries before marking it as failed).
Returning a programmatically accessible value would provide us with the
option of trying again at a different time, but that might not be such a
big deal if we could configure it how we wanted it.

On Monday, February 8, 2016, lilidotshi notifications@github.com wrote:

Can it be configurable?

On Mon, Feb 8, 2016 at 5:35 PM Felix Krause <notifications@github.com
javascript:_e(%7B%7D,'cvml','notifications@github.com');>
wrote:

Just to clarify, snapshot should just retry a test run when it fails,
assuming that it might succeed the second time? And it should be marked
as
failed if it fails twice?

—
Reply to this email directly or view it on GitHub
<#372 (comment)
.

—
Reply to this email directly or view it on GitHub
#372 (comment).

[dpettit]

I really like the idea of retrying a failed test, because they are often
successful the 2nd time through. Configurable would be even better (could
we just set our own max number of retries before marking it as failed).
Returning a programmatically accessible value would provide us with the
option of trying again at a different time, but that might not be such a
big deal if we could configure it how we wanted it.

On Monday, February 8, 2016, Felix Krause notifications@github.com wrote:

Just to clarify, snapshot should just retry a test run when it fails,
assuming that it might succeed the second time? And it should be marked as
failed if it fails twice?

—
Reply to this email directly or view it on GitHub
#372 (comment).

[KrauseFx]

I just pushed #431, and yes, you can configure how many times snapshot should retry running the tests. This will be available in the next release 👍

[dpettit]

👍👍👍

On Tuesday, February 9, 2016, Felix Krause notifications@github.com wrote:

I just pushed #431 #431, and
yes, you can configure how many times snapshot should retry running the
tests. This will be available in the next release [image: 👍]

—
Reply to this email directly or view it on GitHub
#372 (comment).

[KrauseFx]

Alright, new version is live: https://github.com/fastlane/snapshot/releases/tag/1.7.0 Please update and let me know how you like it 👍

[buscarini]

awesome!!

[SentoCrespo]

Thank you mister ;)

[dpettit]

Soooooooo nice! Thank you!

On Thursday, February 11, 2016, vCrespoP notifications@github.com wrote:

Thank you mister ;)

—
Reply to this email directly or view it on GitHub
#372 (comment).

[lilidotshi]

100% success rate now with three retries.
On Thu, Feb 11, 2016 at 8:37 AM dpettit notifications@github.com wrote:

Soooooooo nice! Thank you!

On Thursday, February 11, 2016, vCrespoP notifications@github.com wrote:

Thank you mister ;)

—
Reply to this email directly or view it on GitHub
<#372 (comment)
.

—
Reply to this email directly or view it on GitHub
#372 (comment).

[KrauseFx]

Thanks everyone for confirming, I'm so happy this solves the issue 😃

[fastlanebot]

This issue was migrated to fastlane/fastlane#2517. Please post all further comments there.

fastlane is now a mono repo, you can read more about the change in our blog post. All tools are now available in the fastlane main repo 🚀