New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Automatic retry of failed test #372

Closed
lilidotshi opened this Issue Jan 5, 2016 · 19 comments

Comments

Projects
None yet
10 participants
@lilidotshi

lilidotshi commented Jan 5, 2016

It would be nice to have a retry option for failed tests. Sometimes tests fail because of random simulator/UITest issues (network, UITest framework bugs, etc) and not issues with the code itself. If it happens even 1 in 200 times (which is likely given simulator * languages), the HTML doesn't get generated. It would be nice to specify retry x number of times, as often I've experienced that re-running the test succeeds.

@dpettit

This comment has been minimized.

Show comment
Hide comment
@dpettit

dpettit Jan 5, 2016

I agree, I find that one of the tests fails about 10% of the time, but all work fine on a 2nd run. Even if the snapshot command itself returned a value (t=all tests ran successfully, f=at least one test failed) after running all tests, I could just re-run snapshot myself whenever any of the tests failed.

dpettit commented Jan 5, 2016

I agree, I find that one of the tests fails about 10% of the time, but all work fine on a 2nd run. Even if the snapshot command itself returned a value (t=all tests ran successfully, f=at least one test failed) after running all tests, I could just re-run snapshot myself whenever any of the tests failed.

@donaldpiret

This comment has been minimized.

Show comment
Hide comment
@donaldpiret

donaldpiret Jan 16, 2016

+1 on this as well. Having major trouble getting through the entire test suite (all devices + all locales) without a single error.

donaldpiret commented Jan 16, 2016

+1 on this as well. Having major trouble getting through the entire test suite (all devices + all locales) without a single error.

@pedropalmero

This comment has been minimized.

Show comment
Hide comment
@pedropalmero

pedropalmero commented Jan 16, 2016

+1

@muZZkat

This comment has been minimized.

Show comment
Hide comment
@muZZkat

muZZkat Jan 26, 2016

+10000, the pre-1.0 snapshot did this and it was very useful

muZZkat commented Jan 26, 2016

+10000, the pre-1.0 snapshot did this and it was very useful

@buscarini

This comment has been minimized.

Show comment
Hide comment
@buscarini

buscarini commented Feb 8, 2016

+1

@lyptt

This comment has been minimized.

Show comment
Hide comment
@lyptt

lyptt commented Feb 8, 2016

+1

@KrauseFx

This comment has been minimized.

Show comment
Hide comment
@KrauseFx

KrauseFx Feb 8, 2016

Collaborator

Just to clarify, snapshot should just retry a test run when it fails, assuming that it might succeed the second time? And it should be marked as failed if it fails twice?

Collaborator

KrauseFx commented Feb 8, 2016

Just to clarify, snapshot should just retry a test run when it fails, assuming that it might succeed the second time? And it should be marked as failed if it fails twice?

@lilidotshi

This comment has been minimized.

Show comment
Hide comment
@lilidotshi

lilidotshi Feb 9, 2016

Can it be configurable?

On Mon, Feb 8, 2016 at 5:35 PM Felix Krause notifications@github.com
wrote:

Just to clarify, snapshot should just retry a test run when it fails,
assuming that it might succeed the second time? And it should be marked as
failed if it fails twice?


Reply to this email directly or view it on GitHub
#372 (comment).

lilidotshi commented Feb 9, 2016

Can it be configurable?

On Mon, Feb 8, 2016 at 5:35 PM Felix Krause notifications@github.com
wrote:

Just to clarify, snapshot should just retry a test run when it fails,
assuming that it might succeed the second time? And it should be marked as
failed if it fails twice?


Reply to this email directly or view it on GitHub
#372 (comment).

@dpettit

This comment has been minimized.

Show comment
Hide comment
@dpettit

dpettit Feb 9, 2016

I really like the idea of retrying a failed test, because they are often
successful the 2nd time through. Configurable would be even better (could
we just set our own max number of retries before marking it as failed).
Returning a programmatically accessible value would provide us with the
option of trying again at a different time, but that might not be such a
big deal if we could configure it how we wanted it.

On Monday, February 8, 2016, lilidotshi notifications@github.com wrote:

Can it be configurable?

On Mon, Feb 8, 2016 at 5:35 PM Felix Krause <notifications@github.com
javascript:_e(%7B%7D,'cvml','notifications@github.com');>
wrote:

Just to clarify, snapshot should just retry a test run when it fails,
assuming that it might succeed the second time? And it should be marked
as
failed if it fails twice?


Reply to this email directly or view it on GitHub
<#372 (comment)
.


Reply to this email directly or view it on GitHub
#372 (comment).

dpettit commented Feb 9, 2016

I really like the idea of retrying a failed test, because they are often
successful the 2nd time through. Configurable would be even better (could
we just set our own max number of retries before marking it as failed).
Returning a programmatically accessible value would provide us with the
option of trying again at a different time, but that might not be such a
big deal if we could configure it how we wanted it.

On Monday, February 8, 2016, lilidotshi notifications@github.com wrote:

Can it be configurable?

On Mon, Feb 8, 2016 at 5:35 PM Felix Krause <notifications@github.com
javascript:_e(%7B%7D,'cvml','notifications@github.com');>
wrote:

Just to clarify, snapshot should just retry a test run when it fails,
assuming that it might succeed the second time? And it should be marked
as
failed if it fails twice?


Reply to this email directly or view it on GitHub
<#372 (comment)
.


Reply to this email directly or view it on GitHub
#372 (comment).

@dpettit

This comment has been minimized.

Show comment
Hide comment
@dpettit

dpettit Feb 9, 2016

I really like the idea of retrying a failed test, because they are often
successful the 2nd time through. Configurable would be even better (could
we just set our own max number of retries before marking it as failed).
Returning a programmatically accessible value would provide us with the
option of trying again at a different time, but that might not be such a
big deal if we could configure it how we wanted it.

On Monday, February 8, 2016, Felix Krause notifications@github.com wrote:

Just to clarify, snapshot should just retry a test run when it fails,
assuming that it might succeed the second time? And it should be marked as
failed if it fails twice?


Reply to this email directly or view it on GitHub
#372 (comment).

dpettit commented Feb 9, 2016

I really like the idea of retrying a failed test, because they are often
successful the 2nd time through. Configurable would be even better (could
we just set our own max number of retries before marking it as failed).
Returning a programmatically accessible value would provide us with the
option of trying again at a different time, but that might not be such a
big deal if we could configure it how we wanted it.

On Monday, February 8, 2016, Felix Krause notifications@github.com wrote:

Just to clarify, snapshot should just retry a test run when it fails,
assuming that it might succeed the second time? And it should be marked as
failed if it fails twice?


Reply to this email directly or view it on GitHub
#372 (comment).

@KrauseFx

This comment has been minimized.

Show comment
Hide comment
@KrauseFx

KrauseFx Feb 9, 2016

Collaborator

I just pushed #431, and yes, you can configure how many times snapshot should retry running the tests. This will be available in the next release 👍

Collaborator

KrauseFx commented Feb 9, 2016

I just pushed #431, and yes, you can configure how many times snapshot should retry running the tests. This will be available in the next release 👍

@KrauseFx KrauseFx added the feature label Feb 9, 2016

@dpettit

This comment has been minimized.

Show comment
Hide comment
@dpettit

dpettit Feb 9, 2016

👍👍👍

On Tuesday, February 9, 2016, Felix Krause notifications@github.com wrote:

I just pushed #431 #431, and
yes, you can configure how many times snapshot should retry running the
tests. This will be available in the next release [image: 👍]


Reply to this email directly or view it on GitHub
#372 (comment).

dpettit commented Feb 9, 2016

👍👍👍

On Tuesday, February 9, 2016, Felix Krause notifications@github.com wrote:

I just pushed #431 #431, and
yes, you can configure how many times snapshot should retry running the
tests. This will be available in the next release [image: 👍]


Reply to this email directly or view it on GitHub
#372 (comment).

@KrauseFx KrauseFx closed this in #431 Feb 9, 2016

@KrauseFx

This comment has been minimized.

Show comment
Hide comment
@KrauseFx

KrauseFx Feb 9, 2016

Collaborator

Alright, new version is live: https://github.com/fastlane/snapshot/releases/tag/1.7.0 Please update and let me know how you like it 👍

Collaborator

KrauseFx commented Feb 9, 2016

Alright, new version is live: https://github.com/fastlane/snapshot/releases/tag/1.7.0 Please update and let me know how you like it 👍

@buscarini

This comment has been minimized.

Show comment
Hide comment
@buscarini

buscarini commented Feb 11, 2016

awesome!!

@vCrespoP

This comment has been minimized.

Show comment
Hide comment
@vCrespoP

vCrespoP Feb 11, 2016

Thank you mister ;)

vCrespoP commented Feb 11, 2016

Thank you mister ;)

@dpettit

This comment has been minimized.

Show comment
Hide comment
@dpettit

dpettit Feb 11, 2016

Soooooooo nice! Thank you!

On Thursday, February 11, 2016, vCrespoP notifications@github.com wrote:

Thank you mister ;)


Reply to this email directly or view it on GitHub
#372 (comment).

dpettit commented Feb 11, 2016

Soooooooo nice! Thank you!

On Thursday, February 11, 2016, vCrespoP notifications@github.com wrote:

Thank you mister ;)


Reply to this email directly or view it on GitHub
#372 (comment).

@lilidotshi

This comment has been minimized.

Show comment
Hide comment
@lilidotshi

lilidotshi Feb 11, 2016

100% success rate now with three retries.
On Thu, Feb 11, 2016 at 8:37 AM dpettit notifications@github.com wrote:

Soooooooo nice! Thank you!

On Thursday, February 11, 2016, vCrespoP notifications@github.com wrote:

Thank you mister ;)


Reply to this email directly or view it on GitHub
<#372 (comment)
.


Reply to this email directly or view it on GitHub
#372 (comment).

lilidotshi commented Feb 11, 2016

100% success rate now with three retries.
On Thu, Feb 11, 2016 at 8:37 AM dpettit notifications@github.com wrote:

Soooooooo nice! Thank you!

On Thursday, February 11, 2016, vCrespoP notifications@github.com wrote:

Thank you mister ;)


Reply to this email directly or view it on GitHub
<#372 (comment)
.


Reply to this email directly or view it on GitHub
#372 (comment).

@KrauseFx

This comment has been minimized.

Show comment
Hide comment
@KrauseFx

KrauseFx Feb 12, 2016

Collaborator

Thanks everyone for confirming, I'm so happy this solves the issue 😃

Collaborator

KrauseFx commented Feb 12, 2016

Thanks everyone for confirming, I'm so happy this solves the issue 😃

@fastlanebot

This comment has been minimized.

Show comment
Hide comment
@fastlanebot

fastlanebot Mar 7, 2016

This issue was migrated to fastlane/fastlane#2517. Please post all further comments there.

fastlane is now a mono repo, you can read more about the change in our blog post. All tools are now available in the fastlane main repo 🚀

fastlanebot commented Mar 7, 2016

This issue was migrated to fastlane/fastlane#2517. Please post all further comments there.

fastlane is now a mono repo, you can read more about the change in our blog post. All tools are now available in the fastlane main repo 🚀

@fastlane-old fastlane-old locked and limited conversation to collaborators Mar 11, 2016

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.