Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Flaky integration tests #1600

Closed
atbore-phx opened this issue May 14, 2017 · 5 comments
Closed

Flaky integration tests #1600

atbore-phx opened this issue May 14, 2017 · 5 comments
Labels
Milestone

Comments

@atbore-phx
Copy link

atbore-phx commented May 14, 2017

We have switched our tests to Semaphore CI (#1266) and we got a execution time boost (~2x).
We also restored the ci_retry function (#1573) because the integration tests are still flaking.

I'm starting to deploy a simple scenario based on tests series to obtain a closer perimeter and to understand which tests have the higher "flaky probability".

Reiterating the same test on semaphore 100 times (deactivating the ci_retry function),
i get those results: link

@atbore-phx
Copy link
Author

Series : S1
KO: 14%
higher errors:
acme_test 5
eureka_test 4

@atbore-phx
Copy link
Author

Series: S2 (acme and eureka tests are executed first):
KO: 15%
higher errors:
acme_test 7
eureka_test 4

Acme and eureka tests are still the tests with the higher flaky probability.
The test's order doesn't affect the final results so we "could" exclude an hw resource exhaustion issue.

@atbore-phx
Copy link
Author

Series: S3 (increasing acme and eureka timeouts).
KO: 8%
higher errors:
acme_test 2
eureka_test 3

An higher timeout on the most flaky tests seems reduce the error probability (especially for the acme test )

@atbore-phx
Copy link
Author

This PR #1599 has the main goal to deflake the integration tests.

@ldez ldez added the kind/enhancement a new or improved feature. label May 14, 2017
@ldez
Copy link
Member

ldez commented May 19, 2017

Closed by #1599

@ldez ldez closed this as completed May 19, 2017
@ldez ldez added this to the 1.4 milestone May 19, 2017
@traefik traefik locked and limited conversation to collaborators Sep 1, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Projects
None yet
Development

No branches or pull requests

3 participants