-
Notifications
You must be signed in to change notification settings - Fork 14
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Retry tests on failure #328
Comments
ylobankov
added a commit
that referenced
this issue
Mar 15, 2022
test-run supports functionality to rerun failed tests in place, but these tests have to be on so called fragile list. To add a test to the fragile list we need to add a special configuration to the suite.ini file of a test suite. Configuration example: fragile = { "retries": 5, "tests": { "tarantoolctl.test.lua": { "issues": [ "gh-5059", "gh-5346" ] }, "debug.test.lua": { "issues": [ "gh-5346" ] }, ... } } Rerunning failed tests in place is quite convenient because it allows us to avoid rerunning all tests again and thus save time. But to make it work as expected we should keep the list of fragile tests always up-to-date. Flaky tests may be introduced every day and keeping the list of fragile tests always up-to-date becomes extremely difficult to do. So our solusion is quite simple: just rerun all failed tests. By default, the number of retries for regular and fragile tests is 3. But for fragile tests this number can be overriden in the suite.ini file. Closes #328
ylobankov
added a commit
that referenced
this issue
Mar 16, 2022
test-run supports functionality to rerun failed tests in place, but these tests have to be on so called fragile list. To add a test to the fragile list we need to add a special configuration to the suite.ini file of a test suite. Configuration example: fragile = { "retries": 5, "tests": { "tarantoolctl.test.lua": { "issues": [ "gh-5059", "gh-5346" ] }, "debug.test.lua": { "issues": [ "gh-5346" ] }, ... } } Rerunning failed tests in place is quite convenient because it allows us to avoid rerunning all tests again and thus save time. But to make it work as expected we should keep the list of fragile tests always up-to-date. Flaky tests may be introduced every day and keeping the list of fragile tests always up-to-date becomes extremely difficult to do. So our solusion is quite simple: just rerun all failed tests. By default, the number of retries for regular and fragile tests is 3. But for fragile tests this number can be overriden in the suite.ini file. Closes #328
kyukhin
pushed a commit
that referenced
this issue
Mar 17, 2022
test-run supports functionality to rerun failed tests in place, but these tests have to be on so called fragile list. To add a test to the fragile list we need to add a special configuration to the suite.ini file of a test suite. Configuration example: fragile = { "retries": 5, "tests": { "tarantoolctl.test.lua": { "issues": [ "gh-5059", "gh-5346" ] }, "debug.test.lua": { "issues": [ "gh-5346" ] }, ... } } Rerunning failed tests in place is quite convenient because it allows us to avoid rerunning all tests again and thus save time. But to make it work as expected we should keep the list of fragile tests always up-to-date. Flaky tests may be introduced every day and keeping the list of fragile tests always up-to-date becomes extremely difficult to do. So our solusion is quite simple: just rerun all failed tests. By default, the number of retries for regular and fragile tests is 3. But for fragile tests this number can be overriden in the suite.ini file. Closes #328
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
When a test fails, retry it up to N times, expecting a successful run. Effectively, this means treating all tests as flaky (fragile).
Report a retried test in the logs. Use different log messages for tests in the "fragile" list and not in the list.
The text was updated successfully, but these errors were encountered: