Skip to content

Commit

Permalink
test: update test-run
Browse files Browse the repository at this point in the history
Retry a failed test when it is marked as fragile (and several other
conditions are met, see below).

The test-run already allows to set a list of fragile tests. They are run
one-by-one after all parallel ones in order to eliminate possible
resource starvation and fit timings to ones when the tests pass. See
[1].

In practice this approach does not help much against our problem with
flaky tests. We decided to retry failed tests, when they are known as
flagile. See [2].

The core idea is to split responsibility: known flaky fails will not
deflect attention of a developer, but each fragile test will be marked
explicitly, trackerized and will be analyzed by the quality assurance
team.

The default behaviour is not changed: each test from the fragile list
will be run once after all parallel ones. But now it is possible to set
retries amount.

Beware: the implementation does not allow to just set retries count, it
also requires to provide an md5sum of a failed test output (so called
reject file). The idea here is to ensure that we retry the test only in
case of a known fail: not some other fail within the test.

This approach has the limitation: in case of fail a test may output an
information that varies from run to run or depend of a base directory.
We should always verify the output before put its checksum into the
configuration file.

Despite doubts regarding this approach, it looks simple and we decided
to try and revisit it if there will be a need.

See configuration example in [3].

[1]: tarantool/test-run#187
[2]: tarantool/test-run#189
[3]: tarantool/test-run#217

Part of #5050

(cherry picked from commit 43482ee)
  • Loading branch information
Totktonada committed Sep 24, 2020
1 parent 3a837eb commit ef330c3
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion test-run

0 comments on commit ef330c3

Please sign in to comment.