-
Notifications
You must be signed in to change notification settings - Fork 30
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
retest-until-fail vs repeat-until-fail #19
Comments
The option you use locally only applies to CMake packages. It does not cover non-ctest tests like The nightly job was designed to catch if there are any flaky tests and "fail fast". Since the goal is to not have any flaky tests there was no focus on enumerating all of them. As you have noticed doing so requires a lot of time. |
Good point for the non Cmake packages. Thanks for the explanation I was not sure if it was intentional behavior or an overlooked side effect. |
@mikaelarguedas what is the status of this ticket? Should it be closed? |
If there is no solution at all it can be closed or marked as untargeted. |
@mikaelarguedas Can you update this issue into a todo (rather than a question) describing what should be done? |
TODO: Change the behaviour of repeat-until-fail to:
I am not sure how doable it is and how much work this will require but that will be a more accurate way to identify all flaky tests |
I believe this can be closed as it has been addressed by the switch to colcon. From my understanding, colcon runs and repeat tests one by one in a package (compared to ament tools that was running then entire test suite of a package N times). So tests in a package are not impacted by the result of other tests from the same package anymore. @dirk-thomas Feel free to comment or re-open if I missed something here. |
By investigating flaky tests I noticed that the repeated nightlies are using the
--retest-until-fail <N>
option, whereas when testing locally I run ament test with the option--ctest-args --repeat-until-fail <N> --
I'm wondering if
--retest-until-fail <N>
has the desired behavior for our repeated jobs. Below explained the difference of behavior between the two optionsctest --repeat-until-fail <N>
:Ensures that all tests are run until failure or N runs
ament test --retest-until-fail <N>
Stop testing a package as soon as a single test of this package fails
This could explain why the reported flaky tests are different every night. Especially on packages with a lot of tests like the
test_communication
Options we may want to consider:
ctest --repeat-until-fail <N>
argument to have an exhaustive list of the flaky testsTo have an idea of how long the nightlies would take with
ctest --repeat-until-fail 20
I launched http://ci.ros2.org/job/ci_windows/2030/, on windows it took 5h30min instead of 2hThe text was updated successfully, but these errors were encountered: