Retest only test that fail and not entire package

[ruffsl]

Correct me if I'm mistaken, but it seems that colcon test --retest-until-fail N colcon test --retest-until-pass will rerun all tests for a failing package, as opposed to only the specific tests in the package that failed. Could this be changed so the results from a previous job be used as a block list to scope which specific tests need to be repeated, not merely the package. This could help speed up CI pipelines with flaky tests.

@SteveMacenski, could you link to a previous job that exhibits the behavior you described to me offline.

[dirk-thomas]

... as opposed to only the specific tests in the package that failed

The option --retest-until-fail doesn't ever re-run a test which has failed. It only re-runs all tests when in the initial / previous run all tests have passed. If any test fails no more re-runs are attempted.

The use case of the option is to check for flaky tests. So passing --retest-until-fail N ensures that all tests passed N+1 times and never failed. It doesn't achieve finding as many flaky tests as possible by re-running the passed tests even when some tests have already failed.

[ruffsl]

Woops, I typo-ed, I meant to reference --retest-until-pass.

[dirk-thomas]

For CMake packages the option --retest-until-pass maps to the CTest option --rerun-failed which only re-runs the tests which have failed: https://github.com/colcon/colcon-cmake/blob/b1963768a8484aebdf827fdf10303ebe73003b45/colcon_cmake/task/cmake/test.py#L105-L107

For Python packages using pytest the option --retest-until-pass maps to the pytest-rerunfailures option --reruns which should do the same: see

colcon-core/colcon_core/task/python/test/pytest.py

Lines 126 to 137 in 2a12b22

	try:
	import pytest_rerunfailures # noqa: F401
	except ImportError:
	logger.warning(
	"Ignored '--retest-until-pass' for package "
	"'{context.pkg.name}' since pytest extension "
	"'rerunfailures' was not found".format_map(locals()))
	else:
	args += [
	'--reruns={context.args.retest_until_pass}'
	.format_map(locals()),
	]

[SteveMacenski]

@ruffsl https://app.circleci.com/pipelines/github/ros-planning/navigation2/3553/workflows/67da27fb-e87b-46d9-b33b-840882c9bb19/jobs/13927/steps this is one such job. You can see it fails the first time on nav2_system_tests after about 10 minutes (about as long as it takes to run all of the tests), then it re-runs and take another 10 minutes to fail. No single test takes 10 minutes so that was re-running the full suite.

Feel free to ignore the actual failure from this PR CI run. This is just the first demonstration I could find.

[dirk-thomas]

The log shows a completely separate second invocation of colcon test with the argument --packages-select-test-failures which will run all tests for the packages which had test failures in the first invocation of colcon test.

This kind of repetition has nothing to do with the re-running of tests based on the arguments --retest-until-fail / --retest-until-pass.

[ruffsl]

The log shows a completely separate second invocation of colcon test

This may have gotten lost in the line nose thanks to RCUTILS_CONSOLE_STDOUT_LINE_BUFFERED.
Although, from ros-navigation/navigation2#1271 (comment) , not sure why time ordering of console messages is lost without it.
Thanks for spotting this @dirk-thomas !

Looks this is a side effect from ros-navigation/navigation2#1284 . I'll close this and move discussion there.