[ci] Test retries don't clean between runs #8687
Assignees
Labels
Milestone
Comments
microsoft-github-policy-service
bot
added
the
needs-triage
jpobst
changed the title
jpobst
added a commit
that referenced
this issue
Mar 14, 2024
Context: #8687 In #7963, we added an automatic retry mechanism to the APK test suites. However this mechanism does not give us a chance to run any tasks between the retry. This means that when the tests are run a second time the output directories have not been cleaned and thus we are running incremental builds instead of full builds. We could unknowingly have tests that always fail on a clean build and succeed on an incremental build. Thus we are going to remove the automatic retries. The good news is these test suites seem to be much more stable now than when we implemented the retries, so hopefully this will not result in too many flaky CI builds.
|
This was fixed for the APK test suites in #8811. Still need to verify the |
jonathanpeppers
pushed a commit
that referenced
this issue
May 8, 2024
Context: #8687 In #7963, we added an automatic retry mechanism to the APK test suites. However this mechanism does not give us a chance to run any tasks between the retry. This means that when the tests are run a second time the output directories have not been cleaned and thus we are running incremental builds instead of full builds. We could unknowingly have tests that always fail on a clean build and succeed on an incremental build. Thus we are going to remove the automatic retries. The good news is these test suites seem to be much more stable now than when we implemented the retries, so hopefully this will not result in too many flaky CI builds.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
For the tests that use Azure Pipelines' built-in
retryCountOnTaskFailure(likeMono.Android.NET-Tests), the test suite is re-run immediately after the failure. We cannot inject any extra steps before the retry. This means that the second run is an incremental build because theobj/bindirectories are not cleaned. Additionally, the logs from the first run are not captured.This could lead to passing of tests that fail on initial builds and only succeed on incremental builds.
Given the inflexibility of
retryCountOnTaskFailure, we likely need to turn it off completely.We should also examine our usage of
dotnet-test-slicerto see if it suffers from the same issue. At least withdotnet-test-slicerwe can run steps between the initial run and the retry, so we can add manual cleans if needed.The text was updated successfully, but these errors were encountered: