Add build support for FlakyAttribute

[analogrelay]

I'm filing this to track adding support for FlakyAttribute to the aspnet/EntityFramework build scripts.

The attribute allows you to sequester a test by marking it as "Flaky". That test will no longer be run during the normal run but will be run during a separate second test pass. The results of the second test pass will not affect the build outcome, but the test results will be published and recorded.

What's involved (see dotnet/extensions#1239)

• Add a script to run flaky tests eng/scripts/ci-flaky-tests.cmd/.sh
• Add some MSBuild goop to skip flaky tests on normal runs but run it when ci-flaky-tests is run
• Add logic to run ci-flaky-tests after the main build in Azure Pipelines (AzP)
• Add logic to publish test results from flaky tests to AzP so they still show up in the Tests pane in the AzP build summary.

Happy to do the work, but wanted an issue tracking it here and wanted to bring it to the attention of the team to confirm you want it :).

[smitpatel]

Do we have flaky tests?

[analogrelay]

That's part of the reason I'm not just going ahead and adding the support :). Having said that, if I add the support and you never use the attribute, the impact is minimal but you can apply the attribute as soon as you find a flaky test you want to sequester in this way. If we don't do the work now, that's fine, but we have to do that work whenever you do decide you want to separate out a flaky test.

cc @ajcvickers

[ajcvickers]

@smitpatel We do have a lot of test failures, but of the ones that aren't infra related, the rest seem to be flakey SQL Server. Current best plan to help with that is to run SQL Server on a different box. Other types of bad tests we're petty good at fixing straight away.

Let's discuss in triage.

[analogrelay]

Just give me a ping if you want the Flaky test infrastructure in your repo :).