Gardening: mark flaky tests flaky.

[gspencergoog]

This marks the following tests as flaky, per the gardener rotation:

• linux_complex_layout_scroll_perf__devtools_memory: complex_layout_scroll_perf__devtools_memory is 4.65% flaky #82741
• linux_large_image_changer_perf_android: large_image_changer_perf_android is 2.44% flaky #82747
• linux_new_gallery__crane_perf: new_gallery__crane_perf is 2.56% flaky #82745
• linux_platform_channels_benchmarks: platform_channels_benchmarks is 3.33% flaky #82743

[zanderso]

Please cross-reference the individual issues for the flaky tests in the PR description so that github will add links, and link to the individual issues for the flaky tests in prod_builders.json.

[keyonghan]

For the tests exceeding 2%, the proposal is to remove them out of the dashboard. They need to go through the staging env. for validation before enabled back: https://github.com/flutter/flutter/wiki/Reducing-Test-Flakiness#fixing-flaky-tests

[gaaclarke]

Can I get a link to a failure for linux_platform_channels_benchmarks? I want to verify that the issue was intrinsic to the actual test. FWIW that test has been running for a week so you might not have an accurate flake percentage for it. A single failure from before the test was marked not flakey might be giving you a reading with that threshold.

[zanderso]

@keyonghan If "remove them out of the dashboard" means that they won't run at all, then I don't think that's the right thing to do. Especially since these are benchmarks, removing them entirely will mean that we will miss regressions.

[keyonghan]

@keyonghan If "remove them out of the dashboard" means that they won't run at all, then I don't think that's the right thing to do. Especially since these are benchmarks, removing them entirely will mean that we will miss regressions.

"remove them out of the dashboard" means they will not show up in the flutter build dashboard, but they will still be triggered and run. Milo dashboard will still show the build info.

[gspencergoog]

I've removed the entries (and so didn't add the bug cross-references, since there's nothing left to connect them to), but I'll wait to commit this until you LGTM this updated PR again, @zanderso.

[zanderso]

I think we need to leave these on the dashboard so that we have a visual reminder of what is marked flaky.

[keyonghan]

I think we need to leave these on the dashboard so that we have a visual reminder of what is marked flaky.

I guess our intention is to unblock the tree and make sure the test passes 50 successful runs in staging pool before enabled back.
I guess your suggestion here is okay just by enforcing the validation in prod pool instead of staging pool.

We can start this way, and adjust the process if needed. I will update the doc accordingly.

[gspencergoog]

OK, done. Since this is pure jSON (no comments), I added an extra field called "issue_url" to the flaky tests, that will hopefully be ignored. If not, well, I'll revert it.

[CaseyHillers]

I haven't announced it yet, but I'm aiming to deprecate this JSON config Wednesday night in favor of the .ci.yaml. Just commenting here as I'll need to move these to comments in the migration PR (@CaseyHillers @christopherfujino )

[gspencergoog]

I think you just announced it. :-)

[keyonghan]

I suppose new properties could be naturally supported in .ci.yaml?

[CaseyHillers]

Yes, PRs welcome if you want to add automation around this

[christopherfujino]

I suppose new properties could be naturally supported in .ci.yaml?

You would have to update the protobuf in flutter/cocoon first. I actually see this as a feature, as if you add a new target and mispell a field, CI will fail on an unknown field, rather than silently ignoring it.

[gspencergoog]

Sorry, @zanderso, I can't add extra fields or comments to the JSON, so I had to remove the cross-linking for the issues.

@CaseyHillers might be able to add a field for that into .ci.yaml for this (seems useful in any case).

[gspencergoog]

When re-enabling these tests, they should still probably go through the "ran 50 times and didn't flake" soak time, right?

[keyonghan]

When re-enabling these tests, they should still probably go through the "ran 50 times and didn't flake" soak time, right?

Yes (https://github.com/flutter/flutter/wiki/Reducing-Test-Flakiness#fixing-flaky-tests)
Whenever a test is marked as flaky, it has to be validated before enabled back.

[gaaclarke]

Will it show up in go/flutter_15day_flakiness_dashboard to verify that it isn't flakey if it has been marked flakey though? We might want to add instructions on how to validate 50 successful runs.

[keyonghan]

Will it show up in go/flutter_15day_flakiness_dashboard to verify that it isn't flakey if it has been marked flakey though? We might want to add instructions on how to validate 50 successful runs.

It will still show up in the 15 day dashboard. Created #82767 to enable an easy validation. Before that, the flutter build dashboard will be the source of truth (no task box with exclamation point for top 50 consecutive commits).