Windows tests timing out

[justinmc]

If this doesn't merit its own issue, feel free to close.

Happened on Windows tool_integration_tests_1_6. The test ran fine but stopped partway through with the last message being Terminate batch job (Y/N)?.

Examples:
https://ci.chromium.org/ui/p/flutter/builders/prod/Windows%20tool_integration_tests_1_6/42/overview
https://ci.chromium.org/ui/p/flutter/builders/prod/Windows%20tool_integration_tests_3_6/43/overview

[zanderso]

These look like timeouts. @christopherfujino Do we already have an issue for this?

[jmagman]

This reminds me of #88060, but Windows. The tests are just running slower, compare the test timings to a successful run: https://logs.chromium.org/logs/flutter/buildbucket/cr-buildbucket/8824125284678159905/+/u/run_test.dart_for_tool_integration_tests_shard_and_subshard_1_6/test_stdout

[justinmc]

Another flakey failure happened, not sure if it's related. It was on Mac framework_tests_libraries though, and instead of this error the output seems to just end. https://logs.chromium.org/logs/flutter/buildbucket/cr-buildbucket/8824121197004872865/+/u/run_test.dart_for_framework_tests_shard_and_subshard_libraries/test_stdout

Update: Full build link instead: https://ci.chromium.org/ui/p/flutter/builders/prod/Mac%20framework_tests_libraries/3394/overview

[zanderso]

@justinmc Could you link to the page for the build instead of directly to the logs? Then we can look at the timings for the full run and not just the stdout for the failing step.

[christopherfujino]

These look like timeouts. @christopherfujino Do we already have an issue for this?

I recently increased the shard count. I suspect the problem is particular tests are disproportionately slow. Looking into this now...

[justinmc]

@zanderso Sorry, I've updated my comments above with the build links!

[christopherfujino]

These look like timeouts. @christopherfujino Do we already have an issue for this?

I recently increased the shard count. I suspect the problem is particular tests are disproportionately slow. Looking into this now...

In looking at the runtime of all the tests, I don't see particular bad tests. The five slowest tests are:

name	runtime of passing build	runtime of failing build
test\integration.shard\android_plugin_example_app_build_test.dart - plugin example can be built using current Flutter Gradle plugin	02:47	03:37
test\integration.shard\analyze_size_test.dart - --analyze-size allows overriding the directory for code size files	01:46	02:33
test\integration.shard\android_plugin_compilesdkversion_mismatch_test.dart - error logged when plugin Android compileSdkVersion higher than project	00:55	01:34
test\integration.shard\background_isolate_test.dart - Hot reload updates background isolates	00:21	01:18
test\integration.shard\command_output_test.dart - flutter run --machine uses AppRunLogger	00:55	01:06

[justinmc]

I saw another recent timeout on Windows web_tool_tests, not sure if it's related: https://ci.chromium.org/ui/p/flutter/builders/prod/Windows%20web_tool_tests/4662/overview

Same thing with "Terminate batch job? (Y/N)".

[jmagman]

Same thing with "Terminate batch job? (Y/N)".

You can see that one hit 30 minutes, which is the timeout.

.

[jmagman]

@CaseyHillers does someone from the LUCI team need to look at why these machines are crawling?

[justinmc]

Another timeout here. It was on Linux_android flutter_gallery__back_button_memory so it looked a bit different to me, but maybe it's just the same problem.

And here, but that one does seem to be the same thing as elsewhere in this issue.