Web hot restart tests flaky timeout frequently causing `web_tool_tests` failure #85043

zanderso · 2021-06-22T15:27:07Z

See https://ci.chromium.org/p/flutter/builders/prod/Windows%20web_tool_tests

In particular:

https://logs.chromium.org/logs/flutter/buildbucket/cr-buildbucket/8843718898922094865/+/u/run_test.dart_for_web_tool_tests_shard_and_subshard_web/stdout?format=raw

It looks like this started happening with an Engine roll containing only a Fuchsia SDK roll, so it is probably unrelated.

/cc @yjbanov @shihaohong

For #85043

zanderso · 2021-06-22T15:48:49Z

@annagrin

* Update hot_reload_web_test.dart For #85043 * Add missing import

alexmarkov · 2021-06-23T17:15:15Z

test/web.shard/hot_reload_web_test.dart also flaky times out on Linux on head-head-head bots:

https://logs.chromium.org/logs/dart/buildbucket/cr-buildbucket/8843692410606035153/+/u/flutter_test_web_tool_tests/stdout

https://logs.chromium.org/logs/dart/buildbucket/cr-buildbucket.appspot.com/8843646616757342784/+/u/flutter_test_tool_tests/stdout

jonahwilliams · 2021-06-25T16:14:11Z

looking in the logs I see some canvaskit errors. ill switch the apps to use html

zanderso · 2021-06-30T01:19:19Z

@yjbanov ptal

alexmarkov · 2021-07-12T20:17:27Z

The test failed on the head-head-head bot today in the following way:

01:57 +1 ~15 -1: test/web.shard/hot_reload_web_test.dart: hot restart works without error [E]                                                                                                          
  Exception: Exception: Received app.stop event while waiting for app.started event
  
  [ +101 ms] FINE: Pub 2.14.0-edge.d5a97be46681feb703031587bfc35945d9d36744
  [  +44 ms] SLVR: fact: test is 0.0.0
  [  +13 ms] SLVR: derived: test
  [  +29 ms] SLVR: fact: test depends on flutter any from sdk
  [   +1 ms] SLVR:   selecting test
  [        ] SLVR:   derived: flutter any from sdk
  [  +19 ms] SLVR:   fact: flutter 0.0.0 from sdk depends on characters 1.1.0
  [        ] SLVR:   fact: flutter 0.0.0 from sdk depends on collection 1.15.0
  [        ] SLVR:   fact: flutter 0.0.0 from sdk depends on meta 1.7.0
  [        ] SLVR:   fact: flutter 0.0.0 from sdk depends on typed_data 1.3.0
  [        ] SLVR:   fact: flutter 0.0.0 from sdk depends on vector_math 2.1.0
  [        ] SLVR:   fact: flutter 0.0.0 from sdk depends on sky_engine any from sdk
  [   +1 ms] SLVR:     selecting flutter 0.0.0 from sdk
  [        ] SLVR:     derived: sky_engine any from sdk
  [        ] SLVR:     derived: vector_math 2.1.0
  [        ] SLVR:     derived: typed_data 1.3.0
  [        ] SLVR:     derived: meta 1.7.0
  [        ] SLVR:     derived: collection 1.15.0
  [        ] SLVR:     derived: characters 1.1.0
  [   +4 ms] SLVR:       selecting sky_engine 0.0.99 from sdk
  [   +8 ms] SLVR:         selecting vector_math 2.1.0
  [   +6 ms] SLVR:         fact: typed_data 1.3.0 depends on collection ^1.15.0
  [        ] SLVR:           selecting typed_data 1.3.0
  [   +3 ms] SLVR:             selecting meta 1.7.0
  [   +2 ms] SLVR:               selecting collection 1.15.0
  [   +2 ms] SLVR:                 selecting characters 1.1.0
  [  +87 ms] SLVR: Version solving took 0:00:00.192917 seconds.
  [        ]     | Tried 1 solutions.
  [        ] FINE: Resolving dependencies finished (0.236s).
  [  +68 ms] IO  : Writing 1105 characters to text file ./pubspec.lock.
  [        ] FINE: Contents:
  [        ]     | # Generated by pub
  [        ]     | # See https://dart.dev/tools/pub/glossary#lockfile
  [        ]     | packages:
  [        ]     |   characters:
  [        ]     |     dependency: transitive
  [        ]     |     description:
  [        ]     |       name: characters
  [        ]     |       url: "https://pub.dartlang.org"
  [        ]     |     source: hosted
  [        ]     |     version: "1.1.0"
  [        ]     |   collection:
  [        ]     |     dependency: transitive
  [        ]     |     description:
  [        ]     |       name: collection
  [        ]     |       url: "https://pub.dartlang.org"
  [        ]     |     source: hosted
  [        ]     |     version: "1.15.0"
  [        ]     |   flutter:
  [        ]     |     dependency: "direct main"
  [        ]     |     description: flutter
  [        ]     |     source: sdk
  [        ]     |     version: "0.0.0"
  [        ]     |   meta:
  [        ]     |     dependency: transitive
  [        ]     |     description:
  [        ]     |       name: meta
  [        ]     |       url: "https://pub.dartlang.org"
  [        ]     |     source: hosted
  [        ]     |     version: "1.7.0"
  [        ]     |   sky_engine:
  [        ]     |     dependency: transitive
  [        ]     |     description: flutter
  [        ]     |     source: sdk
  [        ]     |     version: "0.0.99"
  [        ]     |   typed_data:
  [        ]     |     dependency: transitive
  [        ]     |     description:
  [        ]     |       name: typed_data
  [        ]     |       url: "https://pub.dartlang.org"
  [        ]     |     source: hosted
  [        ]     |     version: "1.3.0"
  [        ]     |   vector_math:
  [        ]     |     dependency: transitive
  [        ]     |     description:
  [        ]     |       name: vector_math
  [        ]     |       url: "https://pub.dartlang.org"
  [        ]     |     source: hosted
  [        ]     |     version: "2.1.0"
  [        ]     | sdks:
  [        ]     |   dart: ">=2.12.0 <3.0.0"
  [   +3 ms] IO  : Writing 826 characters to text file ./.packages.
  [        ] FINE: Contents:
  [        ]     | # This file is deprecated. Tools should instead consume 
  [        ]     | # `.dart_tools/package_config.json`.
  [        ]     | # 
  [        ]     | # For more info see: https://dart.dev/go/dot-packages-deprecation
  [        ]     | # 
  [        ]     | # Generated by pub on 2021-07-12 12:52:47.642237.
  [        ]     | characters:file:///home/chrome-bot/.pub-cache/hosted/pub.dartlang.org/characters-1.1.0/lib/
  [        ]     | collection:file:///home/chrome-bot/.pub-cache/hosted/pub.dartlang.org/collection-1.15.0/lib/
  [        ]     | flutter:file:///b/s/w/ir/cache/builder/flutter/packages/flutter/lib/
  [        ]     | meta:file:///home/chrome-bot/.pub-cache/hosted/pub.dartlang.org/meta-1.7.0/lib/
  [        ]     | sky_engine:file:///b/s/w/ir/cache/builder/flutter/bin/cache/pkg/sky_engine/lib/
  [        ]     | typed_data:file:///home/chrome-bot/.pub-cache/hosted/pub.dartlang.org/typed_data-1.3.0/lib/
  [        ]     | vector_math:file:///home/chrome-bot/.pub-cache/hosted/pub.dartlang.org/vector_math-2.1.0/lib/
  [        ]     | test:lib/
  [   +9 ms] IO  : Writing 1645 characters to text file ./.dart_tool/package_config.json.
  [        ] FINE: Contents:
  [        ]     | {
  [        ]     |   "configVersion": 2,
  [        ]     |   "packages": [
  [        ]     |     {
  [        ]     |       "name": "characters",
  [        ]     |       "rootUri": "file:///home/chrome-bot/.pub-cache/hosted/pub.dartlang.org/characters-1.1.0",
  [        ]     |       "packageUri": "lib/",
  [        ]     |       "languageVersion": "2.12"
  [        ]     |     },
  [        ]     |     {
  [        ]     |       "name": "collection",
  [        ]     |       "rootUri": "file:///home/chrome-bot/.pub-cache/hosted/pub.dartlang.org/collection-1.15.0",
  [        ]     |       "packageUri": "lib/",
  [        ]     |       "languageVersion": "2.12"
  [        ]     |     },
  [        ]     |     {
  [        ]     |       "name": "flutter",
  [        ]     |       "rootUri": "file:///b/s/w/ir/cache/builder/flutter/packages/flutter",
  [        ]     |       "packageUri": "lib/",
  [        ]     |       "languageVersion": "2.12"
  [        ]     |     },
  [        ]     |     {
  [        ]     |       "name": "meta",
  [        ]     |       "rootUri": "file:///home/chrome-bot/.pub-cache/hosted/pub.dartlang.org/meta-1.7.0",
  [        ]     |       "packageUri": "lib/",
  [        ]     |       "languageVersion": "2.12"
  [        ]     |     },
  [        ]     |     {
  [        ]     |       "name": "sky_engine",
  [        ]     |       "rootUri": "file:///b/s/w/ir/cache/builder/flutter/bin/cache/pkg/sky_engine",
  [        ]     |       "packageUri": "lib/",
  [        ]     |       "languageVersion": "2.12"
  [        ]     |     },
  [        ]     |     {
  [        ]     |       "name": "typed_data",
  [        ]     |       "rootUri": "file:///home/chrome-bot/.pub-cache/hosted/pub.dartlang.org/typed_data-1.3.0",
  [        ]     |       "packageUri": "lib/",
  [        ]     |       "languageVersion": "2.12"
  [        ]     |     },
  [        ]     |     {
  [        ]     |       "name": "vector_math",
  [        ]     |       "rootUri": "file:///home/chrome-bot/.pub-cache/hosted/pub.dartlang.org/vector_math-2.1.0",
  [        ]     |       "packageUri": "lib/",
  [        ]     |       "languageVersion": "2.12"
  [        ]     |     },
  [        ]     |     {
  [        ]     |       "name": "test",
  [        ]     |       "rootUri": "../",
  [        ]     |       "packageUri": "lib/",
  [        ]     |       "languageVersion": "2.12"
  [        ]     |     }
  [        ]     |   ],
  [        ]     |   "generated": "2021-07-12T19:52:47.656560Z",
  [        ]     |   "generator": "pub",
  [        ]     |   "generatorVersion": "2.14.0-edge.d5a97be46681feb703031587bfc35945d9d36744"
  [        ]     | }
  Failed to establish connection with the application instance in Chrome.
  This can happen if the websocket connection used by the web tooling is unable to correctly establish a connection, for example due to a firewall.
  
  #0      throwToolExit (package:flutter_tools/src/base/common.dart:10:3)
  #1      ResidentWebRunner.run (package:flutter_tools/src/isolated/resident_web_runner.dart:328:7)
  <asynchronous suspension>
  #2      AppDomain.launch.<anonymous closure> (package:flutter_tools/src/commands/daemon.dart:595:9)
  <asynchronous suspension>
  #3      AppDomain.launch (package:flutter_tools/src/commands/daemon.dart:593:5)
  <asynchronous suspension>
  #4      RunCommand.runCommand (package:flutter_tools/src/commands/run.dart:552:15)
  <asynchronous suspension>
  #5      FlutterCommand.run.<anonymous closure> (package:flutter_tools/src/runner/flutter_command.dart:1126:27)
  <asynchronous suspension>
  #6      AppContext.run.<anonymous closure> (package:flutter_tools/src/base/context.dart:150:19)
  <asynchronous suspension>
  #7      CommandRunner.runCommand (package:args/command_runner.dart:196:13)
  <asynchronous suspension>
  #8      FlutterCommandRunner.runCommand.<anonymous closure> (package:flutter_tools/src/runner/flutter_command_runner.dart:288:9)
  <asynchronous suspension>
  #9      AppContext.run.<anonymous closure> (package:flutter_tools/src/base/context.dart:150:19)
  <asynchronous suspension>
  #10     FlutterCommandRunner.runCommand (package:flutter_tools/src/runner/flutter_command_runner.dart:236:5)
  <asynchronous suspension>
  #11     run.<anonymous closure>.<anonymous closure> (package:flutter_tools/runner.dart:62:9)
  <asynchronous suspension>
  #12     AppContext.run.<anonymous closure> (package:flutter_tools/src/base/context.dart:150:19)
  <asynchronous suspension>
  #13     main (package:flutter_tools/executable.dart:92:3)
  <asynchronous suspension>
  
  
  
  

01:57 +1 ~15 -1: loading test/web.shard/hot_reload_web_test.dart                                                                                                                                       
Consider enabling the flag chain-stack-traces to receive more detailed exceptions.
For example, 'pub run test --chain-stack-traces'.

annagrin · 2021-07-12T22:04:58Z

@alexmarkov could you please share a link to the failure? I am only seeing engine tests failing currently...

alexmarkov · 2021-07-12T22:08:27Z

@annagrin Here is the link: https://logs.chromium.org/logs/dart/buildbucket/cr-buildbucket.appspot.com/8841900348628040752/+/u/flutter_test_tool_tests/stdout

annagrin · 2021-07-13T19:01:49Z

Looks like the HHH bot is fine now, so the issue seems to be intermittent. Created a separate issue to investigate (above).

yjbanov · 2021-07-21T03:38:53Z

@annagrin Would you mind taking this issue? This is outside my domain expertise.

jonahwilliams · 2021-07-22T17:09:08Z

The failing test is "hot_reload_web_test.dart: newly added code executes during hot restart - canvaskit"

This one should still be disabled since the flakiness is caused by downloading canvaskit over the network

zanderso · 2021-07-22T17:22:48Z

#86885

fluttergithubbot · 2021-07-26T16:00:05Z

Current flaky ratio for the past 15 days is 0.63%.
One recent flaky example for a same commit: https://ci.chromium.org/ui/p/flutter/builders/prod/Windows%20web_tool_tests/2876
Commit: c2f09e0
Flaky builds:
https://ci.chromium.org/ui/p/flutter/builders/prod/Windows%20web_tool_tests/2876

zanderso · 2021-07-26T17:23:38Z

My understanding is that the current status of this is that it is waiting for the canvaskit pin to re-enable the disabled test. Dropping priority based on that.

annagrin · 2021-07-26T18:48:44Z

@annagrin This issue has a tag indicating that a test was disabled for it. Did you also re-enable the tests that were disabled?

@zanderso Looks like this issue bundles a few issues together - hot restart hangs, errors on connection to chrome, and canvaskit binding errors. I enabled two tests in packages/flutter_tools/test/web.shard/hot_reload_web_test.dart but it looks like one of them is still failing due to canvaskit issue. Thanks for disabling it!

I'll keep monitoring this issue to see if my fix stops flakiness due to hot restart hangs.

fluttergithubbot · 2021-08-02T16:00:06Z

Current flaky ratio for the past 15 days is 0.56%.
One recent flaky example for a same commit: https://ci.chromium.org/ui/p/flutter/builders/prod/Windows%20web_tool_tests/2876
Commit: c2f09e0
Flaky builds:
https://ci.chromium.org/ui/p/flutter/builders/prod/Windows%20web_tool_tests/2876

fluttergithubbot · 2021-08-09T16:00:06Z

Current flaky ratio for the past 15 days is 0.00%.

fluttergithubbot · 2021-08-16T16:00:05Z

Current flaky ratio for the past 15 days is 0.00%.

fluttergithubbot · 2021-08-23T16:00:07Z

Current flaky ratio for the past 15 days is 0.76%.
One recent flaky example for a same commit: https://ci.chromium.org/ui/p/flutter/builders/prod/Windows%20web_tool_tests/3143
Commit: 3216e9f
Flaky builds:
https://ci.chromium.org/ui/p/flutter/builders/prod/Windows%20web_tool_tests/3143

annagrin · 2021-08-25T21:56:45Z

A new flake - looks like we are not retrying when connecting to a chrome tab:

flutter/packages/flutter_tools/lib/src/isolated/resident_web_runner.dart

Line 543 in 44d9af8

    
           final ChromeTab chromeTab = await chrome.chromeConnection.getTab((ChromeTab chromeTab) {

/cc @jonahwilliams I think we could just pass a retry: 5 or some number there?

fluttergithubbot · 2021-08-30T16:00:07Z

Current flaky ratio for the past 15 days is 0.57%.
One recent flaky example for a same commit: https://ci.chromium.org/ui/p/flutter/builders/prod/Windows%20web_tool_tests/3143
Commit: 3216e9f
Flaky builds:
https://ci.chromium.org/ui/p/flutter/builders/prod/Windows%20web_tool_tests/3143

zanderso · 2021-08-30T22:32:24Z

@annagrin I'm not sure @jonahwilliams is reading here anymore if that question was directed at him. If a retry there could succeed after an initial failure, then that sounds like something that could deflake the test.

fluttergithubbot · 2021-09-06T16:00:04Z

Current flaky ratio for the past 15 days is 0.00%.

annagrin · 2021-09-07T18:32:11Z

@zanderso I will add the retries to see if it has any effect, but it looks like the occurrences are pretty rare at the moment.

fluttergithubbot · 2021-09-13T16:00:07Z

Current flaky ratio for the past 15 days is 0.00%.

fluttergithubbot · 2021-09-20T16:00:05Z

Current flaky ratio for the past 15 days is 0.00%.

annagrin · 2021-09-30T00:55:47Z

Closing this as there have been no flakiness for a month, please feel free to reopen if needed!

github-actions · 2021-10-14T01:22:23Z

This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.

zanderso added a: tests "flutter test", flutter_test, or one of our tests team: flakes platform-web Web applications specifically c: disabled test Tracks the restoration of a disabled test labels Jun 22, 2021

zanderso added a commit that referenced this issue Jun 22, 2021

Skip a hot_reload_web_test on Windows

f66b4d2

For #85043

zanderso added a commit that referenced this issue Jun 22, 2021

Update hot_reload_web_test.dart

1e59c94

For #85043

zanderso mentioned this issue Jun 22, 2021

Skip a flaky Windows/canvaskit test in hot_reload_web_test.dart #85044

Merged

zanderso added the tool Affects the "flutter" command-line tool. See also t: labels. label Jun 22, 2021

zanderso added a commit that referenced this issue Jun 22, 2021

Skip a flaky Windows/canvaskit test in hot_reload_web_test.dart (#85044)

1cf2ce9

* Update hot_reload_web_test.dart For #85043 * Add missing import

keyonghan changed the title ~~We hot restart test flaky timeout frequently on Windows/canvaskit~~ We hot restart test flaky timeout frequently on Windows/canvaskit, causing Windows web_tool_tests 2.74% flaky Jun 22, 2021

keyonghan added the c: flake Tests that sometimes, but not always, incorrectly pass label Jun 22, 2021

zanderso changed the title ~~We hot restart test flaky timeout frequently on Windows/canvaskit, causing Windows web_tool_tests 2.74% flaky~~ Web hot restart test flaky timeout frequently on Windows causing Windows web_tool_tests 2.74% flaky Jun 30, 2021

zanderso added the P1 label Jun 30, 2021

zanderso mentioned this issue Jun 30, 2021

Linux web_tool_tests is flaky: newly added code executes during hot restart: TimeoutException after 0:00:15.000000: Future not completed #85575

Closed

zanderso assigned yjbanov Jun 30, 2021

zanderso changed the title ~~Web hot restart test flaky timeout frequently on Windows causing Windows web_tool_tests 2.74% flaky~~ Web hot restart tests flaky timeout frequently causing web_tool_tests failure Jun 30, 2021

annagrin mentioned this issue Jul 13, 2021

Intermittent failures in web_tool_tests due to failure to connect to chrome #86370

Open

annagrin mentioned this issue Jul 21, 2021

Hot restart hangs periodically in webdev and flutter CI dart-lang/webdev#1359

Closed

annagrin assigned annagrin and unassigned yjbanov Jul 21, 2021

zanderso mentioned this issue Jul 22, 2021

hot_reload_web_test is causing Linux web_tool_tests (3.09%) and Windows web_tool_tests flaky #86884

Closed

fluttergithubbot added P2 and removed P1 labels Jul 26, 2021

zanderso added P1 High-priority issues at the top of the work list and removed P2 labels Jul 26, 2021

annagrin mentioned this issue Jul 26, 2021

expression_evaluation_web_test failed, "Exception: Received app.stop event while waiting for app.started event", causing Linux web_tool_tests 2.29% flaky #84012

Closed

annagrin mentioned this issue Jul 28, 2021

web_tool_tests 'Unable to connect to Chrome debug port:...' #87148

Closed

annagrin mentioned this issue Sep 20, 2021

web_long_running_tests is flaky #86375

Closed

annagrin closed this as completed Sep 30, 2021

github-actions bot locked as resolved and limited conversation to collaborators Oct 14, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Web hot restart tests flaky timeout frequently causing `web_tool_tests` failure #85043

Web hot restart tests flaky timeout frequently causing `web_tool_tests` failure #85043

zanderso commented Jun 22, 2021 •

edited by chunhtai

zanderso commented Jun 22, 2021

alexmarkov commented Jun 23, 2021 •

edited

jonahwilliams commented Jun 25, 2021

zanderso commented Jun 30, 2021

alexmarkov commented Jul 12, 2021

annagrin commented Jul 12, 2021

alexmarkov commented Jul 12, 2021

annagrin commented Jul 13, 2021

yjbanov commented Jul 21, 2021 •

edited

jonahwilliams commented Jul 22, 2021

zanderso commented Jul 22, 2021

fluttergithubbot commented Jul 26, 2021

zanderso commented Jul 26, 2021

annagrin commented Jul 26, 2021

fluttergithubbot commented Aug 2, 2021

fluttergithubbot commented Aug 9, 2021

fluttergithubbot commented Aug 16, 2021

fluttergithubbot commented Aug 23, 2021

annagrin commented Aug 25, 2021

fluttergithubbot commented Aug 30, 2021

zanderso commented Aug 30, 2021

fluttergithubbot commented Sep 6, 2021

annagrin commented Sep 7, 2021

fluttergithubbot commented Sep 13, 2021

fluttergithubbot commented Sep 20, 2021

annagrin commented Sep 30, 2021

github-actions bot commented Oct 14, 2021

Web hot restart tests flaky timeout frequently causing web_tool_tests failure #85043

Web hot restart tests flaky timeout frequently causing web_tool_tests failure #85043

Comments

zanderso commented Jun 22, 2021 • edited by chunhtai

zanderso commented Jun 22, 2021

alexmarkov commented Jun 23, 2021 • edited

jonahwilliams commented Jun 25, 2021

zanderso commented Jun 30, 2021

alexmarkov commented Jul 12, 2021

annagrin commented Jul 12, 2021

alexmarkov commented Jul 12, 2021

annagrin commented Jul 13, 2021

yjbanov commented Jul 21, 2021 • edited

jonahwilliams commented Jul 22, 2021

zanderso commented Jul 22, 2021

fluttergithubbot commented Jul 26, 2021

zanderso commented Jul 26, 2021

annagrin commented Jul 26, 2021

fluttergithubbot commented Aug 2, 2021

fluttergithubbot commented Aug 9, 2021

fluttergithubbot commented Aug 16, 2021

fluttergithubbot commented Aug 23, 2021

annagrin commented Aug 25, 2021

fluttergithubbot commented Aug 30, 2021

zanderso commented Aug 30, 2021

fluttergithubbot commented Sep 6, 2021

annagrin commented Sep 7, 2021

fluttergithubbot commented Sep 13, 2021

fluttergithubbot commented Sep 20, 2021

annagrin commented Sep 30, 2021

github-actions bot commented Oct 14, 2021

Web hot restart tests flaky timeout frequently causing `web_tool_tests` failure #85043

Web hot restart tests flaky timeout frequently causing `web_tool_tests` failure #85043

zanderso commented Jun 22, 2021 •

edited by chunhtai

alexmarkov commented Jun 23, 2021 •

edited

yjbanov commented Jul 21, 2021 •

edited