New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Engine roll causing observatory connection timeouts on 32-bit iPhones #40114
Comments
Note that reverting this also fixed a problem in |
Should we still keep the |
Yes, this should stay TODAY, as the engine auto-roller is blocked on reverting the culprit in the engine or Dart SDK. |
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
Reverting the engine rolls that seem to have caused the following devicelab tests to start timing out: flutter_gallery_ios32__transition_perf and flutter_gallery_ios32__start_up. Specifically, they started failing when af07bb5 was committed.
git@github.com:flutter/engine.git/compare/7ea9884ab00e...50bdbd7 git log 7ea9884..50bdbd7 --no-merges --oneline 2019-09-06 egarciad@google.com Document dependencies and remove support-v13 (flutter/engine#11912) 2019-09-06 bkonyi@google.com Roll src/third_party/dart be66176534..ec7ec4ecf7 (37 commits) 2019-09-06 skia-flutter-autoroll@skia.org Roll src/third_party/skia 080d210e7acc..c30f1a936d84 (21 commits) (flutter/engine#12031) If this roll has caused a breakage, revert this CL and stop the roller using the controls here: https://autoroll.skia.org/r/flutter-engine-flutter-autoroll Please CC franciscojma@google.com on the revert to ensure that a human is aware of the problem. To report a problem with the AutoRoller itself, please file a bug: https://bugs.chromium.org/p/skia/issues/entry?template=Autoroller+Bug Documentation for the AutoRoller is here: https://skia.googlesource.com/buildbot/+/master/autoroll/README.md
…nline with topaz. (flutter/engine#12034) (#40018) git@github.com:flutter/engine.git/compare/50bdbd769eee...c9ea4db git log 50bdbd7..c9ea4db --no-merges --oneline 2019-09-06 iska.kaushik@gmail.com [flutter_runner] Refactor our build rules to make them more inline with topaz. (flutter/engine#12034) If this roll has caused a breakage, revert this CL and stop the roller using the controls here: https://autoroll.skia.org/r/flutter-engine-flutter-autoroll Please CC franciscojma@google.com on the revert to ensure that a human is aware of the problem. To report a problem with the AutoRoller itself, please file a bug: https://bugs.chromium.org/p/skia/issues/entry?template=Autoroller+Bug Documentation for the AutoRoller is here: https://skia.googlesource.com/buildbot/+/master/autoroll/README.md
Okay, I can reliably reproduce this now. It has to do with profile mode specifically. Just The log I mentioned above doesn't appear with this. I think it was just a one-off local build error. |
@mklim thanks for looking into it. Do you know what is causing the app to crash? |
@blasten not yet, picking up the investigation again this morning. |
I've bisected this to a Dart roll, flutter/engine@0c6a538. |
Can we revert the engine back to the Dart revision before that commit, pause the Dart SDK rolls until this is resolved, and resume the engine auto-rolls? |
I'll roll back the Dart revision in the engine. |
Obfuscated stack trace from the Console, seems to be the only thing that has logs:
|
I've created a manual roll of the engine with the Dart SDK revert at #40275. It doesn't appear to have this particular regression, but it's also still going through CI and has 64 commits since the last roll. |
Culprit seems to be dart-lang/sdk@c219a76 |
The manual roll has landed and the engine to framework roller has been turned back on. The dart to engine roller is still paused. |
Thanks @mklim - I think we can remove the TODAY label now to unblock dev rolls. |
Sent out https://dart-review.googlesource.com/c/sdk/+/116883 with revert in dart sdk, so dart-to-engine roll can be resumed |
…m snapshots (excl. blobs).' as it breaks flutter profile execution on ios 32-bit platform. Fixes flutter/flutter#40114 Change-Id: If8d71e9c19c2e794d29f7ecbacb87457890a2fd5 Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/116883 Reviewed-by: Alexander Markov <alexmarkov@google.com> Commit-Queue: Alexander Aprelev <aam@google.com>
@mklim What app did you run to reproduce the crash? Also, did it happen on 64-bit iOS or only 32-bit? |
@sjindel-google I used flutter gallery (in profile/release mode). ios 32-bit was the only affected as far as I know. |
@sjindel-google my testing was the same as @aam's. |
I've resumed the dart to engine roller since the revert landed in dart-lang/sdk@0c481a1. I think it's safe to consider this fixed now. |
There are overlapping TODAY issues (#40303 and #40114) that prevent this roll from being split up further. git log 7ea9884..3c6383f --no-merges --oneline 3c6383f Revert "Smooth out iOS irregular input events delivery (#11817)" (#12251) d6f0b64 pin and auto-install chrome version (#12228) 2698a0e Roll fuchsia/sdk/core/mac-amd64 from Ne2UA... to JVZ_i... (#12250) dfa9498 Enable platform view keyboard input on Android Q (#12085) bf91a8d Roll fuchsia/sdk/core/linux-amd64 from u7Q31... to vuG5q... (#12238) 58ecf52 Roll src/third_party/skia 7c47d41067d4..be194479d27f (4 commits) (#12237) c71580b Roll dart to e6887536aadc7fbd1990448989601cee0224958d. (#12235) cf1d156 Roll fuchsia/sdk/core/mac-amd64 from _nS67... to Ne2UA... (#12236) bbb1f12 Adjust iOS frame start times to match the platform info (#11802) 1de28d0 Roll src/third_party/skia 50f377e275c3..7c47d41067d4 (3 commits) (#12231) da84d59 Revert "Manage iOS contexts separately (#12078)" (#12233) 4ac0663 Manage iOS contexts separately (#12078) 28d7900 Roll src/third_party/skia 120e7d6766e4..50f377e275c3 (7 commits) (#12224) 5b94c8a Revert "Roll src/third_party/dart be66176534..ec7ec4ecf7 (37 commits)" (#12223) 988efe3 Do not generate kernel platform files on topaz tree (#12222) 6c46a17 Don't disable toString in release mode for dart:ui classes (#12204) 80b8ed8 Roll fuchsia/sdk/core/linux-amd64 from 7gDBN... to u7Q31... (#12221) 7a8caaa Roll src/third_party/skia d96ef09317d6..120e7d6766e4 (2 commits) (#12220) c8428ff Roll fuchsia/sdk/core/mac-amd64 from vDk46... to _nS67... (#12219) 2bdfb61 Namespace patched SDK names to not conflict with Topaz (#12218) ff1fcfb Roll src/third_party/dart ca7baa4013..4d5e15abde (29 commits) 2b78c59 Roll src/third_party/skia 14318c140949..d96ef09317d6 (2 commits) (#12216) d74ed76 Roll src/third_party/skia b23a4f9d9442..14318c140949 (2 commits) (#12215) c58c593 Roll src/third_party/skia 26ac0467cb4c..b23a4f9d9442 (2 commits) (#12214) 5e85403 Roll src/third_party/dart 7bbfd532de..ca7baa4013 (3 commits) e2cc04c Roll fuchsia/sdk/core/linux-amd64 from R1yqu... to 7gDBN... (#12212) 6fbfb45 Roll fuchsia/sdk/core/mac-amd64 from spUG2... to vDk46... (#12210) 96443e2 Roll buildroot and Fuchsia toolchain and unblock the Fuchsia/Linux autoroller manually. (#12209) 34cf4f7 Roll src/third_party/skia 4fe30e15c06c..26ac0467cb4c (2 commits) (#12207) 92d42c0 Only build the x64 variant of Fuchsia on the try-jobs. (#12206) e174b4b Don't load Roboto by default (#12205) efb32a6 Roll src/third_party/dart 300c3333d1..7bbfd532de (5 commits) 5566be1 Roll src/third_party/skia 66d8006c2bb1..4fe30e15c06c (11 commits) (#12202) e9c9984 add a convenience CLI tool for building and testing web engine (#12197) bfa43e1 [flutter_runner] Generate symbols for the Dart VM profiler (#12048) 954f198 Add custom test plugin that supports screenshot tests (#12079) d8379f9 Move the Fuchsia tryjob into a its own step and disable LTO. (#12190) c12ac24 Roll src/third_party/dart 62f78a7abb..300c3333d1 (6 commits) cab3a39 Roll src/third_party/skia b88894c8811b..66d8006c2bb1 (5 commits) (#12178) 2592d6e [flutter_runner] Port the accessibility bridge from Topaz (#12054) b569e8c Smooth out iOS irregular input events delivery (#11817) dea813d Roll src/third_party/dart ccb6ba948b..62f78a7abb (3 commits) 2438798 Make ImageShader implement Shader for web ui (#12161) c31583a Roll src/third_party/dart 2e8d912848..ccb6ba948b (30 commits) 4542886 Roll src/third_party/skia 9e5c47936b17..b88894c8811b (3 commits) (#12151) 548998f Roll src/third_party/skia 1bf30ce852e0..9e5c47936b17 (2 commits) (#12129) f1490a2 Roll src/third_party/skia 8cae1e95a23b..1bf30ce852e0 (2 commits) (#12106) 19b2d43 Roll fuchsia/clang/mac-amd64 from H1Qjc... to HfPKR... (#12088) d816755 Don't launch the observatory by default on each embedder unit-test invocation. (#12087) 39c8067 Roll src/third_party/skia c2d84bfa7421..8cae1e95a23b (4 commits) (#12086) b19e75a Roll src/third_party/dart fb14babf59..2e8d912848 (65 commits) 03e773a Guard availability of user notification related methods to iOS 10.0 (#12084) 9c00c26 Add capability to add AppDelegate as UNUserNotificationCenterDelegate (#9864) c2e8289 Add GradientRadial paintStyle implementation (#12081) c3eea0a Don't quote generic font families (#12080) c2b3d88 Roll src/third_party/skia 28d40b2e7ade..c2d84bfa7421 (3 commits) (#12082) a4de006 Remove ENABLE_BITCODE from Scenarios test app (#11839) 6e017f0 Roll src/third_party/skia 4f2674da4bbc..28d40b2e7ade (4 commits) (#12077) e911b05 Roll src/third_party/skia 627d15588f4d..4f2674da4bbc (1 commits) (#12075) 359e663 Roll src/third_party/skia 6c3bd09ead0f..627d15588f4d (3 commits) (#12074) 38d545e Improve Unicode handling on Windows (#11899) 89efb4c Roll fuchsia/clang/linux-amd64 from VoYNW... to 2IT_b... (#12072) dc8e30d Roll fuchsia/clang/mac-amd64 from XAazI... to H1Qjc... (#12071) 8cdb3af Annotate nullability on FlutterEngine to make swift writing more ergonomic (#11808) aa9aaa2 Roll src/third_party/skia f433336585ed..6c3bd09ead0f (1 commits) (#12070) 075a61f Roll src/third_party/skia 69a426f5a427..f433336585ed (1 commits) (#12068) a610505 option for --no-lto for fuchsia (#12010) be39820 Roll src/third_party/skia 380561393385..69a426f5a427 (2 commits) (#12067) 6bafbf9 Roll src/third_party/skia c30f1a936d84..380561393385 (3 commits) (#12059) e2ba93d Roll src/third_party/dart ec7ec4ecf7..fb14babf59 (19 commits) 723a288 Roll fuchsia/clang/linux-amd64 from -mnHl... to VoYNW... (#12058) 35875e0 Revert "Manage resource and onscreen contexts using separate IOSGLContext objects (#11798)" (#12055) a353f93 Manage resource and onscreen contexts using separate IOSGLContext objects (#11798) c9ea4db [flutter_runner] Refactor our build rules to make them more inline with topaz. (#12034) 50bdbd7 Document dependencies and remove support-v13 (#11912) 0c6a538 Roll src/third_party/dart be66176534..ec7ec4ecf7 (37 commits) 37f81cd Roll src/third_party/skia 080d210e7acc..c30f1a936d84 (21 commits) (#12031)
We have had several bugs propagate to Flutter end-users due to our lack of testing of AOT Assembly snapshots on ARM (SIMARM_X64 in particular), causing reverts of SDK commits and disrupting the Dart -> Flutter roll. This leverages our custom ELF loader to load the generated ELF objects, which cannot be loaded by dlopen() because they are marked with the ARM ABI. For example, see: - flutter/flutter#40114 - flutter/flutter#41228 Also do some tree-shaking of the snapshot writers for dart_precompiled_runtime. Fixes #38433. Change-Id: I5990769c623b8a13131b1848349af2c0a026272b Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/118569 Reviewed-by: Clement Skau <cskau@google.com> Reviewed-by: Alexander Thomas <athom@google.com> Commit-Queue: Samir Jindel <sjindel@google.com>
…mp-linux-debug-simarm_x64. The reason is that we can test Assembly snapshot generation on Linux but not on Mac. We have had several bugs propagate to Flutter end-users due to our lack of testing of AOT Assembly snapshots on ARM (SIMARM_X64 in particular), causing reverts of SDK commits and disrupting the Dart -> Flutter roll. For example, see: - flutter/flutter#40114 - flutter/flutter#41228 Change-Id: Ided8116e8230e58ffd350447b2e1a22676bd0506 Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/118584 Reviewed-by: Alexander Thomas <athom@google.com> Commit-Queue: Samir Jindel <sjindel@google.com>
There are overlapping TODAY issues (flutter#40303 and flutter#40114) that prevent this roll from being split up further. git log 7ea9884..3c6383f --no-merges --oneline 3c6383f Revert "Smooth out iOS irregular input events delivery (flutter#11817)" (flutter#12251) d6f0b64 pin and auto-install chrome version (flutter#12228) 2698a0e Roll fuchsia/sdk/core/mac-amd64 from Ne2UA... to JVZ_i... (flutter#12250) dfa9498 Enable platform view keyboard input on Android Q (flutter#12085) bf91a8d Roll fuchsia/sdk/core/linux-amd64 from u7Q31... to vuG5q... (flutter#12238) 58ecf52 Roll src/third_party/skia 7c47d41067d4..be194479d27f (4 commits) (flutter#12237) c71580b Roll dart to e6887536aadc7fbd1990448989601cee0224958d. (flutter#12235) cf1d156 Roll fuchsia/sdk/core/mac-amd64 from _nS67... to Ne2UA... (flutter#12236) bbb1f12 Adjust iOS frame start times to match the platform info (flutter#11802) 1de28d0 Roll src/third_party/skia 50f377e275c3..7c47d41067d4 (3 commits) (flutter#12231) da84d59 Revert "Manage iOS contexts separately (flutter#12078)" (flutter#12233) 4ac0663 Manage iOS contexts separately (flutter#12078) 28d7900 Roll src/third_party/skia 120e7d6766e4..50f377e275c3 (7 commits) (flutter#12224) 5b94c8a Revert "Roll src/third_party/dart be66176534..ec7ec4ecf7 (37 commits)" (flutter#12223) 988efe3 Do not generate kernel platform files on topaz tree (flutter#12222) 6c46a17 Don't disable toString in release mode for dart:ui classes (flutter#12204) 80b8ed8 Roll fuchsia/sdk/core/linux-amd64 from 7gDBN... to u7Q31... (flutter#12221) 7a8caaa Roll src/third_party/skia d96ef09317d6..120e7d6766e4 (2 commits) (flutter#12220) c8428ff Roll fuchsia/sdk/core/mac-amd64 from vDk46... to _nS67... (flutter#12219) 2bdfb61 Namespace patched SDK names to not conflict with Topaz (flutter#12218) ff1fcfb Roll src/third_party/dart ca7baa4013..4d5e15abde (29 commits) 2b78c59 Roll src/third_party/skia 14318c140949..d96ef09317d6 (2 commits) (flutter#12216) d74ed76 Roll src/third_party/skia b23a4f9d9442..14318c140949 (2 commits) (flutter#12215) c58c593 Roll src/third_party/skia 26ac0467cb4c..b23a4f9d9442 (2 commits) (flutter#12214) 5e85403 Roll src/third_party/dart 7bbfd532de..ca7baa4013 (3 commits) e2cc04c Roll fuchsia/sdk/core/linux-amd64 from R1yqu... to 7gDBN... (flutter#12212) 6fbfb45 Roll fuchsia/sdk/core/mac-amd64 from spUG2... to vDk46... (flutter#12210) 96443e2 Roll buildroot and Fuchsia toolchain and unblock the Fuchsia/Linux autoroller manually. (flutter#12209) 34cf4f7 Roll src/third_party/skia 4fe30e15c06c..26ac0467cb4c (2 commits) (flutter#12207) 92d42c0 Only build the x64 variant of Fuchsia on the try-jobs. (flutter#12206) e174b4b Don't load Roboto by default (flutter#12205) efb32a6 Roll src/third_party/dart 300c3333d1..7bbfd532de (5 commits) 5566be1 Roll src/third_party/skia 66d8006c2bb1..4fe30e15c06c (11 commits) (flutter#12202) e9c9984 add a convenience CLI tool for building and testing web engine (flutter#12197) bfa43e1 [flutter_runner] Generate symbols for the Dart VM profiler (flutter#12048) 954f198 Add custom test plugin that supports screenshot tests (flutter#12079) d8379f9 Move the Fuchsia tryjob into a its own step and disable LTO. (flutter#12190) c12ac24 Roll src/third_party/dart 62f78a7abb..300c3333d1 (6 commits) cab3a39 Roll src/third_party/skia b88894c8811b..66d8006c2bb1 (5 commits) (flutter#12178) 2592d6e [flutter_runner] Port the accessibility bridge from Topaz (flutter#12054) b569e8c Smooth out iOS irregular input events delivery (flutter#11817) dea813d Roll src/third_party/dart ccb6ba948b..62f78a7abb (3 commits) 2438798 Make ImageShader implement Shader for web ui (flutter#12161) c31583a Roll src/third_party/dart 2e8d912848..ccb6ba948b (30 commits) 4542886 Roll src/third_party/skia 9e5c47936b17..b88894c8811b (3 commits) (flutter#12151) 548998f Roll src/third_party/skia 1bf30ce852e0..9e5c47936b17 (2 commits) (flutter#12129) f1490a2 Roll src/third_party/skia 8cae1e95a23b..1bf30ce852e0 (2 commits) (flutter#12106) 19b2d43 Roll fuchsia/clang/mac-amd64 from H1Qjc... to HfPKR... (flutter#12088) d816755 Don't launch the observatory by default on each embedder unit-test invocation. (flutter#12087) 39c8067 Roll src/third_party/skia c2d84bfa7421..8cae1e95a23b (4 commits) (flutter#12086) b19e75a Roll src/third_party/dart fb14babf59..2e8d912848 (65 commits) 03e773a Guard availability of user notification related methods to iOS 10.0 (flutter#12084) 9c00c26 Add capability to add AppDelegate as UNUserNotificationCenterDelegate (flutter#9864) c2e8289 Add GradientRadial paintStyle implementation (flutter#12081) c3eea0a Don't quote generic font families (flutter#12080) c2b3d88 Roll src/third_party/skia 28d40b2e7ade..c2d84bfa7421 (3 commits) (flutter#12082) a4de006 Remove ENABLE_BITCODE from Scenarios test app (flutter#11839) 6e017f0 Roll src/third_party/skia 4f2674da4bbc..28d40b2e7ade (4 commits) (flutter#12077) e911b05 Roll src/third_party/skia 627d15588f4d..4f2674da4bbc (1 commits) (flutter#12075) 359e663 Roll src/third_party/skia 6c3bd09ead0f..627d15588f4d (3 commits) (flutter#12074) 38d545e Improve Unicode handling on Windows (flutter#11899) 89efb4c Roll fuchsia/clang/linux-amd64 from VoYNW... to 2IT_b... (flutter#12072) dc8e30d Roll fuchsia/clang/mac-amd64 from XAazI... to H1Qjc... (flutter#12071) 8cdb3af Annotate nullability on FlutterEngine to make swift writing more ergonomic (flutter#11808) aa9aaa2 Roll src/third_party/skia f433336585ed..6c3bd09ead0f (1 commits) (flutter#12070) 075a61f Roll src/third_party/skia 69a426f5a427..f433336585ed (1 commits) (flutter#12068) a610505 option for --no-lto for fuchsia (flutter#12010) be39820 Roll src/third_party/skia 380561393385..69a426f5a427 (2 commits) (flutter#12067) 6bafbf9 Roll src/third_party/skia c30f1a936d84..380561393385 (3 commits) (flutter#12059) e2ba93d Roll src/third_party/dart ec7ec4ecf7..fb14babf59 (19 commits) 723a288 Roll fuchsia/clang/linux-amd64 from -mnHl... to VoYNW... (flutter#12058) 35875e0 Revert "Manage resource and onscreen contexts using separate IOSGLContext objects (flutter#11798)" (flutter#12055) a353f93 Manage resource and onscreen contexts using separate IOSGLContext objects (flutter#11798) c9ea4db [flutter_runner] Refactor our build rules to make them more inline with topaz. (flutter#12034) 50bdbd7 Document dependencies and remove support-v13 (flutter#11912) 0c6a538 Roll src/third_party/dart be66176534..ec7ec4ecf7 (37 commits) 37f81cd Roll src/third_party/skia 080d210e7acc..c30f1a936d84 (21 commits) (flutter#12031)
…m snapshots (excl. blobs).' as it breaks flutter profile execution on ios 32-bit platform. Fixes flutter/flutter#40114 Change-Id: If8d71e9c19c2e794d29f7ecbacb87457890a2fd5 Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/116883 Reviewed-by: Alexander Markov <alexmarkov@google.com> Commit-Queue: Alexander Aprelev <aam@google.com>
This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of |
I reverted two engine rolls in #40112 because the were causing failures in devicelab tests that were failing with timeouts trying to connect to the observatory:
flutter_gallery_ios32__transition_perf
andflutter_gallery_ios32__start_up
.I HAVE STOPPED THE ENGINE ROLL, so someone needs to restart it once this is resolved.
Specifically, they started failing when af07bb5 was committed, which includes a Dart roll that is probably suspect:
Here is a sample log output of one of the tests (both are similar):
The text was updated successfully, but these errors were encountered: