Mac tool_integration_tests_4_4 flaky on multiple unit tests #100245

keyonghan · 2022-03-16T18:43:52Z

(part of #97036)

Since 03/14/2022, Mac tool_integration_tests_4_4 has been flaky 4 times, each hit same multiple unit tests errors:

Flaky builds:
https://ci.chromium.org/ui/p/flutter/builders/prod/Mac%20tool_integration_tests_4_4/3475/overview
https://ci.chromium.org/ui/p/flutter/builders/prod/Mac%20tool_integration_tests_4_4/3484/overview
https://ci.chromium.org/ui/p/flutter/builders/prod/Mac%20tool_integration_tests_4_4/3486/overview
https://ci.chromium.org/ui/p/flutter/builders/prod/Mac%20tool_integration_tests_4_4/3487/overview

01:19 +0 -1: test/integration.shard/ios_content_validation_test.dart: iOS app validation builds all plugin architectures for simulator [E]                                                             
  Expected: <0>
    Actual: <1>
  
  package:test_api                                               expect
  test/integration.shard/ios_content_validation_test.dart 246:7  main.<fn>.<fn>
  dart:async                                                     runZoned
  test/src/common.dart 197:14                                    testWithoutContext.<fn>
  test/src/common.dart 196:18                                    testWithoutContext.<fn>
  test/src/common.dart 166:18                                    test.<fn>
  test/src/common.dart 161:5                                     test.<fn>

01:35 +0 -2: test/integration.shard/ios_content_validation_test.dart: iOS app validation build for simulator with all available architectures [E]                                                      
  Expected: <0>
    Actual: <1>
  
  package:test_api                                               expect
  test/integration.shard/ios_content_validation_test.dart 283:7  main.<fn>.<fn>
  dart:async                                                     runZoned
  test/src/common.dart 197:14                                    testWithoutContext.<fn>
  test/src/common.dart 196:18                                    testWithoutContext.<fn>
  test/src/common.dart 166:18                                    test.<fn>
  test/src/common.dart 161:5                                     test.<fn>

02:28 +0 -3: test/integration.shard/ios_content_validation_test.dart: iOS app validation build in debug mode validate obfuscation [E]                                                                  
  Expected: <1>
    Actual: <2>
  
  package:test_api                                                expect
  test/integration.shard/ios_content_validation_test.dart 228:11  main.<fn>.<fn>.<fn>
  dart:async                                                      runZoned
  test/src/common.dart 197:14                                     testWithoutContext.<fn>
  test/src/common.dart 196:18                                     testWithoutContext.<fn>
  test/src/common.dart 166:18                                     test.<fn>
  test/src/common.dart 161:5                                      test.<fn>

02:28 +0 ~1 -4: test/integration.shard/ios_content_validation_test.dart: iOS app validation build in debug mode flutter build ios builds a valid app [E]                                               
  Expected: a file system entity that exists
    Actual: LocalFile:<LocalFile: '/opt/s/w/ir/x/t/flutter_ios_content_validation.ocq7dS/hello/example/build/ios/iphoneos/Runner.app/Frameworks/hello.framework/hello'>
     Which: does not exist
  
  package:test_api                                               expect
  test/integration.shard/ios_content_validation_test.dart 97:11  main.<fn>.<fn>.<fn>
  dart:async                                                     runZoned
  test/src/common.dart 197:14                                    testWithoutContext.<fn>
  test/src/common.dart 196:18                                    testWithoutContext.<fn>
  test/src/common.dart 166:18                                    test.<fn>
  test/src/common.dart 161:5                                     test.<fn>
  

02:28 +0 ~1 -4: test/integration.shard/ios_content_validation_test.dart: iOS app validation build in debug mode Info.plist dart observatory Bonjour service                                            
02:28 +0 ~1 -5: test/integration.shard/ios_content_validation_test.dart: iOS app validation build in debug mode Info.plist dart observatory Bonjour service [E]                                        
  Expected: <true>
    Actual: <false>
  
  package:test_api                                                expect
  test/integration.shard/ios_content_validation_test.dart 133:11  main.<fn>.<fn>.<fn>
  dart:async                                                      runZoned
  test/src/common.dart 197:14                                     testWithoutContext.<fn>
  test/src/common.dart 196:18                                     testWithoutContext.<fn>
  test/src/common.dart 166:18                                     test.<fn>
  test/src/common.dart 161:5                                      test.<fn>

03:35 +1 ~1 -6: test/integration.shard/ios_content_validation_test.dart: iOS app validation build in release mode validate obfuscation [E]                                                             
  Expected: <1>
    Actual: <2>
  
  package:test_api                                                expect
  test/integration.shard/ios_content_validation_test.dart 228:11  main.<fn>.<fn>.<fn>
  dart:async                                                      runZoned
  test/src/common.dart 197:14                                     testWithoutContext.<fn>
  test/src/common.dart 196:18                                     testWithoutContext.<fn>
  test/src/common.dart 166:18                                     test.<fn>
  test/src/common.dart 161:5                                      test.<fn>
  

03:35 +1 ~1 -6: test/integration.shard/ios_content_validation_test.dart: iOS app validation build in release mode xcode_backend embed_and_thin                                                         
03:35 +1 ~1 -7: test/integration.shard/ios_content_validation_test.dart: iOS app validation build in release mode xcode_backend embed_and_thin [E]                                                     
  FileSystemException: Deletion failed, path = '/opt/s/w/ir/x/t/flutter_ios_content_validation.ocq7dS/hello/example/build/ios/iphoneos/Runner.app/Frameworks/Flutter.framework' (OS Error: No such file or directory, errno = 2)
  dart:io                                                               FileSystemEntity.deleteSync
  package:file/src/forwarding/forwarding_file_system_entity.dart 70:16  ForwardingFileSystemEntity.deleteSync
  test/integration.shard/ios_content_validation_test.dart 165:34        main.<fn>.<fn>.<fn>
  dart:async                                                            runZoned
  test/src/common.dart 197:14                                           testWithoutContext.<fn>
  test/src/common.dart 196:18                                           testWithoutContext.<fn>
  test/src/common.dart 166:18                                           test.<fn>
  test/src/common.dart 161:5                                            test.<fn>
  

03:35 +1 ~1 -7: test/integration.shard/ios_content_validation_test.dart: iOS app validation build in release mode flutter build ios builds a valid app                                                 
03:35 +1 ~1 -8: test/integration.shard/ios_content_validation_test.dart: iOS app validation build in release mode flutter build ios builds a valid app [E]                                             
  Expected: a file system entity that exists
    Actual: LocalFile:<LocalFile: '/opt/s/w/ir/x/t/flutter_ios_content_validation.ocq7dS/hello/example/build/ios/iphoneos/Runner.app/Frameworks/hello.framework/hello'>
     Which: does not exist
  
  package:test_api                                               expect
  test/integration.shard/ios_content_validation_test.dart 97:11  main.<fn>.<fn>.<fn>
  dart:async                                                     runZoned
  test/src/common.dart 197:14                                    testWithoutContext.<fn>
  test/src/common.dart 196:18                                    testWithoutContext.<fn>
  test/src/common.dart 166:18                                    test.<fn>
  test/src/common.dart 161:5                                     test.<fn>
  

03:35 +1 ~1 -8: test/integration.shard/ios_content_validation_test.dart: iOS app validation build in release mode Info.plist dart observatory Bonjour service                                          
03:35 +2 ~1 -8: test/integration.shard/ios_content_validation_test.dart: iOS app validation build in release mode Info.plist dart observatory Bonjour service                                          
03:35 +2 ~1 -8: test/integration.shard/ios_content_validation_test.dart: iOS app validation build in release mode check symbols                                                                        
03:35 +2 ~1 -9: test/integration.shard/ios_content_validation_test.dart: iOS app validation build in release mode check symbols [E]                                                                    
  Expected: <true>
    Actual: <false>
  
  package:test_api                                                expect
  test/integration.shard/ios_content_validation_test.dart 161:11  main.<fn>.<fn>.<fn>
  dart:async                                                      runZoned
  test/src/common.dart 197:14                                     testWithoutContext.<fn>
  test/src/common.dart 196:18                                     testWithoutContext.<fn>
  test/src/common.dart 166:18                                     test.<fn>
  test/src/common.dart 161:5                                      test.<fn>

18:32 +46 ~1 -10: test/integration.shard/analyze_size_test.dart: --analyze-size flag produces expected output on hello_world for iOS [E]                                                               
  Expected: contains 'Dart AOT symbols accounted decompressed size'
    Actual: 'Warning: Building for device with codesigning disabled. You will have to manually codesign before deploying to device.\n'
              'Building io.flutter.examples.hello-world for device (ios-release)...\n'
              'Warning: Missing build name (CFBundleShortVersionString).\n'
              'Warning: Missing build number (CFBundleVersion).\n'
              'Running Xcode build...                                          \n'
              'Xcode build done.                                           25.2s\n'
              'Failed to build iOS app\n'
              'Error output from Xcode build:\n'
              '↳\n'
              '    2022-03-15 21:51:43.660 xcodebuild[12578:73409] [MT] DVTSDK: Skipped SDK /opt/s/w/ir/cache/osx_sdk/XCode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk; its version (10.15.6) is below required minimum (11.0) for the macosx platform.\n'
              '    2022-03-15 21:51:43.662 xcodebuild[12578:73409] [MT] DVTSDK: Skipped SDK /opt/s/w/ir/cache/osx_sdk/XCode.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator13.6.sdk; its version (13.6) is below required minimum (15.0) for the iphonesimulator platform.\n'
              '    2022-03-15 21:51:43.662 xcodebuild[12578:73409] [MT] DVTSDK: Skipped SDK /opt/s/w/ir/cache/osx_sdk/XCode.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator14.0.sdk; its version (14.0) is below required minimum (15.0) for the iphonesimulator platform.\n'
              '    2022-03-15 21:51:43.663 xcodebuild[12578:73409] [MT] DVTSDK: Skipped SDK /opt/s/w/ir/cache/osx_sdk/XCode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS13.6.sdk; its version (13.6) is below required minimum (15.0) for the iphoneos platform.\n'
              '    2022-03-15 21:51:43.663 xcodebuild[12578:73409] [MT] DVTSDK: Skipped SDK /opt/s/w/ir/cache/osx_sdk/XCode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS14.0.sdk; its version (14.0) is below required minimum (15.0) for the iphoneos platform.\n'
              '    ** BUILD FAILED **\n'
              '\n'
              '\n'
              'Xcode\'s output:\n'
              '↳\n'
              '    Writing result bundle at path:\n'
              '    \t/opt/s/w/ir/x/t/flutter_tools.6CBhPi/flutter_ios_build_temp_diru69dKJ/temporary_xcresult_bundle\n'
              '\n'
              '    2022-03-15 21:52:08.614 ibtoold[12136:70583] [MT] IBPlatformTool: *** Failed to launch tool with description <IBCocoaTouchPlatformToolDescription: 0x600002532ce0> System content for IBCocoaTouchFramework-EightAndLater <IBScaleFactorDeviceTypeDescription: 0x600002533fc0> scaleFactor=2x, renderMode.identifier=(null): Failed to find or create execution context for description \'<IBCocoaTouchPlatformToolDescription: 0x600002532ce0> System content for IBCocoaTouchFramework-EightAndLater <IBScaleFactorDeviceTypeDescription: 0x600002533fc0> scaleFactor=2x, renderMode.identifier=(null)\'.\n'
              '    Device type: IBSimDeviceTypeiPad2x (com.apple.dt.Xcode.IBSimDeviceType.iPad-2x)\n'
              '    Sim runtime: (null)\n'
              '    ** Please also include the output of `xcrun simctl list` and `xcode-select -p`.: Failed to find newest available Simulator runtime: Could not find any supported simulator runtimes\n'
              '    /* com.apple.ibtool.errors */\n'
              '    /opt/s/w/ir/x/w/flutter/examples/hello_world/ios/Runner/Base.lproj/LaunchScreen.storyboard: error: Failed to find or create execution context for description \'<IBCocoaTouchPlatformToolDescription: 0x600002532ce0> System content for IBCocoaTouchFramework-EightAndLater <IBScaleFactorDeviceTypeDescription: 0x600002533fc0> scaleFactor=2x, renderMode.identifier=(null)\'.\n'
              '    Device type: IBSimDeviceTypeiPad2x (com.apple.dt.Xcode.IBSimDeviceType.iPad-2x)\n'
              '    Sim runtime: (null)\n'
              '    ** Please also include the output of `xcrun simctl list` and `xcode-select -p`.\n'
              '        Underlying Errors:\n'
              '            Description: Failed to find newest available Simulator runtime\n'
              '            Underlying Errors:\n'
              '                Description: Could not find any supported simulator runtimes\n'
              '    note: Using new build system\n'
              '    note: Planning\n'
              '    note: Build preparation complete\n'
              '    note: Building targets in parallel\n'
              '\n'
              '    Result bundle written to path:\n'
              '    \t/opt/s/w/ir/x/t/flutter_tools.6CBhPi/flutter_ios_build_temp_diru69dKJ/temporary_xcresult_bundle\n'
              '\n'
              '\n'
              ''
  
  package:test_api                                    expect
  test/integration.shard/analyze_size_test.dart 72:5  main.<fn>

The text was updated successfully, but these errors were encountered:

zanderso · 2022-03-16T21:45:19Z

Routing to @jmagman to triage.

godofredoc · 2022-03-17T22:05:30Z

Removing infra for now, please add it back if this is caused by infrastructure.

jmagman · 2022-03-22T23:12:37Z

These are all issues with the iOS simulator runtimes missing.

== Runtimes ==
== Devices ==

jmagman · 2022-03-22T23:15:00Z

Is this cleared up as of #99429 ?

keyonghan · 2022-03-22T23:20:25Z

Closing as this doesn't repro after runtime issue fix.

jmagman · 2022-03-25T00:17:42Z

Seen again in https://ci.chromium.org/p/flutter/builders/try/Mac%20build_tests_4_4/17955?

keyonghan · 2022-03-25T01:32:52Z

The failed test ran on bot: https://chromium-swarm.appspot.com/bot?id=build842-m9, and tests started failing after a release run: https://luci-milo.appspot.com/ui/p/flutter/builders/try/Mac%20module_test/14975/overview. It used refs/heads/flutter-2.8-candidate.16, "sdk_version": "12c33", and Mac-10.

/cc @godofredoc @CaseyHillers could you help cherry pick https://flutter-review.googlesource.com/c/recipes/+/28064 from recipes and #98297 from flutter?

CaseyHillers · 2022-03-25T01:45:38Z

#98297 will need a dedicated CP request - https://github.com/flutter/flutter/wiki/Flutter-Cherrypick-Process

I can CP the recipes change in

keyonghan · 2022-03-25T17:05:15Z

#98297 will need a dedicated CP request - https://github.com/flutter/flutter/wiki/Flutter-Cherrypick-Process

I can CP the recipes change in

Added CP:review label. Please let me know if anything else is needed here.

keyonghan · 2022-03-25T17:06:38Z

Lowering the priority to P3.

CaseyHillers · 2022-03-28T18:28:53Z

Synced with @keyonghan over chat. The reason this is in the ticket queue as the current infra state allows the tests to corrupt the bot, which will cause it to fail until it is quarantined and wiped. CP'ing the ToT fixes into the release branch would prevent it getting into a bad state.

@keyonghan since this requires doing a major rev upgrade to xcode, it's unlikely for this to be approved as a CP. Is there a way we can update device doctor to check for this state, or catch the failure in the recipes?

keyonghan · 2022-03-28T18:51:04Z

This happens on host only bots, where we are not running device doctor.
One thing we can do is fail the test earlier as infra failure from recipe side when no runtime in list simulators step.

CaseyHillers · 2022-03-28T19:21:12Z

Are there any blockers to running device doctor on the host only bots?

keyonghan · 2022-03-28T21:37:54Z

This needs device doctor package installed, but these bots are maintained by chromium team. Also, we need to separate the bot config file from fuchsia.py. Another blocker: this needs xcode preinstalled.

Instead, could you help check why the release build failure caused runtime not available? The issue in #99429 is due to xcode dependency not consistent before tot and release builders. is it same here?

keyonghan · 2022-04-02T00:21:34Z

This is also on bot 842-m9. Duplicate of #99039

github-actions · 2022-04-16T01:28:02Z

This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.

keyonghan assigned zanderso Mar 16, 2022

keyonghan added team: flakes c: flake Tests that sometimes, but not always, incorrectly pass labels Mar 16, 2022

keyonghan added this to To Do in Shard Flake via automation Mar 16, 2022

keyonghan mentioned this issue Mar 16, 2022

Host only tests with shards - flakes #97036

Open

70 tasks

zanderso changed the title ~~Linux tool_integration_tests_4_4 flaky on multiple unit tests~~ Mac tool_integration_tests_4_4 flaky on multiple unit tests Mar 16, 2022

zanderso assigned jmagman and unassigned zanderso Mar 16, 2022

darshankawar added the team-infra Owned by Infrastructure team label Mar 17, 2022

godofredoc added passed secondary triage and removed team-infra Owned by Infrastructure team labels Mar 17, 2022

jmagman added the tool Affects the "flutter" command-line tool. See also t: labels. label Mar 22, 2022

jmagman added team-infra Owned by Infrastructure team and removed tool Affects the "flutter" command-line tool. See also t: labels. labels Mar 22, 2022

jmagman removed their assignment Mar 22, 2022

jmagman added this to New in Infra Ticket Queue via automation Mar 22, 2022

keyonghan closed this as completed Mar 22, 2022

Infra Ticket Queue automation moved this from New to Done Mar 22, 2022

Shard Flake automation moved this from To Do to Done Mar 22, 2022

jmagman reopened this Mar 25, 2022

Infra Ticket Queue automation moved this from Done to In progress Mar 25, 2022

keyonghan added the cp: review Cherry-picks in the review queue label Mar 25, 2022

keyonghan added P1 High-priority issues at the top of the work list and removed P1 labels Mar 25, 2022

CaseyHillers assigned keyonghan Mar 28, 2022

jmagman mentioned this issue Mar 29, 2022

[flutter_tool] Have long-running validators fail #100936

Merged

keyonghan closed this as completed Apr 2, 2022

Infra Ticket Queue automation moved this from In progress to Done Apr 2, 2022

github-actions bot locked as resolved and limited conversation to collaborators Apr 16, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Mac tool_integration_tests_4_4 flaky on multiple unit tests #100245

Mac tool_integration_tests_4_4 flaky on multiple unit tests #100245

keyonghan commented Mar 16, 2022 •

edited by zanderso

Loading

zanderso commented Mar 16, 2022

godofredoc commented Mar 17, 2022

jmagman commented Mar 22, 2022

jmagman commented Mar 22, 2022

keyonghan commented Mar 22, 2022

jmagman commented Mar 25, 2022

keyonghan commented Mar 25, 2022

CaseyHillers commented Mar 25, 2022

keyonghan commented Mar 25, 2022

keyonghan commented Mar 25, 2022

CaseyHillers commented Mar 28, 2022

keyonghan commented Mar 28, 2022

CaseyHillers commented Mar 28, 2022

keyonghan commented Mar 28, 2022

keyonghan commented Apr 2, 2022

github-actions bot commented Apr 16, 2022

Mac tool_integration_tests_4_4 flaky on multiple unit tests #100245

Mac tool_integration_tests_4_4 flaky on multiple unit tests #100245

Comments

keyonghan commented Mar 16, 2022 • edited by zanderso Loading

zanderso commented Mar 16, 2022

godofredoc commented Mar 17, 2022

jmagman commented Mar 22, 2022

jmagman commented Mar 22, 2022

keyonghan commented Mar 22, 2022

jmagman commented Mar 25, 2022

keyonghan commented Mar 25, 2022

CaseyHillers commented Mar 25, 2022

keyonghan commented Mar 25, 2022

keyonghan commented Mar 25, 2022

CaseyHillers commented Mar 28, 2022

keyonghan commented Mar 28, 2022

CaseyHillers commented Mar 28, 2022

keyonghan commented Mar 28, 2022

keyonghan commented Apr 2, 2022

github-actions bot commented Apr 16, 2022

keyonghan commented Mar 16, 2022 •

edited by zanderso

Loading