2.2.0-10.1.pre breaks flutter/plugins CI

[stuartmorgan]

Since the release of 2.2 to stable, flutter/plugins CI has failing a number of iOS tests just on stable. An example failure:
https://github.com/flutter/plugins/runs/2632402452
The symptoms are generally crashes when trying to run xctest tests, although it seems like sometimes we get hangs instead.

We never saw this on master, even when master was at point where it would have branched off to 2.2. (We don't currently run tests on beta, but I tried enabling them for the current beta and it fails as well.)

This started on unrelated changes; the last passing run was with Flutter 2.0.6, and the first failing run was with 2.2 (we use channel names, not pinned versions, in the CI currently, so out-of-band breakage like this is a known possibility). After some bisecting I narrowed it down to

• 2.2.0-10.0.pre: Good
• 2.2.0-10.1.pre: Bad

Which points to: #80459

Locally I was able to reproduce and further narrow it down to the engine roll from
56b1355
to
d2a2e93510ad6cfc3d62a90d903b7056e4da8264
in that PR.

I'm able to consistently reproduce locally; my setup is Big Sur 11.3.1, Xcode 12.5.
The repro steps I was following since I was trying to stick close to flutter/plugins CI:

• Check out flutter/plugins (at head)
• xcrun simctl create Flutter-iPhone com.apple.CoreSimulator.SimDeviceType.iPhone-11 com.apple.CoreSimulator.SimRuntime.iOS-14-5 | xargs xcrun simctl boot
• ./script/tool_runner.sh build-examples --plugins local_auth --ipa
• ./script/tool_runner.sh xctest --plugins local_auth --ios-destination "platform=iOS Simulator,name=iPhone 11,OS=latest"

Other plugins are broken as well, but doing just local_auth is sufficient to repro.

/cc @christopherfujino @gaaclarke @jmagman

[gaaclarke]

Were you able to verify what process is actually hanging to cause the timeout? I see that XCTestCommand is waiting for the xcodebuild subprocess but when it is hanging is the xcodebuild subprocess still executing? Is plugin_tools still running? Can you link one of the crashes as well as the hangs?

[jmagman]

I was able to reproduce this from the

$ packages/image_picker/image_picker/example
$ flutter build ios --config-only
$ xcodebuild test -workspace ios/Runner.xcworkspace -configuration Debug -scheme Runner -destination platform="iOS Simulator,name=iPhone 11,OS=latest" CODE_SIGN_IDENTITY="" CODE_SIGNING_REQUIRED=NO GCC_TREAT_WARNINGS_AS_ERRORS=YES

which is similar to what the build-examples and xctest commands do.

For some reason I can't save the crash logs on Xcode 12.5 or 12.4, but managed it on 12.3:

codesign_crash.txt

UI Test Activity: 
Crash: Runner (59450) <external symbol>: Namespace CODESIGNING, Code 0x2. dyld: launch, loading dependent libraries
DYLD_SHARED_CACHE_DIR=/Users/magder/Library/Developer/CoreSimulator/Caches/dyld/20E241/com.apple.CoreSimulator.SimRuntime.iOS-14-5.18E5164d
DYLD_ROOT_PATH=/Users/magder/Applications/Xcode-12-5_beta3.app/Contents/Developer/Platforms/iPhoneOS.platform/Library/Developer/CoreSimulator/Profiles/Runtimes/iOS.simruntime/Contents/Resources/RuntimeRoot
DYLD_LIBRARY_PATH=/Users/magder/Library/Developer/Xcode/DerivedData/Runner-fynydyyyghdhcuavkiausxbvctjd/Build/Products/Debug-iphonesimulator
DYLD_INSERT_LIBRARIES=/Users/magder/Applications/Xcode-12-5_beta3.app/Contents/Developer/Platforms/iPhoneOS.platform/Library/Developer/CoreSimulator/Profiles/Runtimes/iOS.simruntime/Contents/Resources/RuntimeRoot/usr/lib/libMainThreadChecker.dylib
DYLD_FALLBACK_FRAMEWORK_PATH=/Users/magder/Applications/Xcode-12-5_beta3.app/Contents/Developer/Platforms/iPhoneOS.platform/Library/Developer/CoreSimulator/Profiles/Runtimes/iOS.simruntime/Contents/Resources/RuntimeRoot/System/Library/Frameworks
 
@rpath/Flutter.framework/Flutter

[jmagman]

Were you able to verify what process is actually hanging to cause the timeout? I see that XCTestCommand is waiting for the xcodebuild subprocess but when it is hanging is the xcodebuild subprocess still executing? Is plugin_tools still running? Can you link one of the crashes as well as the hangs?

I'm not sure why 2 of the 3 test shards time out, when I reproduce locally it crashes and finishes quickly. Here's the one shard that's crashing instead of hanging: https://cirrus-ci.com/task/5379387391475712

[gaaclarke]

I think you already know but it isn't stated, this appears to be the result of loading a dynamic library that isn't signed correctly.

Exception Type:        EXC_BAD_ACCESS (Code Signature Invalid)
Exception Codes:       0x0000000000000032, 0x000000010e4d1000
Exception Note:        EXC_CORPSE_NOTIFY

Termination Reason:    Namespace CODESIGNING, Code 0x2

kernel messages:

VM Regions Near 0x10e4d1000:
    __LINKEDIT                  10e4c9000-10e4d1000    [   32K] r--/rwx SM=COW  /Users/*/Xcode-12-3.app/Contents/Developer/Platforms/iPhoneOS.platform/Library/Developer/CoreSimulator/Profiles/Runtimes/iOS.simruntime/Contents/Resources/RuntimeRoot/usr/lib/libMainThreadChecker.dylib
--> mapped file                 10e4d1000-10e4d3000    [    8K] r--/r-x SM=PRV  Object_id=1711d16f
    __TEXT                      11e0af000-11e14b000    [  624K] r-x/r-x SM=COW  /usr/lib/dyld

Application Specific Information:
dyld: launch, loading dependent libraries
DYLD_SHARED_CACHE_DIR=/Users/magder/Library/Developer/CoreSimulator/Caches/dyld/20E241/com.apple.CoreSimulator.SimRuntime.iOS-14-3.18C61
DYLD_ROOT_PATH=/Users/magder/Applications/Xcode-12-3.app/Contents/Developer/Platforms/iPhoneOS.platform/Library/Developer/CoreSimulator/Profiles/Runtimes/iOS.simruntime/Contents/Resources/RuntimeRoot
DYLD_LIBRARY_PATH=/Users/magder/Library/Developer/Xcode/DerivedData/Runner-fynydyyyghdhcuavkiausxbvctjd/Build/Products/Debug-iphonesimulator
DYLD_INSERT_LIBRARIES=/Users/magder/Applications/Xcode-12-3.app/Contents/Developer/Platforms/iPhoneOS.platform/Library/Developer/CoreSimulator/Profiles/Runtimes/iOS.simruntime/Contents/Resources/RuntimeRoot/usr/lib/libMainThreadChecker.dylib
DYLD_FALLBACK_FRAMEWORK_PATH=/Users/magder/Applications/Xcode-12-3.app/Contents/Developer/Platforms/iPhoneOS.platform/Library/Developer/CoreSimulator/Profiles/Runtimes/iOS.simruntime/Contents/Resources/RuntimeRoot/System/Library/Frameworks
DYLD_FALLBACK_LIBRARY
@rpath/Flutter.framework/Flutter

Thread 0 Crashed:
0   dyld_sim                      	0x000000010e31923c memcmp + 11
1   dyld_sim                      	0x000000010e304b9b ImageLoaderMachO::validateFirstPages(linkedit_data_command const*, int, unsigned char const*, unsigned long, long long, ImageLoader::LinkContext const&) + 71
2   dyld_sim                      	0x000000010e308456 ImageLoaderMachOCompressed::instantiateFromFile(char const*, int, unsigned char const*, unsigned long, unsigned long long, unsigned long long, stat const&, unsigned int, unsigned int, linkedit_data_command const*, encryption_info_command const*, ImageLoader::LinkContext const&) + 310
3   dyld_sim                      	0x000000010e303ced ImageLoaderMachO::instantiateFromFile(char const*, int, unsigned char const*, unsigned long, unsigned long long, unsigned long long, stat const&, ImageLoader::LinkContext const&) + 143
4   dyld_sim                      	0x000000010e2f46be dyld::loadPhase6(int, stat const&, char const*, dyld::LoadContext const&) + 793
5   dyld_sim                      	0x000000010e2f9653 dyld::loadPhase5(char const*, char const*, dyld::LoadContext const&, unsigned int&, std::__1::vector >*) + 1693
6   dyld_sim                      	0x000000010e2f8f54 dyld::loadPhase4(char const*, char const*, dyld::LoadContext const&, unsigned int&, std::__1::vector >*) + 185
7   dyld_sim                      	0x000000010e2f85c6 dyld::loadPhase2(char const*, char const*, dyld::LoadContext const&, char const* const*, char const* const*, unsigned int&, std::__1::vector >*) + 209
8   dyld_sim                      	0x000000010e2f842c dyld::loadPhase1(char const*, char const*, dyld::LoadContext const&, unsigned int&, std::__1::vector >*) + 160
9   dyld_sim                      	0x000000010e2f4380 dyld::loadPhase0(char const*, char const*, dyld::LoadContext const&, unsigned int&, std::__1::vector >*) + 226
10  dyld_sim                      	0x000000010e2f4036 dyld::load(char const*, dyld::LoadContext const&, unsigned int&) + 185
11  dyld_sim                      	0x000000010e2f9b88 dyld::libraryLocator(char const*, bool, char const*, ImageLoader::RPathChain const*, unsigned int&) + 55
12  dyld_sim                      	0x000000010e30014c ImageLoader::recursiveLoadLibraries(ImageLoader::LinkContext const&, bool, ImageLoader::RPathChain const&, char const*) + 346
13  dyld_sim                      	0x000000010e2ff158 ImageLoader::link(ImageLoader::LinkContext const&, bool, bool, bool, ImageLoader::RPathChain const&, char const*) + 90
14  dyld_sim                      	0x000000010e2f6671 dyld::link(ImageLoader*, bool, bool, ImageLoader::RPathChain const&, unsigned int) + 383
15  dyld_sim                      	0x000000010e2f7adc dyld::_main(macho_header const*, unsigned long, int, char const**, char const**, char const**, unsigned long*) + 3803
16  dyld_sim                      	0x000000010e2f21c7 start_sim + 122
17  dyld                          	0x000000011e0b8a8e dyld::useSimulatorDyld(int, macho_header const*, char const*, int, char const**, char const**, char const**, unsigned long*, unsigned long*) + 2093
18  dyld                          	0x000000011e0b6168 dyld::_main(macho_header const*, unsigned long, int, char const**, char const**, char const**, unsigned long*) + 1198
19  dyld                          	0x000000011e0b0224 dyldbootstrap::start(dyld3::MachOLoaded const*, int, char const**, dyld3::MachOLoaded const*, unsigned long*) + 450
20  dyld                          	0x000000011e0b0025 _dyld_start + 37

[stuartmorgan]

Downgrading to P3 since we've worked around this in CI by changing the script.

[christopherfujino]

Downgrading to P3 since we've worked around this in CI by changing the script.

Is this not affecting end users?

[gaaclarke]

Is this not affecting end users?

I don't think so. It seems like specifying those environment variables when running the tests was erroneous. We don't have an explanation why they were there and we have invocations elsewhere in our code that doesn't specify them. It's unfortunate that we didn't get a consistent behavior though.

In order for this to affect users they'd have to be running xctests with the same environment variables (which I've never seen anyone specify before).

edit: By "erroneous" I meant non-idiomatic for running tests on the simulator. I agree with @jmagman that they shouldn't create problems.

[jmagman]

It seems like specifying those environment variables when running the tests was erroneous.

I don't think so, it's that it's manifesting when codesigning, and removing those variables skips codesigning. When I set set up a development team needed to sign and run the tests from Xcode on a real device, I see the same crash.

Also the image_picker base configuration for the test target set up in flutter/plugins#3835 isn't right, and the Podfile isn't set up to handle that target (I'll put out a PR for that). The other packages reproduce with the same crash though.

Bootstrap Failure_0_AF2CD977-D7CE-4EDF-922A-5F45B0CB51CD.txt

[jmagman]

When I set set up a development team needed to sign and run the tests from Xcode on a real device, I see the same crash.

Actually that only seems to be true on the bad image_picker test. local_auth doesn't have a UI test target, and passes fine on a real device with code signing set up. quick_actions does have a UI test target and also passes fine on a real device with code signing set up.

[jmagman]

Also I may have ignored this on the hotfix this was bisected to:

#80459 (comment)

module_test_ios is failing with an add to app host runtime codesigning crash:
codesign_crash.txt

Exception Type:        EXC_BAD_ACCESS (Code Signature Invalid)
Exception Codes:       0x0000000000000032, 0x0000000107614000
Exception Note:        EXC_CORPSE_NOTIFY

Termination Reason:    Namespace CODESIGNING, Code 0x2

The top of the log says This bot does not support codesigning
https://logs.chromium.org/logs/flutter/buildbucket/cr-buildbucket.appspot.com/8849863879990991456/+/steps/run_module_test_ios/0/stdout

[gaaclarke]

I don't think so, it's that it's manifesting when codesigning, and removing those variables skips codesigning. When I set set up a development team needed to sign and run the tests from Xcode on a real device, I see the same crash.

That makes sense you'd need to codesign for a real device, but since CI is always running on the simulator it shouldn't be signing right? Is this tool intended to be used outside of CI?

[jmagman]

That makes sense you'd need to codesign for a real device, but since CI is always running on the simulator it shouldn't be signing right? Is this tool intended to be used outside of CI?

@stuartmorgan removed the settings in flutter/plugins#3952, though they aren't incorrect as far as I can tell, just unnecessary. The crash doesn't reproduce on master, and it doesn't reproduce on the hotfix parent 0941968^, but it does reproduce on the 0941968 hotfix.

[stuartmorgan]

Closing as obsolete, since this was never an issue on master.

[github-actions]

This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.