iOS simulator unavailable, device type profile not found #97303

jmagman · 2022-01-26T17:39:47Z

Try job module_test_ios trying to create a simulator:

[module_test_ios] [STDOUT] Executing "xcrun simctl create TestAdd2AppSim com.apple.CoreSimulator.SimDeviceType.iPhone-11 com.apple.CoreSimulator.SimRuntime.iOS-14-3" in "/opt/s/w/ir/x/w/recipe_cleanup/tmp81ddsf_o/flutter sdk" with environment {BOT: true, LANG: en_US.UTF-8}
[module_test_ios] [STDOUT] stderr: Invalid device type: com.apple.CoreSimulator.SimDeviceType.iPhone-11

List of simulators shows all the simulators are "unavailable, device type profile not found"

-- iOS 13.0 --
    iPad Air (3rd generation) (C657E8D4-422B-4449-94E7-3B04431E2F18) (Shutdown) (unavailable, device type profile not found)
    iPad Pro (11-inch) (C3480239-0A68-47BF-97E9-20CFDA93F7D9) (Shutdown) (unavailable, device type profile not found)
    iPad Pro (12.9-inch) (3rd generation) (19F3AE1E-AFAB-447B-9AF6-7B8BB57D9897) (Shutdown) (unavailable, device type profile not found)
    iPad Pro (9.7-inch) (117D9CD0-82C8-458D-B3C4-8CA0331A03AD) (Shutdown) (unavailable, device type profile not found)
    iPhone 11 (6B11A7F3-E077-42F1-9911-D7C1B8D55CCE) (Shutdown) (unavailable, device type profile not found)
    iPhone 11 Pro (2E9AD3C2-AEE6-4152-A14F-882F3DFFDF91) (Shutdown) (unavailable, device type profile not found)
    iPhone 11 Pro Max (8C49B646-040E-4CB6-9FDC-10362CEB54B1) (Shutdown) (unavailable, device type profile not found)
    iPhone 8 (5DCC7E66-B73E-463A-A91D-F28E3B4E329B) (Shutdown) (unavailable, device type profile not found)
    iPhone 8 Plus (18E32AAD-47C2-424D-8A9E-7BFEEA4727A1) (Shutdown) (unavailable, device type profile not found)

https://logs.chromium.org/logs/flutter/buildbucket/cr-buildbucket/8823963857735512193/+/u/list_simulators/stdout

Compare to a successful prod run. Both are running 12c33.

-- iOS 13.0 --
    iPhone 8 (D6D03B46-EA7B-4A19-9115-6DB597FCF8B6) (Shutdown) 
    iPhone 8 Plus (D23E78A6-2A7F-410D-91FD-2D6779CC411B) (Shutdown) 
    iPhone 11 (3ACE52C8-0530-491A-9226-6A95068DFEF8) (Shutdown) 
    iPhone 11 Pro (6C3DB249-0409-4726-BABB-8BF3417E3E12) (Shutdown) 
    iPhone 11 Pro Max (98B27E12-DECA-41F3-AD22-530C3FE08CC9) (Shutdown) 
    iPad Pro (9.7-inch) (9A5CB7DA-3709-4970-BA8E-A359F60F2A2C) (Shutdown) 
    iPad Pro (11-inch) (EB1BBB3C-9519-4976-B394-D5E3F1C81455) (Shutdown) 
    iPad Pro (12.9-inch) (3rd generation) (420854E4-EE4A-4962-AE41-157DF43EA734) (Shutdown) 
    iPad Air (3rd generation) (4DEB6175-773A-45C8-A791-AF125666FDD9) (Shutdown)

https://logs.chromium.org/logs/flutter/buildbucket/cr-buildbucket/8823956734184440353/+/u/list_simulators/stdout

Blocking the plugin roll #97295, which is blocking an engine roll.

The text was updated successfully, but these errors were encountered:

jmagman · 2022-01-26T17:43:33Z

Re-running the test in #97295, I don't see it failing in other PRs (that PR is definitely not the issue though)

jmagman · 2022-01-26T17:44:32Z

Suspect the re-run will succeed, the simulator list looks correct:

-- iOS 13.0 --
    iPhone 8 (96198A19-44E4-4FAB-8516-DD50D00289BF) (Shutdown) 
    iPhone 8 Plus (91CF9D98-4DBC-4451-A823-A0A5B8A59EC6) (Shutdown) 
    iPhone 11 (D6B26EFF-F0AC-4071-A061-94336006AC33) (Shutdown) 
    iPhone 11 Pro (B03469C1-55BD-4B36-BD88-A082F905115B) (Shutdown) 
    iPhone 11 Pro Max (16B9D00E-E5F5-42E1-B5F7-C5B78458BFF0) (Shutdown) 
    iPad Pro (9.7-inch) (C19AAD9E-F393-4AEE-86B0-D3B99F6CAFD2) (Shutdown) 
    iPad Pro (11-inch) (B4B9BB52-BFE4-422D-8B5E-B06A759E3FEB) (Shutdown) 
    iPad Pro (12.9-inch) (3rd generation) (07FC72C6-605D-4A1D-ACB6-A1FF09BA7D09) (Shutdown) 
    iPad Air (3rd generation) (E552B27C-FDCD-4228-B33E-74C9A8207A54) (Shutdown)

https://logs.chromium.org/logs/flutter/buildbucket/cr-buildbucket/8823953796628412001/+/u/list_simulators/stdout

Reducing priority.

jmagman · 2022-01-26T17:48:14Z

I've seen that unavailable, device type profile not found before while investigating #91689 when simulator runtimes were missing, but the runtimes look correct this time:

== Runtimes ==
iOS 13.0 (13.0 - 17A577) - com.apple.CoreSimulator.SimRuntime.iOS-13-0
iOS 14.3 (14.3 - 18C61) - com.apple.CoreSimulator.SimRuntime.iOS-14-3
watchOS 6.1 (6.1.1 - 17S445) - com.apple.CoreSimulator.SimRuntime.watchOS-6-1
watchOS 7.2 (7.2 - 18S561) - com.apple.CoreSimulator.SimRuntime.watchOS-7-2

jmagman · 2022-01-26T17:49:58Z

And interestingly only the 14.3 iPhone 12* devices are available...?

== Device Types ==
iPhone 12 mini (com.apple.CoreSimulator.SimDeviceType.iPhone-12-mini)
iPhone 12 (com.apple.CoreSimulator.SimDeviceType.iPhone-12)
iPhone 12 Pro (com.apple.CoreSimulator.SimDeviceType.iPhone-12-Pro)
iPhone 12 Pro Max (com.apple.CoreSimulator.SimDeviceType.iPhone-12-Pro-Max)
...
-- iOS 14.3 --
    iPhone 12 mini (5B76E7EE-4109-4327-BD3C-08B088482192) (Shutdown) 
    iPhone 12 (5F6A55BC-C974-4D3B-AC0F-B92F55ED1AD6) (Shutdown) 
    iPhone 12 Pro (E7FEF58E-626E-4EFB-AE6A-5CB49FCE6321) (Shutdown) 
    iPhone 12 Pro Max (787AB908-C531-44CD-9530-524CCA545D60) (Shutdown)

https://logs.chromium.org/logs/flutter/buildbucket/cr-buildbucket/8823963857735512193/+/u/list_simulators/stdout

CaseyHillers · 2022-01-26T18:10:05Z

I'll monitor the builds to see if this flake occurs again

godofredoc · 2022-01-27T23:09:26Z

Waiting for the battery healthcheck to land in production.

jmagman · 2022-01-27T23:12:35Z

I don't think this is related to the tethered device's health, it's that the simulators aren't set up in a way Xcode can find them.

godofredoc · 2022-01-28T02:10:10Z

Would this be fixed with the new way to provide runtimes?

keyonghan · 2022-02-02T18:56:00Z

The error repeated intermittently:
https://ci.chromium.org/ui/p/flutter/builders/try/Mac%20module_test_ios/11849/overview (02/01/2022)
https://ci.chromium.org/ui/p/flutter/builders/try/Mac%20module_test_ios/11846/overview (02/01/2022)
https://ci.chromium.org/ui/p/flutter/builders/try/Mac%20module_test_ios/11804/overview (01/31/2022)
https://ci.chromium.org/ui/p/flutter/builders/try/Mac%20module_test_ios/11764/overview (01/28/2022)
https://ci.chromium.org/ui/p/flutter/builders/try/Mac%20module_test_ios/11718/overview (01/27/2022)

keyonghan · 2022-02-02T18:58:52Z

Seem the failures happened on only two bots: build857-m9 and build866-m9.

Take a further look, builder Mac module_test_ios has been consistently failing on these two bots.

/cc @jmagman Any settings/config that may cause this behavior from bot side? All bots should be configured in the same way though.

jmagman · 2022-02-02T21:12:46Z

Maybe the simulator device types were removed on those machines? I can't figure out where on disk the different types of simulator device images would live though.

godofredoc · 2022-02-02T21:42:23Z

Could this be related to testing the runtimes functionality of osx_sdk in those two bots?

jmagman · 2022-02-02T21:49:31Z

Not sure, from #97303 (comment) the symptoms look different, the simulator runtimes (OS version iOS-14-3) is present, but the actual device type images (iPhone-11) is missing.

I could change the test to iPhone 12 which does seem available, though it would be good to know why the builders are different.

godofredoc · 2022-02-02T23:54:57Z

@keyonghan can we delete the caches on those two machines and manually re-run the test?

jmagman · 2022-02-03T01:01:32Z

I can't figure out where on disk the different types of simulator device images would live though.

I think they are in Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Library/Developer/CoreSimulator/Profiles/DeviceTypes/?

keyonghan · 2022-02-03T05:52:51Z

@keyonghan can we delete the caches on those two machines and manually re-run the test?

Cleaning caches helps! Here is a success run with a pass on build857-m9.
https://luci-milo.appspot.com/raw/build/logs.chromium.org/flutter/led/keyonghan_google.com/b66b7c9f91b799685187926648693d492f39716dc480d0c09add5ab16ba6c5c1/+/build.proto

Will run a couple of times, and also on the other bot to verify.

keyonghan · 2022-02-03T17:26:06Z

Five times all passed on build857-m9.
Same pass on build866-m9 after caches clean.
Closing. Please reopen if issue happens again.

jmagman · 2022-02-10T04:47:20Z

This just happened again:
https://logs.chromium.org/logs/flutter/buildbucket/cr-buildbucket/8822650102210541713/+/u/list_simulators/stdout

Failing test:

plugin_lint_mac] [STDOUT] stdout:  -> integration_test (0.0.1)
[plugin_lint_mac] [STDOUT] stdout:     - ERROR | [iOS] unknown: Encountered an unknown error (Could not find a `ios` simulator (valid values: watchos). Ensure that Xcode -> Window -> Devices has at least one `ios` simulator listed or otherwise add one.

keyonghan · 2022-03-30T00:05:29Z

The above cl added only two more runtime caches without affecting existing osx_sdk cache:

keyonghan · 2022-03-30T00:11:30Z

The two failed builds happened on https://chromium-swarm.appspot.com/bot?id=build818-m9 and https://chromium-swarm.appspot.com/bot?id=build817-m9, both had an earlier failed run on Mac clang-tidy at step install xcode. Wondering if we need a cache refresh.

Trying an led run on a good bot to validate.

keyonghan · 2022-03-30T00:24:42Z

An led run on build819: https://luci-milo.appspot.com/raw/build/logs.chromium.org/flutter/led/keyonghan_google.com/728275e5db28c25ac54938b9e956ab0e40a80813c891077d4d8edf567cc8398b/+/build.proto

So far the device type profile is there. Expectation is the build will pass.

godofredoc · 2022-03-30T01:02:43Z

Can we try led runs on build818 and 817 and check the content of the xcode simruntime folder from ssh?

keyonghan · 2022-03-30T04:38:15Z

Latest update:
Led runs on two failed bots:
build 817, build 818

I was blocked on permission issue to ssh to these bots, and @godofredoc is helping validate some cases instead of simply refreshing the caches. From godofredo:

The problem was that the build was run with runtimes property and the old cache name

zanderso · 2022-03-30T15:02:57Z

What's the status of this? The Engine tree has been closed for 17 hours.

CaseyHillers · 2022-03-30T15:48:33Z

Out of curiosity, why is Mac Web Engine left on Mac 10? I kicked off a build in https://ci.chromium.org/ui/p/flutter/builders/try/Mac%20Web%20Engine/24081/overview to see if it would pass on Mac 12.

jmagman · 2022-03-30T16:44:22Z

Out of curiosity, why is Mac Web Engine left on Mac 10?

It's on macOS 10.15 and Xcode 12.3 because Safari won't launch on the Xcode 13 simulators #99885

CaseyHillers · 2022-03-30T16:59:33Z

Copying from internal chat (thanks @keyonghan for the recap):

@jmagman landed PR to push engine builders to xcode13+mac12 with Web Engine remaining in mac10
In post-submit, the ci.yaml roller broke (os dimension not populated and post-submit scheduler had a bug), those builders are still running on mac10 but with new xcode version and runtime (which is not compatible)
@CaseyHillers helped manual roll the ci yaml, and other engine builders are switched to mac12 and running through
However for Web Engine, it is still running on mac10, where earlier failed builds (new xcode and runtime) have polluted the cache.

The hesitation here is there's an issue with the recipes not correctly identifying older xcodes/simulators being used, and not wiped the directory.

AIs:

@CaseyHillers to move build825-m9 to staging for future investigation on the true root cause CL
Infra will wipe the mac caches on Mac 10 bots

CaseyHillers · 2022-03-30T17:12:40Z

I went through the engine build dashboard to find Mac 10 bots run that failed with xcode 13. I gathered the list and will start wiping them:

# Failed bot list
build812-m9
build813-m9
build814-m9
build815-m9
build816-m9
build817-m9
build818-m9
build820-m9
build821-m9
build822-m9
build823-m9
build824-m9
build826-m9
build827-m9

Update: I'm blocked from accessing m9 bots. Requesting permissions for the engprod team to have access (currently @godofredoc is the only one)

CaseyHillers · 2022-03-30T17:55:12Z

Talked with @yjbanov and it seems like Mac Web Engine is running on Mac 12 now. We landed flutter/engine#32337 as the issue is running this test on the corrupt Mac 10 bots.

jmagman · 2022-03-30T17:57:53Z

Talked with @yjbanov and it seems like Mac Web Engine is running on Mac 12 now. We landed flutter/engine#32337 as the issue is running this test on the corrupt Mac 10 bots.

This builder failed many times on Xcode 13 due to #99885 /cc @godofredoc it passed a few times for reasons we couldn't figure out.

keyonghan · 2022-03-30T19:14:21Z

The latest build is hanging. I guess we need a revert: flutter/engine#32340
Meanwhile, I am cleaning up caches.

keyonghan · 2022-03-30T19:18:13Z

Led run on a refreshed bot: https://luci-milo.appspot.com/raw/build/logs.chromium.org/flutter/led/keyonghan_google.com/340047dc7d61d76dffb91c87cff50b598859bc233a41f175c627a520692da442/+/build.proto

godofredoc · 2022-03-30T20:38:51Z

I have some local changes I used for validation yesterday unfortunately I haven't been able to test them yet. I'll upload the current version of the recipe change just in case someone has free cycles to validate it.

godofredoc · 2022-03-30T21:06:56Z

Here is a run with the changes to remove the runtimes properly: https://chromium-swarm.appspot.com/task?id=59f5766358a6e610

keyonghan · 2022-03-30T21:12:31Z

All bots are cleaned up, and we should be ready to land flutter/engine#32340 to unblock the tree.

CaseyHillers · 2022-03-31T14:02:48Z

After 15+ green runs, this looks fixed. Thanks Keyong!

github-actions · 2022-04-14T14:05:38Z

This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.

jmagman added team-infra Owned by Infrastructure team P0 Critical issues such as a build break or regression labels Jan 26, 2022

jmagman added this to New in Infra Ticket Queue via automation Jan 26, 2022

jmagman added P2 and removed P0 Critical issues such as a build break or regression labels Jan 26, 2022

CaseyHillers moved this from New to Triaged in Infra Ticket Queue Jan 26, 2022

CaseyHillers self-assigned this Jan 26, 2022

godofredoc added the passed secondary triage label Jan 27, 2022

keyonghan unassigned CaseyHillers Feb 1, 2022

keyonghan closed this as completed Feb 3, 2022

Infra Ticket Queue automation moved this from Triaged to Done Feb 3, 2022

jmagman moved this from Done to New in Infra Ticket Queue Mar 29, 2022

zanderso assigned keyonghan Mar 30, 2022

CaseyHillers moved this from New to In progress in Infra Ticket Queue Mar 30, 2022

CaseyHillers mentioned this issue Mar 30, 2022

[ci.yaml] Test web engine on Mac 12 flutter/engine#32337

Merged

keyonghan mentioned this issue Mar 30, 2022

Revert "[ci.yaml] Test web engine on Mac 12" flutter/engine#32340

Merged

This was referenced Mar 30, 2022

xcodebuild: error: unable execute utility "xcodebuild" because it requires a newer version of macOS. #100985

Closed

Add flake stress test mode to .ci.yaml #101084

Open

CaseyHillers closed this as completed Mar 31, 2022

Infra Ticket Queue automation moved this from In progress to Done Mar 31, 2022

github-actions bot locked as resolved and limited conversation to collaborators Apr 14, 2022

flutter-triage-bot bot added the c: flake Tests that sometimes, but not always, incorrectly pass label Jul 7, 2023

iOS simulator unavailable, device type profile not found #97303

iOS simulator unavailable, device type profile not found #97303

Comments

jmagman commented Jan 26, 2022

jmagman commented Jan 26, 2022

jmagman commented Jan 26, 2022

jmagman commented Jan 26, 2022

jmagman commented Jan 26, 2022

CaseyHillers commented Jan 26, 2022

godofredoc commented Jan 27, 2022

jmagman commented Jan 27, 2022

godofredoc commented Jan 28, 2022

keyonghan commented Feb 2, 2022

keyonghan commented Feb 2, 2022

jmagman commented Feb 2, 2022

godofredoc commented Feb 2, 2022

jmagman commented Feb 2, 2022

godofredoc commented Feb 2, 2022

jmagman commented Feb 3, 2022

keyonghan commented Feb 3, 2022

keyonghan commented Feb 3, 2022 • edited Loading

jmagman commented Feb 10, 2022

keyonghan commented Mar 30, 2022

keyonghan commented Mar 30, 2022

keyonghan commented Mar 30, 2022

godofredoc commented Mar 30, 2022

keyonghan commented Mar 30, 2022

zanderso commented Mar 30, 2022

CaseyHillers commented Mar 30, 2022

jmagman commented Mar 30, 2022

CaseyHillers commented Mar 30, 2022 • edited Loading

CaseyHillers commented Mar 30, 2022 • edited Loading

CaseyHillers commented Mar 30, 2022

jmagman commented Mar 30, 2022

keyonghan commented Mar 30, 2022

keyonghan commented Mar 30, 2022

godofredoc commented Mar 30, 2022

godofredoc commented Mar 30, 2022

keyonghan commented Mar 30, 2022

CaseyHillers commented Mar 31, 2022

github-actions bot commented Apr 14, 2022

keyonghan commented Feb 3, 2022 •

edited

Loading

CaseyHillers commented Mar 30, 2022 •

edited

Loading

CaseyHillers commented Mar 30, 2022 •

edited

Loading