This issue was moved to a discussion.
You can continue the conversation there. Go to discussion →
XCode simulators failing when booting or using openurl #7971
Comments
|
Hello @fpkamp . Thanks for your report. MacOS 13 in beta state for now so it can have some issues like this. We will investigate problem and return with feedback as we get any result. |
|
@fpkamp , we are preparing new macos-13 image with xcode update. we are not certain on time framing, probably 2 weeks or so. can you provide repro steps so we can test your scenario on updated image ? |
|
@fpkamp , we deployed new image today (macos-13.5 + xcode-15.0 Beta2), please try |
|
Crossposting my message here as it might relate: #8023 (comment)
I think this new deploy might be to blame as there were no issues on Monday. This is a considerable amount of added time. EDIT: This is for a private repo (with extensive tooling set up). Repro steps are to create a Package with multiple test targets and include the same empty test in each of them. Then set up CI to run all tests and it should take well over an hour to test 0 code. This could be an Apple issue (not at all ruling that out). SOLUTION: Set the simulator to |
|
@joshuapoq , can you help with repro steps (or link to repo/build) ? |
|
@ilia-shipitsin sorry, I missed your ping from last week. The repro steps are in the original ticket - to clarify, the part I pasted in the repro steps is a bash script you should execute in your workflow. You may need to install the simulators before, |
|
hmm, is
|
|
Sorry, I meant just using a normal run action, feel free to just use this 14.3.1 because it worked better, but obviously you can also use 15.0, I am currently running a build on the 13.5 image so I'll update soon as I get the results. |
|
starting with yesterday we included simulators for XCode-15 as well, i.e. |
|
Just to confirm - macos-13.5 is the expected input? I see that my job has been stuck in waiting for queue for the past 4 hours, but running macos13-xl was immediate (just to rule out something on my org end I haven't been notified about) |
sorry, what do you mean by "input" ? |
is |
no.... and yes )) labels supported: macos-11 if you specify "macos-13.5", GHA will try to find your attached self hosted runner with that label. |
Alright, I see, so can you please ELI5 how to use the aforementioned image? |
|
aforementioned image is used automatically if you specify |
|
@ilia-shipitsin okay so I finally ran it on the new version and looks pretty much the same in terms of stability. |
|
hi there, was wondering if there's any updates? Is there a new version of the runner coming, maybe? Anything I can do to help? |
|
Hey, @fpkamp , my access to test farm was restored today, I'll have a look |
|
@ilia-shipitsin using macOS-13 and Xcode 14.3.1 😢 |
|
@stancho-stanchev , in current issue we investigate "Attempts to boot simulators and navigate to a URL fail very frequently." if you observe another regression, please open an issue. |
|
I acknowledge issue, @fpkamp , thank for repro steps.
|
|
@ilia-shipitsin cool, thank you. Maybe Arthur's observations here can help: Ticket 2248137. And just to confirm - did you get any exceptions or errors while issuing the commands? |
|
@fpkamp , XCode 15 Beta 6 was just deployed, please test. I did some dive on local installation. No conclusion yet, it does not work as I expect, but I'm not sure whether it is bug or feature. |
|
@ilia-shipitsin |
|
hello, @NorseGaud , @wtto00 , @joshuapoq , @fpkamp , @stancho-stanchev I've made some investigation, so far it looks like
mDNSResponder on the host seem to be nonneccessary and it is a leftover os using desktop mac mini. can you please try those commands in your pipelines to check
thanks! |
|
@ilia-shipitsin - I tried that as I'm very interested in restoring performance to the macOS-12 levels, but it appeared to make DNS in general non-functional. Test results indicate that after running that it is no longer possible to resolve anything so git checkout fails. Perhaps it is possible to carefully order when these commands are run such that the only activity after running them is machine-local but that seems like a fragile idea so I won't pursue it. Runs linked to failure line, "macOS-13-runner speedup" steps are just above them: |
|
@mikehardy , thank for your feedback. (also, I've found that simulator has its own diagnosticd, I'm looking at it) |
|
@ilia-shipitsin with that one line unloading diagnosticd.plist my workflow was functional (a good thing of course) but performance did not seem to be greatly enhanced if enhanced at all. Performance testing usually requires a large number of pre/post runs which I don't have the time to do, so I can't tell you with any confidence if it was some small percentage faster or not, but I can say that the difference between macos12 and macos13 runners (and/or Xcode15/ios17 vs behavior) was kind of a "two or more times slower" performance hit, and your change was not enough to restore that large of a difference - it was marginal if anything Hopefully that's useful. I won't have time to do much more testing today as I was just piggy-backing on a PR that was running workflows anyway and I need to move on |
|
@mikehardy , I'll try different things on https://github.com/invertase/react-native-firebase |
|
@mikehardy , is there a branch for macos 12 ? (I would like to compare things between macos-12 and macos-13 using that repo)
|
|
@ilia-shipitsin hmm not easily. I think you could change one line that specifies the simulator to something available on both and it would work though? https://github.com/invertase/react-native-firebase/blob/main/tests/package.json#L66 https://github.com/actions/runner-images/blob/main/images/macos/macos-12-Readme.md#installed-simulators "iPhone 14" is allegedly available for the value of Xcode that is "xcode-current" on macOS12 and macOS13, although that wasn't working for me for some reason on Xcode 15 - I don't recall trying too hard to investigate or fix it though If that doesn't work, could easily add a matrix check in the workflow yaml you are working with such that you execute a "tests:ios:test-cover" for macOS-13 and a new "tests:ios:test-cover-macos12" script you add here that references a new "ios.sim.debug.macos12" stanza you add here with iPhone 14 for macOS12 a little bit of a detour vs your goal here but I think that would work well and be quick to do |
|
@mikehardy , thank for the tip. I'm able to run "iPhone 14" on both macos-12 and macos-13. give me some time to compare results. actually, macos-12 runs test in 7 min while macos-13 need ~50 min even with "yeetd" enabled. no idea why, but we can try to find out |
macos-13 has been about that painful, yes 😂. Appreciate that you're examining, I believe everyone will appreciate any progress! |
|
@mikehardy , what caught my eyes is really huge simulator log. I noticed that tests run in debug mode. can we try to either disable logging or reduce them to release ? I collected logs from macos-12 and macos-13 and parsed them by Microsoft LogParser using the following query
something interesting with SpringBoard. what's that )) ? (well, I suspect it might be a regression on XCode itself or some side effect of running simulators on virtualized hardware, @NorseGaud do you have any idea why SpringBoard can be that noisy under anka platform maybe ?) |
|
the most noisy
|
|
as for "xpc connection" ....
|
Well, anything is possible @ilia-shipitsin :-) - my use case for the simulator.log is so that I can troubleshoot things like this when they go wrong though - it is not always our app, sometimes it is some rogue networking thing though, and the simulator.log is how it is discovered - as you've done (awesome by the way) I'm willing to try anything but I don't know how I'll do a more efficient job than you are already doing on the fork. From the perspective of our workflows the simulator.log is not used as any sort of end product it is just for troubleshooting, so you could disable the simuator.log capture / zip / upload entirely and our workflow would still be doing it's job (assuming there wasn't an app crash where I needed to grab the stack trace...) I'm going to guess though, disabling the simulator log will provide a substantial speedup by removing IO, but if you assume that the job is compute-bound and the IO is streaming/buffered then my guess is the macos13/Xcode15/ios17 slowdown we are all seeing is because the Simulator's networking/springboard subsysytem is doing something terrible that is also CPU heavy, so the speedup will not be nearly enough to get back to macOS12/Xcode14/ios16 levels https://developer.apple.com/documentation/xpc/1448777-xpc_connection_cancel - XPC is some interprocess communication, not sure which processes are trying to communicate and/or why it's failing but something is definitely not happy on these new versions of the iOS simulator stack |
|
@mikehardy , we did more investigation (will provide result later). also, we tried to run react-native-firebase on arm64 (also virtualized) runner https://github.com/bbq-beets/react-native-firebase/actions/runs/7030335355/job/19129684183 as far as I understand, arm64 does not support nested virtualization, but it should only affect android simulators, not iOS. can we add some debug to find why |
|
@ilia-shipitsin sorry for the delay! The test app connection is the last step in a wobbly tower of things that have to go correctly, it happens when the test infrastructure correctly requests Simulator start, the operating system has started the simulator correctly, the simulator has booted completely and the test infrastructure can see it is up, then the test infrastructure has correctly loaded the app on to the simulator and asked it to start and finally the app itself has started completely and fetched the javascript bundle from the bundle server, loaded it and started executing Determining why it did not start is what the simulator.log running in Debug is for :-) I look for a few markers to see what stage things got to. One is the case-insensitive string "crashlytics" - if this never shows up, the app native code never booted and ran so as a binary search, we're looking at simulator startup failure or app load/start failure (where failure may be timeout / just took too long) If "crashlytics" shows up then things should start happening pretty quickly and I search for the app name and/or "react" and/or "rnfb" to see if the javascript bundle loaded and app components started loading and getting chatty My guess is this was just more poor performance (a 2GB+ log file! ouch) meaning things were happening slowly enough that the test infrastructure considered it a failure and timed it out |
|
Hey, I'm dealing with similar issues using I was unable to get Detox to work using multiple workers. Then I tried using a single worker. This was using Xcode 14.3.1 and iOS 16.4 simulators. Is there a limitation on how many simulators can be run? Anyway, thanks for all the useful info in this thread. I'm gonna try a few more things. |
thank you, it was helpful. from current observation it looks like simulators are created on arm64, but due to degraded performance it looks like they are not responsive. |
|
Quick update from my testing with If I run this manually, I can choose the GPU type in the Simulator menu and see different times with a slight performance improvement using Integrated GPU. Not much of a difference, but it's something to note as a possibility for improvement. The console-messaging-full.log I don't know what |
|
Interesting @NorseGaud - do you have a workflow run URL you can point me to where you extracted those logs? Or could you specify the execution environment? I'm to go on a hunch that this was an Apple Silicon machine of some sort? That is a difference (intel silicon mac vs apple silicon mac) that varies the messaging testing as apple silicon macs with latest emulators can actually generate APNS tokens and receive APNS messages and we attempt to test that if it's recent enough simulator on apple silicon. It should work of course, but - for the purposes of the testing here, we are an open source repository limited to running on the currently available intel silicon runners, so I don't think it's germane to the current focus here unfortunately |
I don't :( I set up the project manually in a VM of several virtualization tools and got the same results across them all. The error is identical to the ones in the github runners. Specs for the logs provided:
Understood about not testing on Arm, though the result is exactly the same on macOS 13-14 and Xcode 15.x, regardless of the architecture. It's just faster and easier for me to test on ARM right now :) Unless there is something in these logs that indicates a problem with a service that VMs don't have or isn't working right, I do worry we won't be able to do much until we can describe in detail what Apple has to fix :( |
|
I am unable to reproduce boot issues anymore with iOS 17.2. The simulators load just fine (but are still a bit heavy on usage) for my test apps and no longer hang. Regarding certain functions, I don't see a difference though. |
|
Good to know @NorseGaud but unfortunately does not look like that's available in hosted runners yet https://github.com/actions/runner-images/blob/main/images/macos/macos-13-Readme.md#xcode |
|
Sorry @mikehardy , I was speaking about simulator booting issues. The issues with your tests persist AFAIK. |
This issue was moved to a discussion.
You can continue the conversation there. Go to discussion →
Description
Hi, I have been redirected from GitHub Support to describe our use case and perhaps influence future performance of your runner for macOS.
We have a use case where we need to use XCode 14.3.1 and XCode 15.0 on the macOS13 runner.
We create simulators with iPhones and navigate to a url and then we dispose the simulator.
All of the above works pretty well locally, but it fails when executing in GitHub Actions.
The behavior between XCode versions is different, with XCode 14 timing out on booting or navigating, however XCode 15 fails on other functions (perhaps an effect of changes to XCode itself) like binding launchd_sim.
An example failure message I get would be:
In the ticket that has originally redirected me here, Arthur says he was successful with adding a cache clean (
~/Library/Caches/and~/Library/Developer/CoreSimulator/Caches/) and waiting 60-120 seconds, however upon trying that I observed that it only works for a single simulator and subsequent simulators would fail (at least using a rinse and repeat approach).Platforms affected
Runner images affected
Image version and build link
version: 20230611.2
workflow run: https://github.com/fingerprintjs/fingerprintjs-pro/actions/runs/5646633166/job/15295123030
Is it regression?
no
Expected behavior
Simulators work smoothly and boot / open urls without crashing.
Actual behavior
Attempts to boot simulators and navigate to a URL fail very frequently.
Repro steps
Use the following script in a macOS 13 runner workflow:
open /Applications/Xcode_14.3.1.app/Contents/Developer/Applications/Simulator.app/ phone1=$(/Applications/Xcode_14.3.1.app/Contents/Developer/usr/bin/simctl create iPhone-hzso25lt6h9 com.apple.CoreSimulator.SimDeviceType.iPhone-14 com.apple.CoreSimulator.SimRuntime.iOS-16-4) echo "${phone1}" boot1=$(/Applications/Xcode_14.3.1.app/Contents/Developer/usr/bin/simctl boot ${phone1}) echo "${boot1}" phone2=$(/Applications/Xcode_14.3.1.app/Contents/Developer/usr/bin/simctl create iPhone-sios1839ti com.apple.CoreSimulator.SimDeviceType.iPhone-14 com.apple.CoreSimulator.SimRuntime.iOS-17-0) echo "${phone2}" boot2=$(/Applications/Xcode_14.3.1.app/Contents/Developer/usr/bin/simctl boot ${phone2}) rm -r ~/Library/Caches/* rm -r ~/Library/Developer/CoreSimulator/Caches/* sleep 120 nav1=$(/Applications/Xcode_14.3.1.app/Contents/Developer/usr/bin/simctl openurl booted 'https://google.com') echo "${nav1}" echo "${boot2}" rm -r ~/Library/Caches/* rm -r ~/Library/Developer/CoreSimulator/Caches/* sleep 120 nav2=$(/Applications/Xcode_14.3.1.app/Contents/Developer/usr/bin/simctl openurl ${phone2} 'https://google.com') echo "${nav2}"The text was updated successfully, but these errors were encountered: