iOS UI tests broken in actions using Xcode 15 / iOS17

[johnmalatras]

Select Topic Area

Bug

Body

Hi, not sure if this is better suited for an issue on https://github.com/actions/runner but wanted to check here first. After moving my CI checks over to Xcode 15 / iOS 17 simulator / Swift 5.8 I get intermittent timeouts when running UI tests. Sometimes I get:
The test runner failed to initialize for UI testing. (Underlying Error: Timed out while loading Accessibility.))

Other times it seems to make it past initializing, but individual tests fail with:
Waiting for startup block completion timed out (NSInternalInconsistencyException)

I added an issue with a helper library I'm using for UI tests and they pointed out that others are seeing hangs as well with other providers like CircleCI: https://discuss.circleci.com/t/severe-performance-problems-with-xcode-15/49205/35

Is this a known issue? Any workaround? And any update on plans to make macOS Sonoma available in runners?

[mschuetz-viz]

Same issue here - was working fine with Xcode 14.3.1. I believe this is related to a known issue in Xcode 15 (here), but I may be wrong.

Executing Unit/UI tests from Xcode on the iOS Simulator takes an extended time to launch on first run of the suite. (115187363) (110330776) (FB12237092)

[NachoSoto]

The same issue occurs in CircleCI, and apparently Azure too.

Unfortunately Xcode 15.1 doesn't fix this either: https://discuss.circleci.com/t/severe-performance-problems-with-xcode-15/49205/47

[DylanRJ]

Hi, did anyone get to the bottom of this issue? Seeing the same in our pipelines

[leopic]

For our workflows we use Monterrey (macOS-12) and Xcode 14.2, this is until now the ideal scenario where our entire UI Test suite takes around an hour to complete.

We've tested different combinations of macOS and iOS versions but they all got incrementally worst:

• macOS-12 and Xcode 14.2: 1:01 to completion
• macOS-13 and Xcode 14.2: 2:30 hours before timing out
• macOS-13 and Xcode 14.3.1: 6:02 before timing out
• macOS-13 and Xcode 15.0.1: 3:32 multiple failures

I've tried running Sonoma (macOS-14) and Xcode 15.1.0 locally and things are slightly better, but nowhere as stable as with Monterrey and Xcode 14.2.

I understand this is not a GitHub only issue and have noticed, as others, that every other CI provider: Bitrise, CircleCI, etc. are currently struggling with this problem. Is there any recommendation in the mean time?

We will most likely pause our efforts to move away from Monterrey in the meantime.

[sssuourabh]

Any solution ?

[cah-iswaria-sasi]

?