Description
Simulator-based tests (iOS and tvOS) are randomly failing with LaunchTimedOut across multiple test suites (linker, monotouch, fsharp, xcframework, introspection). When this occurs, typically ALL tests in the affected test suite on a given agent fail, suggesting the issue is at the agent/simulator level rather than individual test level.
macOS and Mac Catalyst tests are never affected (they don't use simulators).
Mechanism
The LaunchTimedOut result is set when the test app fails to connect back to the TCP listener within LaunchTimeout (3 minutes in CI). This means either:
- The simulator fails to boot
- The app fails to install on the simulator
- The app crashes immediately at startup before establishing a TCP connection
- Network/port connectivity issues between the simulator and the host
The relevant code path: tests/xharness/AppRunner.cs line 287-290 sets a 3-minute timeout on listener.ConnectedTask, and tests/xharness/TestReporter.cs line 470 categorizes it as LaunchTimedOut when TestExecutionStarted is false (i.e. the listener never received a connection).
Frequency
This has been happening frequently over the past 2+ weeks. Here are all occurrences from April 20 - May 3, 2026:
| Date |
Build ID |
# Failed |
PR |
| 2026-04-20 |
13882756 |
23 |
#25105 |
| 2026-04-20 |
13882766 |
23 |
#25105 |
| 2026-04-23 |
13918176 |
4 |
#25233 |
| 2026-04-24 |
13929603 |
22 |
#25243 |
| 2026-04-27 |
13945394 |
6 |
#25246 |
| 2026-04-27 |
13944752 |
28 |
#25251 |
| 2026-04-27 |
13944778 |
24 |
#25238 |
| 2026-04-27 |
13945852 |
2 |
#25197 |
| 2026-04-27 |
13946259 |
2 |
#25254 |
| 2026-04-28 |
13952590 |
2 |
#25197 |
| 2026-04-28 |
13952530 |
21 |
#25261 |
| 2026-04-28 |
13952539 |
15 |
#25197 |
| 2026-04-28 |
13954991 |
2 |
#25261 |
| 2026-04-28 |
13954558 |
37 |
#25262 |
| 2026-04-28 |
13955255 |
6 |
#25197 |
| 2026-04-28 |
13955774 |
2 |
#25254 |
| 2026-04-28 |
13956330 |
4 |
#25239 |
| 2026-04-28 |
13956481 |
2 |
#25274 |
| 2026-04-28 |
13956372 |
24 |
#25272 |
| 2026-04-28 |
13956568 |
2 |
#25274 |
| 2026-04-28 |
13957432 |
11 |
#25277 |
| 2026-04-28 |
13957717 |
15 |
#25197 |
| 2026-04-29 |
13959788 |
15 |
#25261 |
| 2026-04-29 |
13965758 |
15 |
#25197 |
| 2026-04-29 |
13968350 |
2 |
#25271 |
| 2026-04-29 |
13968391 |
11 |
#25277 |
| 2026-04-30 |
13977914 |
16 |
#25282 |
| 2026-04-30 |
13980444 |
2 |
#25234 |
| 2026-04-30 |
13980447 |
2 |
#25234 |
| 2026-04-30 |
13981379 |
2 |
#25239 |
| 2026-04-30 |
13981376 |
2 |
#25239 |
| 2026-04-30 |
13983337 |
2 |
#25286 |
| 2026-04-30 |
13983369 |
2 |
#25292 |
| 2026-04-30 |
13983599 |
11 |
#25293 |
| 2026-04-30 |
13983087 |
16 |
#25291 |
| 2026-05-01 |
13989124 |
22 |
#25294 |
| 2026-05-01 |
13989221 |
4 |
#25294 |
| 2026-05-02 |
13998898 |
43 |
#25296 |
| 2026-05-02 |
13998894 |
22 |
#25282 |
| 2026-05-03 |
14001884 |
15 |
#25294 |
| 2026-05-03 |
14001881 |
6 |
#25294 |
Affected test suites
All simulator-based test suites are affected:
- linker tests (dont link, link sdk, link all, trimmode copy, trimmode link)
- monotouch tests (iOS, tvOS)
- fsharp tests
- xcframework tests
- introspection tests
Only iOS and tvOS simulator tests are affected — macOS and Mac Catalyst tests are never affected.
Observations
- When it hits, it tends to affect all tests in a test suite run, not just individual ones.
- There is no correlation with any specific PR or code change — it happens across unrelated PRs.
- The
simctl diagnose step that could provide diagnostic info is currently gated on system.debug == true (line 152 of run-tests.yml), so diagnostic info is rarely captured.
- The 3-minute
LaunchTimeout in CI (Harness.cs line 266) may be too short if the agent is under load.
Possible improvements
- Always run
simctl diagnose when a LaunchTimedOut occurs (not just when system.debug is true).
- Add diagnostic logging in
TestReporter.LaunchCallback and AppRunner to capture simulator state (running simulators, device availability) when the launch times out.
- Consider increasing
LaunchTimeout from 3 minutes to something higher (e.g., 5 minutes).
- Log the agent name in the test result comment to help identify if specific agents are problematic.
Originally reported
#25282 (comment)
Description
Simulator-based tests (iOS and tvOS) are randomly failing with
LaunchTimedOutacross multiple test suites (linker, monotouch, fsharp, xcframework, introspection). When this occurs, typically ALL tests in the affected test suite on a given agent fail, suggesting the issue is at the agent/simulator level rather than individual test level.macOS and Mac Catalyst tests are never affected (they don't use simulators).
Mechanism
The
LaunchTimedOutresult is set when the test app fails to connect back to the TCP listener withinLaunchTimeout(3 minutes in CI). This means either:The relevant code path:
tests/xharness/AppRunner.csline 287-290 sets a 3-minute timeout onlistener.ConnectedTask, andtests/xharness/TestReporter.csline 470 categorizes it asLaunchTimedOutwhenTestExecutionStartedis false (i.e. the listener never received a connection).Frequency
This has been happening frequently over the past 2+ weeks. Here are all occurrences from April 20 - May 3, 2026:
Affected test suites
All simulator-based test suites are affected:
Only iOS and tvOS simulator tests are affected — macOS and Mac Catalyst tests are never affected.
Observations
simctl diagnosestep that could provide diagnostic info is currently gated onsystem.debug == true(line 152 ofrun-tests.yml), so diagnostic info is rarely captured.LaunchTimeoutin CI (Harness.csline 266) may be too short if the agent is under load.Possible improvements
simctl diagnosewhen a LaunchTimedOut occurs (not just whensystem.debugis true).TestReporter.LaunchCallbackandAppRunnerto capture simulator state (running simulators, device availability) when the launch times out.LaunchTimeoutfrom 3 minutes to something higher (e.g., 5 minutes).Originally reported
#25282 (comment)