[tests] Apple mobile test coverage improvements

[kotlarmilos]

Description

This issue lists improvements in Apple mobile infrastructure and test coverage. The tasks are categorized into four main areas: test coverage, reliability, scalability, and monitoring.

Expand runtime tests coverage

The primary goal is to align test coverage with the production environments, ensuring validation across all scenarios. Currently, only legacy runtime tests are executed on Apple mobile platforms using the interpreter, with a subset of smoke tests executed with Native AOT. The library and functional tests utilize the Mono AOT compiler. List of tasks:

• Document and implement test environment that reflects production usage for runtime, library, and functional tests
• Enable building runtime, library, and functional tests locally with a single command
• Improve Mono test coverage
  • [mono][ios] Enable fullAOT mode for runtime tests #92523
  • Implement Xunit runtime tests coverage using Mono runtime
• Improve Native AOT test coverage
  • Enable building/running libraries tests on iOS simulator and device with NativeAOT #81075
  • [infra] Implement XHarness test discovery/running without the XunitFrontController #91871
  • Enable building/running libraries tests on iOS simulator and device with NativeAOT #81075
  • [NativeAOT] Enable running libraries and functional tests locally and on CI #93447
  • [nativeaot][maccat] Add Native AOT runtime tests coverage on maccatalyst platforms #89437
  • Investigate how we can configure NativeAOT runtime tests to include Microsoft.NET.Sdk.props and with it Microsoft.NETCoreSdk.BundledVersions.props which defines NETCoreSdkPortableRuntimeIdentifier
• [mono][ios] Missing support for building native libraries as part of tests #92129

Scalability

The primary goal is to improve the overall test build and execution speed. List of tasks:

• iOS runtime tests should use just-run xharness command instead of run #95060
• [infra] Enable trimming on Helix for Apple mobile #100669
• Enable trimming for runtime tests
• [ios] Add support for multiple app bundles in the runtime pipeline for a test directory #89917
• Reduce test coverage redundancy between Apple mobile jobs and other non-mobile jobs
• [infra] Exclude 0-test groups to reduce the number of machines allocated on Helix #98213

Reliability

The primary goal is to establish a stable infrastructure environment for test execution. List of tasks:

• [tvos][test] mlaunch exited with 0, when test passed #95862
• [NativeAOT] Simplify build integration with external bundlers for Apple platforms #87187
• Failed USB connection via port 54050, error 61, in tvOS arm64 Release AllSubsets_Mono #82637
• Rename all the *.props, *.targets explicitly related to compilation on Helix - see comment
• Improve error and stacktrace logging
• Builds on Helix should not reuse publish folder for different builds tasks - see comment

Monitoring

• Add llvm check similar to this https://github.com/dotnet/runtime/pull/92644/files#diff-058f1e9bc8e4b23fb2d85884c92317671d13ecccb1cafa95f0f6c6430a600423 to the fullAOT jobs

The primary goal of monitoring should be enabling effective triage of active issues and tracking the reliability of tests and infrastructure over time. A list of tasks will be added.

[ghost]

Tagging subscribers to 'os-ios': @steveisok, @akoeplinger
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

This issue aims to expand the coverage of iOS runtime tests and improve the build process.

Expand runtime tests coverage on iOS

Currently, only tracing/eventpipe is running on iOS/iOSSimulator using interpreter as a fallback. This issue aims to expand the coverage of iOS runtime tests by including other testing groups and enabling full AOT compilation.

Parallel AOT compilation

To improve build time, the AOT compilation could be moved to Helix and executed simultaneously before the test execution. Additionally, unifying AOT compilation in one target would improve code maintenance as several targets use MonoAOTCompiler and AppleAppBuilderTask tasks.

Split build and test execution

To mitigate various transient issues, build and test execution could be divided into separate steps. The test build should build the repository, create tests, and zip artifacts (as done in build-test-job.yml). The test execution should download the artifacts, unzip them, and send the tests to Helix for AOT compilation and execution (as done in run-test-job.yml).

Tasks

• Identify other testing groups that could be executed on iOS/iOSSimulator and enable full AOT compilation for iOS/iOSSimulator tests
  • baseservices
  • JIT
  • GC
  • ilverify
  • Interop
  • Loader
  • tracing
• Move AOT compilation to Helix and execute it in parallel with test execution
• Unify AOT compilation to improve code maintenance
• Separate build and test execution into separate builds

Author:	kotlarmilos
Assignees:	kotlarmilos
Labels:	test-enhancement, area-Infrastructure-mono, os-ios
Milestone:	Future

[kotlarmilos]

To mitigate various transient issues, build and test execution could be divided into separate steps. The test build should build the runtime and tests, and zip artifacts. The test execution should download the artifacts, unzip them, and send the tests to Helix for AOT compilation and execution. Additionally, a single build could be shared across runtime, library, and functional tests.

@steveisok I think you mentioned once this may not be a good approach. My assumption is that it may reduce costs and development time in cases when transient issues occur. What could be the potential downsides of such an approach? Are there some bottlenecks?