[iOS][MacOS] Run tests on iOS 17 and MacOS 14 scouting queues

[steveisok]

This will hopefully show us where we currently stand.

[ghost]

Tagging subscribers to this area: @dotnet/area-infrastructure-libraries
See info in area-owners.md if you want to be subscribed.

Issue Details

---

This will hopefully show us where we currently stand.

Author:	steveisok
Assignees:	-
Labels:	area-Infrastructure-libraries
Milestone:	-

[steveisok]

The chances aren't good for any of the failures to show up in azdo as the job will likely time out. However, the work items will finish and can be analyzed.

[steveisok]

@akoeplinger does anything in xharness / mlaunch need updated?

[vcsjones]

I would consider an outerloop run here, too. We have a good number of S.S.Cryptography tests that are outer loop that I would be interested in understanding their behavior on macOS 14. Particularly some of the X.509 things that we know tend to break from release to release.

[steveisok]

/cc @dotnet/runtime-infrastructure

[steveisok]

/azp list

[azure-pipelines]

CI/CD Pipelines for this repository:

• runtime-coreclr outerloop
• runtime-coreclr jitstress
• runtime-coreclr jitstressregs
• runtime-coreclr jitstress2-jitstressregs
• runtime-coreclr gcstress0x3-gcstress0xc
• runtime-coreclr gcstress-extra
• runtime-coreclr r2r-extra
• runtime-coreclr jitstress-isas-x86
• runtime-coreclr jitstress-isas-arm
• runtime-coreclr jitstressregs-x86
• runtime-coreclr libraries-jitstressregs
• runtime-coreclr libraries-jitstress2-jitstressregs
• runtime-coreclr r2r
• runtime-coreclr runincontext
• runtime-coreclr crossgen2
• runtime-libraries-coreclr outerloop
• runtime-libraries-coreclr outerloop-windows
• runtime-libraries-coreclr outerloop-linux
• runtime-libraries-coreclr outerloop-osx
• runtime
• runtime-libraries enterprise-linux
• runtime-libraries stress-http
• runtime-libraries stress-ssl
• runtime-dev-innerloop
• runtime-coreclr crossgen2 outerloop
• coreclr-release-outerloop-nightly
• runtime-coreclr crossgen2-composite
• runtime-jit-experimental
• runtime-coreclr libraries-jitstress
• dotnet-linker-tests
• runtime-coreclr ilasm
• runtime-coreclr crossgen2-composite gcstress
• runtime-staging
• runtime-coreclr pgo
• runtime-coreclr libraries-pgo
• Antigen
• runtime-community
• Fuzzlyn
• runtime-coreclr superpmi-replay
• runtime-wasm
• runtime-coreclr superpmi-diffs
• runtime-coreclr superpmi-asmdiffs-checked-release
• runtime-extra-platforms
• jit-cfg
• runtime-wasm-perf
• runtime-llvm
• runtime-coreclr jitstress-random
• runtime-coreclr libraries-jitstress-random
• runtime-android-grpc-client-tests
• runtime-wasm-libtests
• runtime-wasm-non-libtests
• runtime-android
• runtime-androidemulator
• runtime-ioslike
• runtime-ioslikesimulator
• runtime-linuxbionic
• runtime-maccatalyst
• runtime-coreclr pgostress
• runtime-coreclr jitstress-isas-avx512
• runtime-tools-tests
• runtime-libraries-mono outerloop
• runtime-sanitized
• runtime-wasm-dbgtests
• runtime-wasm-optional

[steveisok]

/azp run runtime-libraries-coreclr outerloop-osx

[azure-pipelines]

No commit pushedDate could be found for PR 88479 in repo dotnet/runtime

[akoeplinger]

@akoeplinger does anything in xharness / mlaunch need updated?

I'm seeing this from xharness in the log so that seems likely:

[09:43:43] crit: System.Exception: Failed to list devices.
                    at Microsoft.DotNet.XHarness.iOS.Shared.Hardware.HardwareDeviceLoader.LoadDevices(ILog log, Boolean includeLocked, Boolean forceRefresh, Boolean listExtraData, Boolean includeWirelessDevices, CancellationToken cancellationToken) in /_/src/Microsoft.DotNet.XHarness.iOS.Shared/Hardware/HardwareDeviceLoader.cs:line 103

[akoeplinger]

More detailed log from mlaunch:

[09:43:43.8203300] error HE0004: Could not load the framework 'IDEProducts' (path: /Applications/Xcode15_beta.app/Contents/SharedFrameworks/IDEProducts.framework/IDEProducts): 
[09:43:43.8203930] dlopen(/Applications/Xcode15_beta.app/Contents/SharedFrameworks/IDEProducts.framework/IDEProducts, 0x0001): Library not loaded: @rpath/IDEDistribution.framework/Versions/A/IDEDistribution
[09:43:43.8204040]   Referenced from: <33B36F10-89CE-3E88-AECE-A71AD0608B28> /Applications/Xcode15_beta.app/Contents/SharedFrameworks/IDEProducts.framework/Versions/A/IDEProducts
[09:43:43.8204400]   Reason: tried: '/private/tmp/helix/working/A4BB094A/p/microsoft.dotnet.xharness.cli/8.0.0-prerelease.23326.1/runtimes/any/native/mlaunch/lib/mlaunch/mlaunch.app/Contents/Frameworks/IDEDistribution.framework/Versions/A/IDEDistribution' (no such file), '/Applications/Xcode15_beta.app/Contents/SharedFrameworks/IDEProducts.framework/Versions/A/Frameworks/IDEDistribution.framework/Versions/A/IDEDistribution' (no such file), '/private/tmp/helix/working/A4BB094A/p/microsoft.dotnet.xharness.cli/8.0.0-prerelease.23326.1/runtimes/any/native/mlaunch/lib/mlaunch/mlaunch.app/Contents/Frameworks/IDEDistribution.framework/Versions/A/IDEDistribution' (no such file), '/Applications/Xcode15_beta.app/Contents/SharedFrameworks/IDEProducts.framework/Versions/A/Frameworks/IDEDistribution.framework/Versions/A/IDEDistribution' (no such file)
[09:43:43.8204570]         
[09:43:43.8239930]   at Xamarin.Hosting.Services.LoadFramework (System.String framework, System.Boolean warnIfMissing) [0x00089] in /Users/builder/azdo/_work/3/s/tools/mlaunch/Xamarin.Hosting/Xamarin.Hosting/Services.cs:275 
[09:43:43.8240200]   at Xamarin.Hosting.Services.LoadFrameworks () [0x00423] in /Users/builder/azdo/_work/3/s/tools/mlaunch/Xamarin.Hosting/Xamarin.Hosting/Services.cs:164 
[09:43:43.8240300]   at Xamarin.Hosting.Services.Initialize (System.String xcode_location, System.Boolean enable_xcode_plugins) [0x000d8] in /Users/builder/azdo/_work/3/s/tools/mlaunch/Xamarin.Hosting/Xamarin.Hosting/Services.cs:91 
[09:43:43.8240370]   at Xamarin.Launcher.DevController.Initialize () [0x00001] in /Users/builder/azdo/_work/3/s/tools/mlaunch/Xamarin.Hosting/Xamarin.Launcher/controller-device.cs:35 
[09:43:43.8240440]   at Xamarin.Launcher.DevController.ListDevicesAsync (Xamarin.Launcher.OutputFormat format, System.String output_file, System.Boolean list_extra_data) [0x0003f] in /Users/builder/azdo/_work/3/s/tools/mlaunch/Xamarin.Hosting/Xamarin.Launcher/controller-device.cs:446 
[09:43:43.8240700]   at Xamarin.Utils.NSRunLoopExtensions.RunUntilTaskCompletion[T] (Foundation.NSRunLoop this, System.Threading.Tasks.Task`1[TResult] task) [0x00082] in /Users/builder/azdo/_work/3/s/tools/mlaunch/Xamarin.Hosting/Xamarin.Utils/Extensions.cs:35 
[09:43:43.8240790]   at Xamarin.Launcher.Driver.Main2 (System.String[] args) [0x00ba3] in /Users/builder/azdo/_work/3/s/tools/mlaunch/Xamarin.Hosting/Xamarin.Launcher/Main.cs:435 
[09:43:43.8240860]   at Xamarin.Launcher.Driver.Main (System.String[] args) [0x0006d] in /Users/builder/azdo/_work/3/s/tools/mlaunch/Xamarin.Hosting/Xamarin.Launcher/Main.cs:123 
[09:43:43.8298690] Process mlaunch exited with 1
[09:43:43.8536290] Failed to parse device list: System.Exception: Failed to list devices.

[akoeplinger]

I've sent a PR to mlaunch with the fix.

[ghost]

Draft Pull Request was automatically closed for 30 days of inactivity. Please let us know if you'd like to reopen it.

[steveisok]

@akoeplinger the latest run with an updated xharness gives me:

[13:03:13.7809040] Xamarin.Hosting: Loaded Xcode plugins successfully.
[13:03:13.8164650] error MT0000: Unexpected error - Please file a bug report at https://github.com/xamarin/xamarin-macios/issues/new
[13:03:13.8830640] System.IO.FileNotFoundException: Could not find required file 'net.dot.System.Runtime.Tests/Info.plist'.
[13:03:13.8831020]    at Xamarin.Hosting.PList.FromFile(String) in /Users/builder/azdo/_work/1/s/tools/mlaunch/Xamarin.Hosting/Xamarin.Hosting/PList.cs:line 24
[13:03:13.8831110]    at Xamarin.Hosting.PlistBasedAppBundle..ctor(String) in /Users/builder/azdo/_work/1/s/tools/mlaunch/Xamarin.Hosting/Xamarin.Hosting/PlistBasedAppBundle.cs:line 21
[13:03:13.8831300]    at Xamarin.Launcher.Driver.LaunchOrDebugUsingDeviceCtl(Action, RealDevice) in /Users/builder/azdo/_work/1/s/tools/mlaunch/Xamarin.Hosting/Xamarin.Launcher/Main.cs:line 576
[13:03:13.8831390]    at Xamarin.Launcher.Driver.LaunchOrDebugAsync(Action) in /Users/builder/azdo/_work/1/s/tools/mlaunch/Xamarin.Hosting/Xamarin.Launcher/Main.cs:line 547
[13:03:13.8831450]    at Xamarin.Launcher.Driver.MainAsync() in /Users/builder/azdo/_work/1/s/tools/mlaunch/Xamarin.Hosting/Xamarin.Launcher/Main.cs:line 491
[13:03:13.8831570]    at Xamarin.Utils.NSRunLoopExtensions.RunUntilTaskCompletion[T](NSRunLoop, Task`1) in /Users/builder/azdo/_work/1/s/tools/mlaunch/Xamarin.Hosting/Xamarin.Utils/Extensions.cs:line 30
[13:03:13.8831640]    at Xamarin.Launcher.Driver.Main2(String[]) in /Users/builder/azdo/_work/1/s/tools/mlaunch/Xamarin.Hosting/Xamarin.Launcher/Main.cs:line 436
[13:03:13.8831710]    at Xamarin.Launcher.Driver.Main(String[]) in /Users/builder/azdo/_work/1/s/tools/mlaunch/Xamarin.Hosting/Xamarin.Launcher/Main.cs:line 123

[ghost]

Draft Pull Request was automatically closed for 30 days of inactivity. Please let us know if you'd like to reopen it.