Enable NativeAOT runtime tests on MacCatalyst #102882

kotlarmilos · 2024-05-30T14:46:14Z

Description

This PR enables a subset of runtime tests to run Native AOT on MacCatalyst. It updates the runtime pipeline to support executing these tests on MacCatalyst.

Changes

This PR updates the CLRTest.Execute.Bash.targets file to set the apple run command for MacCatalyst. The command apple just-run used on Apple mobile is not permitted, and apple test requires the a test runner. Additionally, it is necessary to locate Info.plist in the Contents/ directory and the binary in Contents/MacOS/ within the bundle.

Verification

The pipelines runtime-extra-platforms and runtime-maccatalyst should include Native AOT on MacCatalyst.

Additional Notes

We should review the runtime pipeline and consider consolidating it with the libraries pipeline.

Fixes #89437

kotlarmilos · 2024-05-30T14:47:57Z

/azp run runtime-maccatalyst

azure-pipelines · 2024-05-30T14:48:05Z

Azure Pipelines successfully started running 1 pipeline(s).

kotlarmilos · 2024-05-30T15:50:23Z

/azp run runtime-maccatalyst

azure-pipelines · 2024-05-30T15:50:33Z

Azure Pipelines successfully started running 1 pipeline(s).

kotlarmilos · 2024-05-31T07:41:56Z

/azp run runtime-maccatalyst

azure-pipelines · 2024-05-31T07:42:10Z

Azure Pipelines successfully started running 1 pipeline(s).

kotlarmilos · 2024-05-31T08:38:12Z

/azp run runtime-maccatalyst

azure-pipelines · 2024-05-31T08:38:23Z

Azure Pipelines successfully started running 1 pipeline(s).

kotlarmilos · 2024-05-31T09:11:38Z

/azp run runtime-maccatalyst

azure-pipelines · 2024-05-31T09:11:46Z

Azure Pipelines successfully started running 1 pipeline(s).

kotlarmilos · 2024-05-31T10:06:34Z

/azp run runtime-maccatalyst

azure-pipelines · 2024-05-31T10:06:43Z

Azure Pipelines successfully started running 1 pipeline(s).

kotlarmilos · 2024-05-31T11:35:57Z

/azp run runtime-maccatalyst

azure-pipelines · 2024-05-31T11:36:09Z

Azure Pipelines successfully started running 1 pipeline(s).

kotlarmilos · 2024-05-31T13:16:40Z

/azp run runtime-maccatalyst

azure-pipelines · 2024-05-31T13:16:50Z

Azure Pipelines successfully started running 1 pipeline(s).

kotlarmilos · 2024-06-07T14:17:37Z

The nativeaot/SmokeTests/Reflection is failing on arm64. It's worth looking into it and #100226.

ivanpovazan

Thanks for looking into this.
Overall it looks good, but I made a few comments and have questions before approving this.

ivanpovazan · 2024-06-10T11:23:19Z

eng/pipelines/extra-platforms/runtime-extra-platforms-maccatalyst.yml

+        - template: /eng/pipelines/common/templates/runtimes/build-runtime-tests-and-send-to-helix.yml
+          parameters:
+            creator: dotnet-bot
+            testBuildArgs: tree nativeaot/SmokeTests /p:BuildNativeAOTRuntimePack=true


Why do we need to set /p:BuildNativeAOTRuntimePack=true here, I thought the property is used only when building the runtime bits, not the tests?

I can't recall the initial reason for including it, and it is likely no longer needed since we now use a single build command. I believe it was related to the initial multi-command build process.

Without this property jobs reports:

eng/liveBuilds.targets(135,5): error : The 'coreclr' subset must be built before building this project.

Thanks for checking it out. I think we should investigate/refactor this (as a follow-up).

ivanpovazan · 2024-06-10T11:26:34Z

eng/pipelines/extra-platforms/runtime-extra-platforms-maccatalyst.yml

@@ -77,3 +77,40 @@ jobs:
            creator: dotnet-bot
            interpreter: true
            testRunNamePrefixSuffix: Mono_$(_BuildConfig)
+
+#
+# Build the whole product using Native AOT and run runtime tests


We should be running sandboxed tests as well, as that is matching more what the end user would experience (sandbox entitlement is required by the App Store).

There are failures for the Sandboxed version in the CI, I believe DevTeamProvisioning needs to be set to adhoc (but not 100% sure).

ivanpovazan · 2024-06-10T11:32:28Z

src/tests/Common/CLRTest.Execute.Bash.targets

@@ -484,6 +484,35 @@ $__Command $HARNESS_RUNNER apple just-run %5c
  -v
 CLRTestExitCode=$?

+# Exit code of xharness is zero when tests finished successfully


Why do we need special-case maccatalyst now or to be precise why just-run is not permitted?
How did it work before running maccatalyst tests with Mono?

The xharness doesn't support the just-run command for MacCatalyst: https://github.com/dotnet/xharness/blob/6ec0d581b6f420a225a13784911624eda64b2653/src/Microsoft.DotNet.XHarness.CLI/CommandArguments/Apple/AppleJustRunCommandArguments.cs#L51. I couldn't find the reason for this in the commit history. However, I believe this could be related to the installation process, which differs for MacCatalyst compared to other Apple mobile platforms.

The Mono tests on MacCatalyst include the AppleTestRunner. To support this for Native AOT, we need to implement #91871.

/cc: @premun Do you know why apple [un]install and apple just-run commands are not supported for MacCatalyst?

If I remember correctly, run accepts path to the app which it installs and runs (for all platforms). For maccatalyst it technically just calls open [app].

The just-run command accepts app name and just opens the app on the device without installation. maccatalyst needs to have the app bundle path because technically we don't install anything anywhere. So it could technically get the path instead of the app bundle name but I guess it just didn't make sense because of this.

Thanks for the explanation, I am good with the changes.

ivanpovazan · 2024-06-10T11:41:45Z

src/tests/build.proj

-    <Copy SourceFiles="@(RecursiveCopyHack)" DestinationFolder="$(FinalPath)/%(RecursiveDir)" />
+    <!-- MacCatalyst requires Contents/Info.plist and Contents/MacOS/ApplicationBinary -->
+    <Copy Condition="'$(TargetOS)' == 'maccatalyst' AND '%(BundleContent.Filename)%(BundleContent.Extension)' == 'Info.plist'" SourceFiles="@(BundleContent)" DestinationFolder="$(FinalPath)/Contents/%(RecursiveDir)" />
+    <Copy Condition="'$(TargetOS)' == 'maccatalyst' AND '%(BundleContent.Filename)%(BundleContent.Extension)' != 'Info.plist'" SourceFiles="@(BundleContent)" DestinationFolder="$(FinalPath)/Contents/MacOS/%(RecursiveDir)" />
+    <Copy Condition="'$(TargetOS)' != 'maccatalyst'" SourceFiles="@(BundleContent)" DestinationFolder="$(FinalPath)/%(RecursiveDir)" />


Isn't Contents dir already contained in the BundleContent? (I am not sure why we need to special-case maccatalyst)

It should be already there, but I couldn't recursively copy content in msbuild from the build directory to the bundle. The RecursiveDir metadata for BundleContent is empty here. Please share any ideas on how to avoid "manual" copy.

It is in place to verify E2E tests, but I think we need to resolve this before merging.

The RecursiveDir metadata for BundleContent is empty here.

Yes, that is what I was confused about.

kotlarmilos · 2024-06-11T13:35:05Z

/azp run runtime-maccatalyst

azure-pipelines · 2024-06-11T13:35:18Z

Azure Pipelines successfully started running 1 pipeline(s).

kotlarmilos · 2024-06-12T11:05:43Z

/azp run runtime-maccatalyst

azure-pipelines · 2024-06-12T11:05:58Z

Azure Pipelines successfully started running 1 pipeline(s).

kotlarmilos · 2024-06-12T11:19:16Z

/azp run runtime-maccatalyst

azure-pipelines · 2024-06-12T11:19:26Z

Azure Pipelines successfully started running 1 pipeline(s).

kotlarmilos · 2024-06-12T15:50:26Z

/azp run runtime-maccatalyst

azure-pipelines · 2024-06-12T15:50:36Z

Azure Pipelines successfully started running 1 pipeline(s).

Run nativeaot/SmokeTests on MacCatalyst

252bfd7

kotlarmilos added the area-Infrastructure-coreclr label May 30, 2024

kotlarmilos added this to the 9.0.0 milestone May 30, 2024

kotlarmilos self-assigned this May 30, 2024

kotlarmilos added NO-MERGE The PR is not ready for merge yet (see discussion for detailed reasons) NO-REVIEW Experimental/testing PR, do NOT review it labels May 30, 2024

Initialize CMAKE_OSX_DEPLOYMENT_TARGET for Swift tests on MacCatalyst

454c84b

Include dotnet sdk for MacCatalyst target

7f1201c

Include dotnet sdk for MacCatalyst target

5845dc7

Include dotnet sdk for MacCatalyst target

67100e7

Include dotnet sdk for MacCatalyst target

4faf43f

Add maccatalyst target to xharness command

662dd78

Update test runner to support maccatalyst

2beb301

Fix app path

f321057

kotlarmilos added os-maccatalyst MacCatalyst OS and removed NO-MERGE The PR is not ready for merge yet (see discussion for detailed reasons) NO-REVIEW Experimental/testing PR, do NOT review it labels Jun 7, 2024

kotlarmilos marked this pull request as ready for review June 7, 2024 14:10

kotlarmilos requested review from ivanpovazan and matouskozak June 7, 2024 14:10

kotlarmilos changed the title ~~[WIP] Enable NativeAOT runtime tests on MacCatalyst~~ Enable NativeAOT runtime tests on MacCatalyst Jun 7, 2024

build-analysis bot mentioned this pull request Jun 7, 2024

CI timeout in System.Data.Tests.DataTableReadXmlSchemaTest.XsdSchemaDeserializationIgnoresLocale #103062

Open

ivanpovazan reviewed Jun 10, 2024

View reviewed changes

kotlarmilos added 2 commits June 11, 2024 15:33

Remove BuildNativeAOTRuntimePack prop

86777fc

Add App Sandbox entitlement tests

ed006a2

This was referenced Jun 11, 2024

NativeAOT legs timing out in CI #102239

Closed

The Operation will be canceled. The next steps may not contain expected logs. dotnet/dnceng#3008

Open

Add BuildNativeAOTRuntimePack property

75582d3

Move BuildNativeAOTRuntimePack to build args

f90f0d7

This was referenced Jun 12, 2024

Invariant.Tests.dll test failure on wasm #102128

Open

[browser][ST] Wasm tests timeout #103246

Open

Revert changes

61b4358

Enable NativeAOT runtime tests on MacCatalyst #102882

Are you sure you want to change the base?

Enable NativeAOT runtime tests on MacCatalyst #102882

Conversation

kotlarmilos commented May 30, 2024 • edited Loading

Description

Changes

Verification

Additional Notes

kotlarmilos commented May 30, 2024

azure-pipelines bot commented May 30, 2024

kotlarmilos commented May 30, 2024

azure-pipelines bot commented May 30, 2024

kotlarmilos commented May 31, 2024

azure-pipelines bot commented May 31, 2024

kotlarmilos commented May 31, 2024

azure-pipelines bot commented May 31, 2024

kotlarmilos commented May 31, 2024

azure-pipelines bot commented May 31, 2024

kotlarmilos commented May 31, 2024

azure-pipelines bot commented May 31, 2024

kotlarmilos commented May 31, 2024

azure-pipelines bot commented May 31, 2024

kotlarmilos commented May 31, 2024

azure-pipelines bot commented May 31, 2024

kotlarmilos commented Jun 7, 2024

ivanpovazan left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

premun Jun 11, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kotlarmilos commented Jun 11, 2024

azure-pipelines bot commented Jun 11, 2024

kotlarmilos commented Jun 12, 2024

azure-pipelines bot commented Jun 12, 2024

kotlarmilos commented Jun 12, 2024

azure-pipelines bot commented Jun 12, 2024

kotlarmilos commented Jun 12, 2024

azure-pipelines bot commented Jun 12, 2024

kotlarmilos commented May 30, 2024 •

edited

Loading

premun Jun 11, 2024 •

edited

Loading