Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable NativeAOT runtime tests on MacCatalyst #102882

Open
wants to merge 21 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 16 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -77,3 +77,40 @@ jobs:
creator: dotnet-bot
interpreter: true
testRunNamePrefixSuffix: Mono_$(_BuildConfig)

#
# Build the whole product using Native AOT and run runtime tests
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should be running sandboxed tests as well, as that is matching more what the end user would experience (sandbox entitlement is required by the App Store).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are failures for the Sandboxed version in the CI, I believe DevTeamProvisioning needs to be set to adhoc (but not 100% sure).

#
- template: /eng/pipelines/common/platform-matrix.yml
parameters:
jobTemplate: /eng/pipelines/common/global-build-job.yml
helixQueuesTemplate: /eng/pipelines/coreclr/templates/helix-queues-setup.yml
buildConfig: Release
runtimeFlavor: coreclr
isExtraPlatformsBuild: ${{ parameters.isExtraPlatformsBuild }}
isMacCatalystOnlyBuild: ${{ parameters.isMacCatalystOnlyBuild }}
platforms:
- maccatalyst_x64
- maccatalyst_arm64
variables:
# map dependencies variables to local variables
- name: librariesContainsChange
value: $[ stageDependencies.EvaluatePaths.evaluate_paths.outputs['SetPathVars_libraries.containsChange'] ]
- name: monoContainsChange
value: $[ stageDependencies.EvaluatePaths.evaluate_paths.outputs['SetPathVars_mono_excluding_wasm.containsChange'] ]
jobParameters:
testGroup: innerloop
nameSuffix: AllSubsets_NativeAOT_RuntimeTests
buildArgs: --cross -s clr.alljits+clr.tools+clr.nativeaotruntime+clr.nativeaotlibs+libs -c $(_BuildConfig)
timeoutInMinutes: 180
# extra steps, run tests
extraVariablesTemplates:
- template: /eng/pipelines/common/templates/runtimes/test-variables.yml
parameters:
testGroup: innerloop
postBuildSteps:
- template: /eng/pipelines/common/templates/runtimes/build-runtime-tests-and-send-to-helix.yml
parameters:
creator: dotnet-bot
testBuildArgs: tree nativeaot/SmokeTests /p:BuildNativeAOTRuntimePack=true
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need to set /p:BuildNativeAOTRuntimePack=true here, I thought the property is used only when building the runtime bits, not the tests?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can't recall the initial reason for including it, and it is likely no longer needed since we now use a single build command. I believe it was related to the initial multi-command build process.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Without this property jobs reports:

eng/liveBuilds.targets(135,5): error : The 'coreclr' subset must be built before building this project.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for checking it out. I think we should investigate/refactor this (as a follow-up).

testRunNamePrefixSuffix: NativeAOT_$(_BuildConfig)
2 changes: 1 addition & 1 deletion src/libraries/sendtohelix-mobile.targets
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@

<Target Name="PrepareForBuildHelixWorkItems_Mobile">

<PropertyGroup Condition="'$(TargetOS)' == 'ios' or '$(TargetOS)' == 'tvos' or '$(TargetOS)' == 'iossimulator' or '$(TargetOS)' == 'tvossimulator' or '$(TargetOS)' == 'maccatalyst'">
<PropertyGroup Condition="'$(TargetsAppleMobile)' == 'true'">
<AppleTestTarget Condition="'$(TargetOS)' == 'iossimulator'">ios-simulator-64</AppleTestTarget>
<AppleTestTarget Condition="'$(TargetOS)' == 'tvossimulator'">tvos-simulator</AppleTestTarget>
<AppleTestTarget Condition="'$(TargetOS)' == 'ios'">ios-device</AppleTestTarget>
Expand Down
31 changes: 30 additions & 1 deletion src/tests/Common/CLRTest.Execute.Bash.targets
Original file line number Diff line number Diff line change
Expand Up @@ -457,7 +457,7 @@ CLRTestExitCode=$?
CLRTestExpectedExitCode=0
]]>
</BashCLRTestLaunchCmds>
<BashCLRTestLaunchCmds Condition="'$(CLRTestKind)' == 'BuildAndRun' and ('$(TargetOS)' == 'iossimulator' or '$(TargetOS)' == 'ios' or '$(TargetOS)' == 'tvos' or '$(TargetOS)' == 'tvossimulator' or '$(TargetOS)' == 'maccatalyst')">
<BashCLRTestLaunchCmds Condition="'$(CLRTestKind)' == 'BuildAndRun' and ('$(TargetOS)' == 'iossimulator' or '$(TargetOS)' == 'ios' or '$(TargetOS)' == 'tvos' or '$(TargetOS)' == 'tvossimulator')">
<![CDATA[
__Command=""
if [ ! -z ${__TestDotNetCmd+x} ] %3B then
Expand All @@ -484,6 +484,35 @@ $__Command $HARNESS_RUNNER apple just-run %5c
-v
CLRTestExitCode=$?

# Exit code of xharness is zero when tests finished successfully
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need special-case maccatalyst now or to be precise why just-run is not permitted?
How did it work before running maccatalyst tests with Mono?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The xharness doesn't support the just-run command for MacCatalyst: https://github.com/dotnet/xharness/blob/6ec0d581b6f420a225a13784911624eda64b2653/src/Microsoft.DotNet.XHarness.CLI/CommandArguments/Apple/AppleJustRunCommandArguments.cs#L51. I couldn't find the reason for this in the commit history. However, I believe this could be related to the installation process, which differs for MacCatalyst compared to other Apple mobile platforms.

The Mono tests on MacCatalyst include the AppleTestRunner. To support this for Native AOT, we need to implement #91871.

/cc: @premun Do you know why apple [un]install and apple just-run commands are not supported for MacCatalyst?

Copy link
Member

@premun premun Jun 11, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I remember correctly, run accepts path to the app which it installs and runs (for all platforms). For maccatalyst it technically just calls open [app].

The just-run command accepts app name and just opens the app on the device without installation. maccatalyst needs to have the app bundle path because technically we don't install anything anywhere. So it could technically get the path instead of the app bundle name but I guess it just didn't make sense because of this.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the explanation, I am good with the changes.

CLRTestExpectedExitCode=0
]]>
</BashCLRTestLaunchCmds>
<BashCLRTestLaunchCmds Condition="'$(CLRTestKind)' == 'BuildAndRun' and '$(TargetOS)' == 'maccatalyst'">
<![CDATA[
__Command=""
if [ ! -z ${__TestDotNetCmd+x} ] %3B then
__Command+=" $__TestDotNetCmd"
else
__Command+=" dotnet"
fi

if [ ! -z "$XHARNESS_CLI_PATH" ]; then
# When running in CI, we only have the .NET runtime available
# We need to call the XHarness CLI DLL directly via dotnet exec
HARNESS_RUNNER="exec $XHARNESS_CLI_PATH"
else
HARNESS_RUNNER="xharness"
fi

$__Command $HARNESS_RUNNER apple run %5c
--app="${__TestBinaryBase}/${__Category}.app" %5c
--output-directory="$__OutputDir" %5c
--expected-exit-code=100 %5c
--targets maccatalyst %5c
-v
CLRTestExitCode=$?

# Exit code of xharness is zero when tests finished successfully
CLRTestExpectedExitCode=0
]]>
Expand Down
6 changes: 3 additions & 3 deletions src/tests/Common/helixpublishwitharcade.proj
Original file line number Diff line number Diff line change
Expand Up @@ -109,7 +109,7 @@
SuperPmiCollect=$(_SuperPmiCollect)
</_PropertiesToPass>

<_PropertiesToPass Condition="'$(TargetOS)' == 'browser' Or '$(TargetsAndroid)' == 'true' Or '$(TargetsiOS)' == 'true' Or '$(TargetsiOSSimulator)' == 'true' Or '$(TargetstvOS)' == 'true' Or '$(TargetstvOSSimulator)' == 'true'">
<_PropertiesToPass Condition="'$(TargetOS)' == 'browser' Or '$(TargetsAndroid)' == 'true' Or '$(TargetsAppleMobile)' == 'true'">
$(_PropertiesToPass);
IncludeDotNetCli=$(IncludeDotNetCli);
DotNetCliRuntime=$(DotNetCliRuntime);
Expand Down Expand Up @@ -623,7 +623,7 @@
<TimeoutPerTestInMilliseconds Condition=" '$(TimeoutPerTestInMinutes)' != '' ">$([System.TimeSpan]::FromMinutes($(TimeoutPerTestInMinutes)).TotalMilliseconds)</TimeoutPerTestInMilliseconds>
<WaitForWorkItemCompletion>true</WaitForWorkItemCompletion>
<_XUnitParallelMode>collections</_XUnitParallelMode>
<_XUnitParallelMode Condition=" '$(TargetsAndroid)' == 'true' or '$(TargetsiOSSimulator)' == 'true' or '$(TargetsiOS)' == 'true' or '$(TargetstvOSSimulator)' == 'true' or '$(TargetstvOS)' == 'true'">none</_XUnitParallelMode>
<_XUnitParallelMode Condition=" '$(TargetsAndroid)' == 'true' or '$(TargetsAppleMobile)' == 'true'">none</_XUnitParallelMode>
<_XUnitParallelMode Condition=" '$(LongRunningGCTests)' == 'true' ">none</_XUnitParallelMode>
<_XUnitParallelMode Condition=" '$(GcSimulatorTests)' == 'true' ">none</_XUnitParallelMode>
<XUnitRunnerArgs>-parallel $(_XUnitParallelMode) -nocolor -noshadow -xml testResults.xml</XUnitRunnerArgs>
Expand Down Expand Up @@ -761,7 +761,7 @@
<HelixCorrelationPayload Include="$(CoreRootDirectory)" />

<!-- Browser-Wasm and iOS platforms follow a very different workflow, which is currently out of scope of the Log Checker. It's not useful on any platform that uses xharness. -->
<HelixCorrelationPayload Include="$(XUnitLogCheckerDirectory)" Condition="'$(TargetsBrowser)' != 'true' and '$(TargetsiOS)' != 'true' and '$(TargetsiOSSimulator)' != 'true' and '$(TargetstvOS)' != 'true' and '$(TargetstvOSSimulator)' != 'true'" />
<HelixCorrelationPayload Include="$(XUnitLogCheckerDirectory)" Condition="'$(TargetsBrowser)' != 'true' and '$(TargetsAppleMobile)' != 'true'" />
<HelixCorrelationPayload Condition="'$(TestWrapperTargetsWindows)' == 'true'" Include="dotnet-sos">
<Destination>sos</Destination>
<Uri>https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet-tools/nuget/v3/flat2/dotnet-sos/$(DotnetSosVersion)/dotnet-sos.$(DotnetSosVersion).nupkg</Uri>
Expand Down
5 changes: 4 additions & 1 deletion src/tests/Interop/Swift/SwiftAbiStress/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,12 @@ include ("${CLR_INTEROP_TEST_ROOT}/Interop.cmake")

set(SOURCE SwiftAbiStress)

if (NOT SWIFT_COMPILER_TARGET AND CLR_CMAKE_TARGET_OSX)
if (NOT SWIFT_COMPILER_TARGET AND CLR_CMAKE_TARGET_APPLE)
set(SWIFT_PLATFORM "macosx")
set(SWIFT_PLATFORM_SUFFIX "")
if (NOT CMAKE_OSX_DEPLOYMENT_TARGET)
set(CMAKE_OSX_DEPLOYMENT_TARGET "12.0")
endif()
set(SWIFT_DEPLOYMENT_TARGET ${CMAKE_OSX_DEPLOYMENT_TARGET})
set(SWIFT_COMPILER_TARGET "${CMAKE_OSX_ARCHITECTURES}-apple-${SWIFT_PLATFORM}${SWIFT_DEPLOYMENT_TARGET}${SWIFT_PLATFORM_SUFFIX}")
endif()
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,12 @@ include ("${CLR_INTEROP_TEST_ROOT}/Interop.cmake")

set(SOURCE SwiftCallbackAbiStress)

if (NOT SWIFT_COMPILER_TARGET AND CLR_CMAKE_TARGET_OSX)
if (NOT SWIFT_COMPILER_TARGET AND CLR_CMAKE_TARGET_APPLE)
set(SWIFT_PLATFORM "macosx")
set(SWIFT_PLATFORM_SUFFIX "")
if (NOT CMAKE_OSX_DEPLOYMENT_TARGET)
set(CMAKE_OSX_DEPLOYMENT_TARGET "12.0")
endif()
set(SWIFT_DEPLOYMENT_TARGET ${CMAKE_OSX_DEPLOYMENT_TARGET})
set(SWIFT_COMPILER_TARGET "${CMAKE_OSX_ARCHITECTURES}-apple-${SWIFT_PLATFORM}${SWIFT_DEPLOYMENT_TARGET}${SWIFT_PLATFORM_SUFFIX}")
endif()
Expand Down
5 changes: 4 additions & 1 deletion src/tests/Interop/Swift/SwiftErrorHandling/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,12 @@ include ("${CLR_INTEROP_TEST_ROOT}/Interop.cmake")

set(SOURCE SwiftErrorHandling)

if (NOT SWIFT_COMPILER_TARGET AND CLR_CMAKE_TARGET_OSX)
if (NOT SWIFT_COMPILER_TARGET AND CLR_CMAKE_TARGET_APPLE)
set(SWIFT_PLATFORM "macosx")
set(SWIFT_PLATFORM_SUFFIX "")
if (NOT CMAKE_OSX_DEPLOYMENT_TARGET)
set(CMAKE_OSX_DEPLOYMENT_TARGET "12.0")
endif()
set(SWIFT_DEPLOYMENT_TARGET ${CMAKE_OSX_DEPLOYMENT_TARGET})
set(SWIFT_COMPILER_TARGET "${CMAKE_OSX_ARCHITECTURES}-apple-${SWIFT_PLATFORM}${SWIFT_DEPLOYMENT_TARGET}${SWIFT_PLATFORM_SUFFIX}")
endif()
Expand Down
5 changes: 4 additions & 1 deletion src/tests/Interop/Swift/SwiftInvalidCallConv/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,12 @@ include ("${CLR_INTEROP_TEST_ROOT}/Interop.cmake")

set(SOURCE SwiftInvalidCallConv)

if (NOT SWIFT_COMPILER_TARGET AND CLR_CMAKE_TARGET_OSX)
if (NOT SWIFT_COMPILER_TARGET AND CLR_CMAKE_TARGET_APPLE)
set(SWIFT_PLATFORM "macosx")
set(SWIFT_PLATFORM_SUFFIX "")
if (NOT CMAKE_OSX_DEPLOYMENT_TARGET)
set(CMAKE_OSX_DEPLOYMENT_TARGET "12.0")
endif()
set(SWIFT_DEPLOYMENT_TARGET ${CMAKE_OSX_DEPLOYMENT_TARGET})
set(SWIFT_COMPILER_TARGET "${CMAKE_OSX_ARCHITECTURES}-apple-${SWIFT_PLATFORM}${SWIFT_DEPLOYMENT_TARGET}${SWIFT_PLATFORM_SUFFIX}")
endif()
Expand Down
5 changes: 4 additions & 1 deletion src/tests/Interop/Swift/SwiftRetAbiStress/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,12 @@ include ("${CLR_INTEROP_TEST_ROOT}/Interop.cmake")

set(SOURCE SwiftRetAbiStress)

if (NOT SWIFT_COMPILER_TARGET AND CLR_CMAKE_TARGET_OSX)
if (NOT SWIFT_COMPILER_TARGET AND CLR_CMAKE_TARGET_APPLE)
set(SWIFT_PLATFORM "macosx")
set(SWIFT_PLATFORM_SUFFIX "")
if (NOT CMAKE_OSX_DEPLOYMENT_TARGET)
set(CMAKE_OSX_DEPLOYMENT_TARGET "12.0")
endif()
set(SWIFT_DEPLOYMENT_TARGET ${CMAKE_OSX_DEPLOYMENT_TARGET})
set(SWIFT_COMPILER_TARGET "${CMAKE_OSX_ARCHITECTURES}-apple-${SWIFT_PLATFORM}${SWIFT_DEPLOYMENT_TARGET}${SWIFT_PLATFORM_SUFFIX}")
endif()
Expand Down
5 changes: 4 additions & 1 deletion src/tests/Interop/Swift/SwiftSelfContext/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,12 @@ include ("${CLR_INTEROP_TEST_ROOT}/Interop.cmake")

set(SOURCE SwiftSelfContext)

if (NOT SWIFT_COMPILER_TARGET AND CLR_CMAKE_TARGET_OSX)
if (NOT SWIFT_COMPILER_TARGET AND CLR_CMAKE_TARGET_APPLE)
set(SWIFT_PLATFORM "macosx")
set(SWIFT_PLATFORM_SUFFIX "")
if (NOT CMAKE_OSX_DEPLOYMENT_TARGET)
set(CMAKE_OSX_DEPLOYMENT_TARGET "12.0")
endif()
set(SWIFT_DEPLOYMENT_TARGET ${CMAKE_OSX_DEPLOYMENT_TARGET})
set(SWIFT_COMPILER_TARGET "${CMAKE_OSX_ARCHITECTURES}-apple-${SWIFT_PLATFORM}${SWIFT_DEPLOYMENT_TARGET}${SWIFT_PLATFORM_SUFFIX}")
endif()
Expand Down
11 changes: 6 additions & 5 deletions src/tests/build.proj
Original file line number Diff line number Diff line change
Expand Up @@ -460,14 +460,15 @@

<!-- This target moves app bundles to the payload directory -->
<Target Name="AfterAppleBuild">
<!-- Apparently MSBuild cannot move directories and recursively copying a
a directory requires writing some sort of recursive traversal
logic yourself. -->
<!-- MSBuild cannot move directories or recursively copy content. -->
<ItemGroup>
<RecursiveCopyHack Include="$(AppBundlePath)/**/*.*" />
<BundleContent Include="$(AppBundlePath)/**/*.*" />
</ItemGroup>
<MakeDir Directories="$(FinalPath)" />
<Copy SourceFiles="@(RecursiveCopyHack)" DestinationFolder="$(FinalPath)/%(RecursiveDir)" />
<!-- MacCatalyst requires Contents/Info.plist and Contents/MacOS/ApplicationBinary -->
<Copy Condition="'$(TargetOS)' == 'maccatalyst' AND '%(BundleContent.Filename)%(BundleContent.Extension)' == 'Info.plist'" SourceFiles="@(BundleContent)" DestinationFolder="$(FinalPath)/Contents/%(RecursiveDir)" />
<Copy Condition="'$(TargetOS)' == 'maccatalyst' AND '%(BundleContent.Filename)%(BundleContent.Extension)' != 'Info.plist'" SourceFiles="@(BundleContent)" DestinationFolder="$(FinalPath)/Contents/MacOS/%(RecursiveDir)" />
<Copy Condition="'$(TargetOS)' != 'maccatalyst'" SourceFiles="@(BundleContent)" DestinationFolder="$(FinalPath)/%(RecursiveDir)" />
Comment on lines -470 to +471
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isn't Contents dir already contained in the BundleContent? (I am not sure why we need to special-case maccatalyst)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It should be already there, but I couldn't recursively copy content in msbuild from the build directory to the bundle. The RecursiveDir metadata for BundleContent is empty here. Please share any ideas on how to avoid "manual" copy.

It is in place to verify E2E tests, but I think we need to resolve this before merging.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The RecursiveDir metadata for BundleContent is empty here.

Yes, that is what I was confused about.

<RemoveDir Directories="$(AppBundlePath)" />
<Message Importance="High" Text="Final app: $(FinalPath)" />
</Target>
Expand Down
Loading