[nativeaot] Add Native AOT cross-build support for iOS-like platforms #88242

kotlarmilos · 2023-06-30T11:14:31Z

This PR adds support for the Native AOT cross-build targeting iOS-like platforms. When building for the target platforms, only the jitinterface component is installed, while clrjit is skipped since it is not required for the target platform. The jitinterface, aotsdk, and native libraries components are built for the target platforms, while jitinterface and clrjit components are built for the host as cross-components.

Additionally, this PR makes the cross build path to be a constant (/host), to allow both cross-arch and cross-os scenarios. Previously, the path was architecture-specific and did not support cross-os scenarios where the target and host architectures are the same.

The updated cross build path caused a lot of regressions and would require updating the tests to match the new path. The idea is to use the architecture-specific path in the cross-os scenarios as well.

It should unblock #87260 and #87773.

ghost · 2023-06-30T11:14:39Z

Tagging subscribers to this area: @agocke, @MichalStrehovsky, @jkotas
See info in area-owners.md if you want to be subscribed.

Issue Details

This PR adds support for the Native AOT cross-build targeting iOS-like platforms. When building for the target platforms, only the jitinterface component is installed, while clrjit is skipped since it is not required for the target platform. The jitinterface, aotsdk, and native libraries components are built for the target platforms, while jitinterface and clrjit components are built for the host as cross-components.

Additionally, this PR makes the cross build path to be a constant (/host), to allow both cross-arch and cross-os scenarios. Previously, the path was architecture-specific and did not support cross-os scenarios where the target and host architectures are the same.

It should unblock #87260 and #87773.

Author:	kotlarmilos
Assignees:	kotlarmilos
Labels:	`area-NativeAOT-coreclr`
Milestone:	8.0.0

jkotas · 2023-06-30T17:50:12Z

src/coreclr/build-runtime.sh

@@ -109,6 +109,9 @@ __ConfigTriplet="$__TargetOS.$__TargetArch.$__BuildType"
 if [[ "$__TargetOS" == "linux-bionic" ]]; then
    __ConfigTriplet="linux.$__TargetArch.$__BuildType"
 fi
+if [[ ! -z "$__OutputRIDOS" ]]; then
+    __ConfigTriplet="$__OutputRIDOS.$__TargetArch.$__BuildType"


Instead of introducing this override for output path, would it be better to keep the compiler in its default path and invoke it from there?

In that case, the default path for the cross-compiler is $(TargetOS).$(TargetArchitecture).$(Configuration)/$(BuildArchitecture), while the jitinterface and clrjit cross-components are generated in $(HostOS).$(TargetArchitecture).$(Configuration)/$(BuildArchitecture), and these components cannot be found.

runtime/src/coreclr/tools/aot/ILCompiler/ILCompiler.props

Line 107 in 36481bf

<CoreCLRArtifactsPath Condition="'$(CoreCLRArtifactsPath)' == ''">$(RuntimeBinDir)$(CrossHostArch)</CoreCLRArtifactsPath>

When building cross-components for cross-arch scenarios, the DCLR_CMAKE_TARGET_ARCH is used for the output path.

runtime/src/coreclr/build-runtime.sh

Lines 166 to 168 in 36481bf

if [[ "$__TargetArch" != "$__HostArch" ]]; then

__CMakeArgs="-DCLR_CMAKE_TARGET_ARCH=$__TargetArch $__CMakeArgs"

fi

One approach that I will try is to introduce the -hostos parameter along with -hostarch to indicate the target platform for the cross-components build. Another approach would be to adjust the components path in the _crossarch projects.

MichalStrehovsky

Looks good otherwise! Thanks for cleaning it up!

MichalStrehovsky · 2023-07-03T00:22:14Z

eng/Subsets.props

@@ -267,6 +267,22 @@
      Category="clr" />
  </ItemGroup>

+  <!-- Build the CoreCLR cross tools when we're doing a cross OS build for apple device targets -->


Could this be folded into the above Build the CoreCLR cross tools when we're doing a cross build block? We currently exclude it with '$(TargetsMobile)' != 'true' but it looks pretty much the same. The additional TargetOS and OutputRIDOS properties will probably not harm anything if we add them.

I think so. Initially, I didn't want to extend the cross-build for unsupported scenarios.

MichalStrehovsky · 2023-07-03T00:23:30Z

eng/Subsets.props

@@ -336,8 +332,8 @@
    <ProjectToBuild Include="$(CoreClrProjectRoot)tools\aot\ILCompiler\ILCompiler.csproj" Category="clr" Condition="'$(NativeAotSupported)' == 'true'" />
    <ProjectToBuild Include="$(CoreClrProjectRoot)nativeaot\BuildIntegration\BuildIntegration.proj" Category="clr" Condition="'$(NativeAotSupported)' == 'true'" />

-    <ProjectToBuild Condition="'$(NativeAotSupported)' == 'true' and ('$(CrossBuild)' == 'true' or '$(BuildArchitecture)' != '$(TargetArchitecture)')" Include="$(CoreClrProjectRoot)tools\aot\ILCompiler\ILCompiler_crossarch.csproj" Category="clr" />
-    <ProjectToBuild Condition="'$(TargetArchitecture)' != '$(BuildArchitecture)'" Include="$(CoreClrProjectRoot)tools\aot\crossgen2\crossgen2_crossarch.csproj" Category="clr" />
+    <ProjectToBuild Condition="'$(NativeAotSupported)' == 'true' and ('$(CrossBuild)' == 'true' or '$(BuildArchitecture)' != '$(TargetArchitecture)' or ('$(HostOS)' != '$(TargetOS)' and '$(TargetsAppleMobile)' == 'true'))" Include="$(CoreClrProjectRoot)tools\aot\ILCompiler\ILCompiler_crossarch.csproj" Category="clr" />


Do we need the '$(TargetsAppleMobile)' == 'true' condition here and below? It feels like extending it to HostOS != TargetOS should be general goodness.

eng/build.sh

MichalStrehovsky · 2023-07-03T00:24:49Z

eng/liveBuilds.targets

@@ -25,7 +25,7 @@
    <CoreCLRSharedFrameworkDir>$([MSBuild]::NormalizeDirectory('$(CoreCLRArtifactsPath)', 'sharedFramework'))</CoreCLRSharedFrameworkDir>
    <CoreCLRCrossgen2Dir>$([MSBuild]::NormalizeDirectory('$(CoreCLRArtifactsPath)', 'crossgen2'))</CoreCLRCrossgen2Dir>
    <CoreCLRILCompilerDir>$([MSBuild]::NormalizeDirectory('$(CoreCLRArtifactsPath)', 'ilc-published'))</CoreCLRILCompilerDir>
-    <CoreCLRCrossILCompilerDir Condition="'$(CrossBuild)' == 'true' or '$(BuildArchitecture)' != '$(TargetArchitecture)'">$([MSBuild]::NormalizeDirectory('$(CoreCLRArtifactsPath)', '$(BuildArchitecture)', 'ilc'))</CoreCLRCrossILCompilerDir>
+    <CoreCLRCrossILCompilerDir Condition="'$(CrossBuild)' == 'true' or '$(BuildArchitecture)' != '$(TargetArchitecture)' or ('$(HostOS)' != '$(TargetOS)' and '$(TargetsAppleMobile)' == 'true')">$([MSBuild]::NormalizeDirectory('$(CoreCLRArtifactsPath)', '$(BuildArchitecture)', 'ilc'))</CoreCLRCrossILCompilerDir>


Same question about the TargetsAppleMobile condition.

MichalStrehovsky · 2023-07-03T00:29:11Z

eng/liveBuilds.targets

@@ -71,7 +71,7 @@
      <CoreCLRSharedFrameworkDir>$([MSBuild]::NormalizeDirectory('$(CoreCLRArtifactsPath)','sharedFramework'))</CoreCLRSharedFrameworkDir>
      <CoreCLRSharedFrameworkPdbDir>$([MSBuild]::NormalizeDirectory('$(CoreCLRSharedFrameworkDir)','PDB'))</CoreCLRSharedFrameworkPdbDir>
      <CoreCLRCrossTargetComponentDir
-        Condition="'$(CoreCLRCrossTargetComponentDirName)' != ''">$([MSBuild]::NormalizeDirectory('$(CoreCLRArtifactsPath)','$(CoreCLRCrossTargetComponentDirName)','sharedFramework'))</CoreCLRCrossTargetComponentDir>
+        Condition="'$(CoreCLRCrossTargetComponentDirName)' != ''">$([MSBuild]::NormalizeDirectory('$(CoreCLRArtifactsPath)','$(BuildArchitecture)','sharedFramework'))</CoreCLRCrossTargetComponentDir>


Why do we need this change (I was not able to find a use of this property that looked relevant)?

Not needed, it was a temporary change.

MichalStrehovsky · 2023-07-03T00:35:37Z

src/coreclr/tools/aot/ILCompiler/ILCompiler.csproj

@@ -42,7 +42,7 @@
    </PropertyGroup>

    <ItemGroup>
-      <LinkerArg Include="--target=$(TargetTriple)" Condition="'$(TargetOS)' != 'osx' and '$(TargetTriple)' != ''" />
+      <LinkerArg Include="--target=$(TargetTriple)" Condition="'$(TargetOS)' != 'osx' and '$(TargetsAppleMobile)' != 'true' and '$(TargetTriple)' != ''" />


Would this work?

Suggested change

<LinkerArg Include="--target=$(TargetTriple)" Condition="'$(TargetOS)' != 'osx' and '$(TargetsAppleMobile)' != 'true' and '$(TargetTriple)' != ''" />

<LinkerArg Include="--target=$(TargetTriple)" Condition="'$(_IsApplePlatform)' != 'true' and '$(TargetTriple)' != ''" />

Co-authored-by: Michal Strehovský <MichalStrehovsky@users.noreply.github.com>

MichalStrehovsky

Looks good, thank you! Cc @jkoritzinsky

kotlarmilos · 2023-07-11T05:59:17Z

https://dev.azure.com/dnceng/internal/_build/results?buildId=2217910&view=results

kotlarmilos · 2023-07-11T08:00:52Z

eng/native/build-commons.sh

@@ -104,7 +104,7 @@ build_native()
            echo "Error: Unknown Android architecture $hostArch."
            exit 1
        fi
-    elif [[ "$__TargetOS" == iossimulator ]]; then


@akoeplinger it looks like the $__TargetOS was used by mistake instead of $targetOS, please check.

steveisok

LGTM!

Add Native AOT cross-build support for iOS-like platforms

0ddc81f

kotlarmilos added the area-NativeAOT-coreclr label Jun 30, 2023

kotlarmilos added this to the 8.0.0 milestone Jun 30, 2023

kotlarmilos requested review from steveisok, akoeplinger, jkoritzinsky, filipnavara, MichalStrehovsky, SamMonoRT and ivanpovazan June 30, 2023 11:14

kotlarmilos self-assigned this Jun 30, 2023

kotlarmilos requested a review from marek-safar as a code owner June 30, 2023 11:14

Use cross architecture path for cross-os builds

560dbf7

build-analysis bot mentioned this pull request Jun 30, 2023

Intermittent build failure in AfterSourceBuild: "Could not write state file" #76488

Open

jkotas reviewed Jun 30, 2023

View reviewed changes

MichalStrehovsky reviewed Jul 3, 2023

View reviewed changes

Merge branch 'dotnet:main' into feature/nativeaot-cross-build

3809ba8

build-analysis bot mentioned this pull request Jul 10, 2023

simpleruntimeeventvalidation test failing in CI #88499

Closed

kotlarmilos and others added 2 commits July 10, 2023 16:15

Add -hostos parameter for cross-components build

02f37c2

Update eng/build.sh

1276b53

Co-authored-by: Michal Strehovský <MichalStrehovsky@users.noreply.github.com>

runfoapp bot mentioned this pull request Jul 10, 2023

Infra improvements for Helix #68176

Closed

build-analysis bot mentioned this pull request Jul 10, 2023

Tracking issue for CI build timeouts #76454

Closed

MichalStrehovsky approved these changes Jul 11, 2023

View reviewed changes

kotlarmilos commented Jul 11, 2023

View reviewed changes

jkoritzinsky approved these changes Jul 11, 2023

View reviewed changes

steveisok approved these changes Jul 11, 2023

View reviewed changes

kotlarmilos merged commit 14e8b82 into dotnet:main Jul 11, 2023
165 of 170 checks passed

kotlarmilos mentioned this pull request Jul 25, 2023

Enable support for NativeAOT targeting iOS via opt-in feature #80905

Open

52 tasks

ghost locked as resolved and limited conversation to collaborators Aug 13, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[nativeaot] Add Native AOT cross-build support for iOS-like platforms #88242

[nativeaot] Add Native AOT cross-build support for iOS-like platforms #88242

kotlarmilos commented Jun 30, 2023 •

edited

Loading

ghost commented Jun 30, 2023

jkotas Jun 30, 2023

kotlarmilos Jul 10, 2023 •

edited

Loading

MichalStrehovsky left a comment

MichalStrehovsky Jul 3, 2023

kotlarmilos Jul 10, 2023

MichalStrehovsky Jul 3, 2023

MichalStrehovsky Jul 3, 2023

MichalStrehovsky Jul 3, 2023

kotlarmilos Jul 10, 2023

MichalStrehovsky Jul 3, 2023

MichalStrehovsky left a comment

kotlarmilos commented Jul 11, 2023

kotlarmilos Jul 11, 2023

steveisok left a comment

	if [[ "$__TargetArch" != "$__HostArch" ]]; then
	__CMakeArgs="-DCLR_CMAKE_TARGET_ARCH=$__TargetArch $__CMakeArgs"
	fi

	<LinkerArg Include="--target=$(TargetTriple)" Condition="'$(TargetOS)' != 'osx' and '$(TargetsAppleMobile)' != 'true' and '$(TargetTriple)' != ''" />
	<LinkerArg Include="--target=$(TargetTriple)" Condition="'$(_IsApplePlatform)' != 'true' and '$(TargetTriple)' != ''" />

[nativeaot] Add Native AOT cross-build support for iOS-like platforms #88242

[nativeaot] Add Native AOT cross-build support for iOS-like platforms #88242

Conversation

kotlarmilos commented Jun 30, 2023 • edited Loading

ghost commented Jun 30, 2023

Choose a reason for hiding this comment

kotlarmilos Jul 10, 2023 • edited Loading

Choose a reason for hiding this comment

MichalStrehovsky left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MichalStrehovsky left a comment

Choose a reason for hiding this comment

kotlarmilos commented Jul 11, 2023

Choose a reason for hiding this comment

steveisok left a comment

Choose a reason for hiding this comment

kotlarmilos commented Jun 30, 2023 •

edited

Loading

kotlarmilos Jul 10, 2023 •

edited

Loading