Use runtime packs for NativeAOT on .NET 9.0 or higher

[filipnavara]

Ref: dotnet/runtime#87060

[filipnavara]

I am not confident on how to validate this end-to-end aside from modifying my local SDK and hoping for the best. Any other ideas? /cc @akoeplinger

[am11]

One easy way is to use this script: https://gist.github.com/am11/afc2787bdecb9592663f4eae405d9116 (download at ~/update-daily-build.sh) apply sdk patches in ~/.dotnet9/sdk and then use dotnet9 publish etc. for testing. To refresh, run ~/update-daily-build.sh again. :)

[filipnavara]

@am11 That's essentially what I do. :-)

[am11]

I think this testing is enough and we can move this forward if local testing passed. After @jtschuster's dotnet/runtime#96858 goes in, source build will be able to validate more interesting runtimepack scenarios (for linux and freebsd etc.)

[filipnavara]

I'm hesitant to call it "ready" unless someone else confirms that it actually works on their machine (tm). It requires a .NET 9 SDK setup like instrumented above, and then running dotnet publish for any AOT app.

[am11]

Just tested on osx-arm64 (host), linux-musl-arm64 (container) and win-x64 (host); seems to work. 👍

[am11]

@filipnavara, a separate / existing but related issue: I just noticed that from runtime pack, we expect it to be present in the installation directory, such as ~/.dotnet9/packs, right? That's not happening for ilcompiler or crossgen2 because both seem to be missing entries corresponding to the corehost ones: https://github.com/dotnet/installer/blob/main/src/redist/targets/GenerateLayout.targets#L159 in that file. Otherwise, nuget just downloads it as a package under the ~/.nuget directory, i.e no special treatment which defeats the purpose of known packs.

[filipnavara]

Otherwise, nuget just downloads it as a package under the ~/.nuget directory, i.e no special treatment which defeats the purpose of known packs.

Wasn't it always working this way? I assumed it did for NativeAOT cross-compilation on .NET 8, and for macOS platform. Similarly for other platforms the ILCompiler package was not part of the bundle and that's where the "runtime pack" files were located.

[am11]

Wasn't it always working this way?

Pretty sure it was. What I understand from known packs is that it'd be available as a preloaded packages in the binary bundle (installer, zip, tar, whatever installer produces); otherwise the first time it will download the workload, it will store those packs to the same location where apphost packs are. But it does not seem to be the case for crossgen2 (for a longer time) and ilc (relatively recently). I checked with ls /usr/local/share/dotnet/packs on macOS, and there seems to be all kinds of packs available (maui, apphost etc.), except for crossgen2/ilc. Unless this is intentional, it seems like we are missing something for crossgen2 and ilc. I grep'd the sources in sdk and installer repos, and GenerateLayout.targets seems to be the relevant place.

e.g. if we clear the nuget caches dotnet nuget locals all --clear and republish the app for host architecture with PublishReadyToRun or PublishAOT flags, it will re-download the packages, which does not happen for the real known packs in $DOTNET_ROOT/packs directory.

[akoeplinger]

Known packs do not need to ship with the SDK, e.g. the normal desktop runtime packs aren't there either and will be fetched from nuget the first time they're used. If you install workloads it'll download packages into the packs folder, but that is not a requirement, just an optimization (e.g. we needed to be able to use workloads offline).

[am11]

that is not a requirement, just an optimization (e.g. we needed to be able to use workloads offline)

That's exactly what I was hoping to achieve from crossgen2/ilc known packs. Otherwise downloading these nugets separately in CI container builds is same pain as before, so I'll continue to mount ~/.nuget directory there.. :)

[akoeplinger]

@MichalStrehovsky any concerns from your side?

[MichalStrehovsky]

I don't think this will work in a .props file. This is so early that we don't know the project has PublishAot in it. It is so early that we probably don't even have _TargetFrameworkVersionWithoutV yet.

[MichalStrehovsky]

I don't think this will work in a .props file. This is so early that we don't know the project has PublishAot in it. It is so early that we probably don't even have _TargetFrameworkVersionWithoutV yet.

Tried this locally and it only does something if I do dotnet publish -bl -p:PublishAot=true -p:_TargetFrameworkVersionWithoutV=9.0. And the thing it does is not good:

    D:\test\test.csproj : error NU1101: Unable to find package Microsoft.NETCore.App.Runtime.NativeAOT.win-x64. No packages exist with this id in source(s): dotnet9, Microsoft Visual Studio Offline Packages, NET8, nuget.org [D:\test\test.csproj]

We don't seem to have the Windows packages published yet: https://dnceng.visualstudio.com/public/_artifacts/feed/dotnet9

[filipnavara]

This is so early that we don't know the project has PublishAot in it. It is so early that we probably don't even have _TargetFrameworkVersionWithoutV yet.

Bummer. I agree about PublishAot, it can probably be left out of the condition. _TargetFrameworkVersionWithoutV is used in the same file quite extensively though.

[MichalStrehovsky]

_TargetFrameworkVersionWithoutV is used in the same file quite extensively though.

I would be surprised if they work. When I tried this locally, it didn't try to use the runtime pack unless I also specified a value for _TargetFrameworkVersionWithoutV. Either something is wrong with the logic in this PR, or this is simply not available here. I only see it defined in .targets.

[MichalStrehovsky]

We don't seem to have the Windows packages published yet: https://dnceng.visualstudio.com/public/_artifacts/feed/dotnet9

Fix in dotnet/runtime#97023

[akoeplinger]

All the existing usages of _TargetFrameworkVersionWithoutV are in Conditions on Item/ItemGroup so maybe the msbuild evaluation order is different for those. You can try moving the checks into the condition on the FrameworkReference.

[filipnavara]

I'm sorry I didn't get back to this. I may find some free cycles to try again once .NET 9 Preview 1 ships with the necessary packages.

[filipnavara]

Side note: We have incorrect runtime.json in Microsoft.DotNet.ILCompiler that is missing entries for new platforms (and linux-bionic).

Nevermind, that's actually fine.

[MichalStrehovsky]

I'm playing with alternatives here. Below is an alternative approach. We'd delete all the PublishAotUsingRuntimePack handling in the runtime repo before doing the change here (only leave the PublishAotUsingRuntimePack==true codepaths in runtime repo). Once it comes over here, apply this diff.

One significant issue with this general approach is that FrameworkReference is conditioned on '$(_IsPublishing)' == 'true'. It's conditioned like that for good reasons. But this totally breaks everyone who does dotnet build /t:Publish (people who do this want to watch the world burn I guess). We'll need to make this error out with a sensible message. I don't know if the sensible message should include advice to set _IsPublishing to true. I think we have a bunch of native aot testing in this very repo that does /t:Publish :/.

diff --git a/src/Tasks/Microsoft.NET.Build.Tasks/targets/Microsoft.NET.Sdk.DefaultItems.props b/src/Tasks/Microsoft.NET.Build.Tasks/targets/Microsoft.NET.Sdk.DefaultItems.props
index 038026fdf5..05f0b41f13 100644
--- a/src/Tasks/Microsoft.NET.Build.Tasks/targets/Microsoft.NET.Sdk.DefaultItems.props
+++ b/src/Tasks/Microsoft.NET.Build.Tasks/targets/Microsoft.NET.Sdk.DefaultItems.props
@@ -83,10 +83,14 @@ Copyright (c) .NET Foundation. All rights reserved.
                         RuntimePackLabels="Mono"
                         Condition="'$(UseMonoRuntime)' == 'true' And ('$(_TargetFrameworkVersionWithoutV)' != '') And ('$(_TargetFrameworkVersionWithoutV)' >= '6.0')" />
 
-    <!-- Allow opt-in to NativeAOT runtime pack for .NET 8.0 or higher -->
+    <!-- Allow opt-in to NativeAOT runtime pack for .NET 8.0 or higher and default to it for .NET 9.0 or higher -->
     <FrameworkReference Update="Microsoft.NETCore.App"
                         RuntimePackLabels="NativeAOT"
-                        Condition="'$(_IsPublishing)' == 'true' and '$(PublishAotUsingRuntimePack)' == 'true' And ('$(_TargetFrameworkVersionWithoutV)' != '') And ('$(_TargetFrameworkVersionWithoutV)' >= '8.0')" />
+                        Condition="'$(PublishAot)' == 'true' And '$(_IsPublishing)' == 'true' And '$(_TargetFrameworkVersionWithoutV)' != ''
+                                   And (
+                                     ('$(PublishAotUsingRuntimePack)' == 'true' And '$(_TargetFrameworkVersionWithoutV)' >= '8.0')
+                                     Or
+                                     ('$(_TargetFrameworkVersionWithoutV)' >= '9.0'))" />
 
   </ItemGroup>
 
diff --git a/src/Tasks/Microsoft.NET.Build.Tasks/targets/Microsoft.NET.Sdk.FrameworkReferenceResolution.targets b/src/Tasks/Microsoft.NET.Build.Tasks/targets/Microsoft.NET.Sdk.FrameworkReferenceResolution.targets
index 63be500baa..e0fd0be66d 100644
--- a/src/Tasks/Microsoft.NET.Build.Tasks/targets/Microsoft.NET.Sdk.FrameworkReferenceResolution.targets
+++ b/src/Tasks/Microsoft.NET.Build.Tasks/targets/Microsoft.NET.Sdk.FrameworkReferenceResolution.targets
@@ -77,6 +77,7 @@ Copyright (c) .NET Foundation. All rights reserved.
     <PropertyGroup>
       <EnableTargetingPackDownload Condition="'$(EnableTargetingPackDownload)' == ''">true</EnableTargetingPackDownload>
       <EnableRuntimePackDownload Condition="'$(EnableRuntimePackDownload)' == ''">true</EnableRuntimePackDownload>
+      <_UsingNativeAotRuntimePack Condition="'%(FrameworkReference.Identity)' ==  'Microsoft.NETCore.App' And '%(FrameworkReference.RuntimePackLabels)' == 'NativeAOT'">true</_UsingNativeAotRuntimePack>
     </PropertyGroup>
 
     <PropertyGroup>
@@ -116,7 +117,7 @@ Copyright (c) .NET Foundation. All rights reserved.
                                 FirstTargetFrameworkVersionToSupportSingleFileAnalyzer="$(_FirstTargetFrameworkVersionToSupportSingleFileAnalyzer)"
                                 SilenceEnableSingleFileAnalyzerUnsupportedWarning="$(_SilenceEnableSingleFileAnalyzerUnsupportedWarning)"
                                 MinNonEolTargetFrameworkForSingleFile="$(_MinNonEolTargetFrameworkForSingleFile)"
-                                AotUseKnownRuntimePackForTarget="$(PublishAotUsingRuntimePack)"
+                                AotUseKnownRuntimePackForTarget="$(_UsingNativeAotRuntimePack)"
                                 RuntimeIdentifier="$(RuntimeIdentifier)"
                                 RuntimeIdentifiers="$(RuntimeIdentifiers)"
                                 RuntimeFrameworkVersion="$(RuntimeFrameworkVersion)"

[ivanpovazan]

@MichalStrehovsky should we try the alternative approach you proposed or do you have any other thoughts on how to proceed with this?

[MichalStrehovsky]

@MichalStrehovsky should we try the alternative approach you proposed or do you have any other thoughts on how to proceed with this?

That approach doesn't really work because it breaks people who run dotnet build /t:Publish (they will have to run dotnet build /t:Publish /p:_IsPublishing=true to fix themselves, which is suboptimal)

We have an email thread with some SDK folks that is failing to get traction for months, I'll add you to it.