Source build runtime and emsdk workload manifests

[steveisok]

Background

In the dotnet SDK, there are baseline workload manifests that define the optional workloads that can be installed via dotnet workload install <workload-name>. Typically during workload install, the SDK will look to nuget.org for more up to date manifests and if found, will update. Each manifest is a package built out of the repository that defines them and then flows to the dotnet/installer repo to be bundled with the SDK. Up until this point, source build has not built the manifests and has manually injected them into the source built SDK.

For more info on workload manifests, consult the design doc

Goal

For preview 1, we should source build the workload manifests for emsdk and runtime. Specifically, microsoft.net.workload.emscripten and microsoft.net.workload.mono.toolchain. In the regular SDK, there are other workloads that support MAUI, but that is beyond the scope of source build for now.

Tasks

• Bring dotnet/emsdk into source build
• Build microsoft.net.workload.emscripten manifest in emsdk
• Build microsoft.net.workload.mono.toolchain manifest in runtime
• Make sure the manifests are picked up from the installer
• Exclude MAUI & Mobile manifests from the installer

[dotnet-issue-labeler]

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[steveisok]

/cc @tkapin @mmitche @MichaelSimons @lewing

[MichaelSimons]

@directhex - Do you have a status update on this? The workloads are the largest group of remaining prebuilts in Preview1.

[directhex]

It's related to dotnet/runtime#80920 which I just merged.

Building the runtime-based manifests is easy, and shouldn't need emsdk at all (it's just json in a nupkg)

[directhex]

WiP for emsdk manifest build dotnet/emsdk#274

[MichaelSimons]

@directhex - the emsdk manifest are now produced as part of source-build thanks to your changes. Do you have an update for when Exclude MAUI & Mobile manifests from the installer will be completed for preview 1? TIA

[directhex]

Michael! Just the man I wanted to check in with!

I added a property to Runtime, to allow building those manifests, ForceBuildMobileManifests. It works with a runtime build on the command line.

I've added that to the definition in repo-projects/runtime.proj

diff --git a/repo-projects/runtime.proj b/repo-projects/runtime.proj
index ddf7432315..888a54e291 100644
--- a/repo-projects/runtime.proj
+++ b/repo-projects/runtime.proj
@@ -24,6 +24,7 @@
     <BuildCommandArgs>$(BuildCommandArgs) /p:BaseOS=$(BaseOS)</BuildCommandArgs>
     <BuildCommandArgs>$(BuildCommandArgs) /p:SourceBuildNonPortable=true</BuildCommandArgs>
     <BuildCommandArgs>$(BuildCommandArgs) /p:UsingToolMicrosoftNetCompilers=false</BuildCommandArgs>
+    <BuildCommandArgs>$(BuildCommandArgs) /p:ForceBuildMobileManifests=true</BuildCommandArgs>
     <BuildCommand>$(StandardSourceBuildCommand) $(BuildCommandArgs)</BuildCommand>
   </PropertyGroup>

It seems to pass through in the source-build run:

  [13:09:29.03] Building 'runtime'
  Running command:
    /home/directhex/Projects/dotnet/src/runtime/build.sh  --ci --configuration Release --restore --build --pack --publish -bl /p:ArcadeBuildFromSource=true /p:CopyWipIntoInnerSourceBuildRepo=true /p:DotNetBuildOffline=true /p:CopySrcInsteadOfClone=true /p:DotNetPackageVersionPropsPath="/home/directhex/Projects/dotnet/artifacts/obj/x64/Release/PackageVersions.runtime.props" /p:AdditionalSourceBuiltNupkgCacheDir="/home/directhex/Projects/dotnet/artifacts/obj/x64/Release/blob-feed/packages/" /p:ReferencePackageNupkgCacheDir="/home/directhex/Projects/dotnet/prereqs/packages/reference/" /p:PreviouslySourceBuiltNupkgCacheDir="/home/directhex/Projects/dotnet/prereqs/packages/previously-source-built/" /p:SourceBuildUseMonoRuntime= /p:TargetRid=ubuntu.22.04-x64 /p:RuntimeOS=linux /p:BaseOS=linux /p:SourceBuildNonPortable=true /p:UsingToolMicrosoftNetCompilers=false /p:ForceBuildMobileManifests=true  /p:DotNetPackageVersionPropsPath=/home/directhex/Projects/dotnet/artifacts/obj/x64/Release/PackageVersions.runtime.props /p:DotNetBuildOffline=true

But then "something happens" in between my definition in runtime.proj, and the actual build logged in src/runtime/artifacts/source-build/seld/src/artifacts/sourcebuild.binlog:

Command line arguments = "/home/directhex/Projects/dotnet/.dotnet/sdk/8.0.100-alpha.1.23080.2/MSBuild.dll -maxcpucount -verbosity:m /m /nologo /clp:Summary /v:minimal /nr:false /p:TreatWarningsAsErrors=false /p:ContinuousIntegrationBuild=false /home/directhex/Projects/dotnet/artifacts/source-built-sdks/Microsoft.DotNet.Arcade.Sdk/tools/Build.proj /p:Configuration=Release /p:RepoRoot=/home/directhex/Projects/dotnet/src/runtime/artifacts/source-build/self/src/ /p:Restore=true /p:Build=true /p:ArcadeBuildFromSource=false /p:Rebuild=false /p:Test=false /p:Pack=false /p:IntegrationTest=false /p:PerformanceTest=false /p:Sign=false /p:Publish=false /p:BuildAllConfigurations=true /p:OutputRid=ubuntu.22.04-x64 /p:PortableBuild=false /p:TargetArchitecture=x64 /p:BuildArchitecture=x64 /p:CMakeArgs="" /p:NoPgoOptimize=true /p:KeepNativeSymbols=true /p:RuntimeOS=linux /p:OfficialBuildId=20230206.6 /p:ContinuousIntegrationBuild=true /p:BuildDebPackage=false /p:EnableNgenOptimization=false /p:AdditionalRuntimeIdentifierParent=linux /p:EnablePackageValidation=false /p:ApiCompatValidateAssemblies=false /p:ArcadeInnerBuildFromSource=true /p:DotNetBuildFromSource=true /p:RepoRoot=/home/directhex/Projects/dotnet/src/runtime/artifacts/source-build/self/src/ /p:ArtifactsDir=/home/directhex/Projects/dotnet/src/runtime/artifacts/source-build/self/src/artifacts/ /bl:/home/directhex/Projects/dotnet/src/runtime/artifacts/source-build/self/src/artifacts/sourcebuild.binlog /p:ContinuousIntegrationBuild=true /p:SourceBuildOutputDir=/home/directhex/Projects/dotnet/src/runtime/artifacts/source-build/ /p:SourceBuiltBlobFeedDir= /p:DotNetBuildOffline=true /p:DotNetPackageVersionPropsPath=/home/directhex/Projects/dotnet/artifacts/obj/x64/Release/PackageVersions.runtime.props"

So the command line gets reconstructed, and is skipping my added values?

Either way, runtime is not producing the manifests when I expect it to, when called by the top-level source-build build.sh

[MichaelSimons]

ForceBuildMobileManifests should be specified at the repo's SourceBuild.props. This means when you build runtime with the sourcebuild flag (e.g. ./build.sh -sb) it will build with this option. This will help catch source-build issues at the runtime level when PRs are made versus waiting it to flow into the VMR.

The properties specified in the repo-projects/runtime.proj don't automatically get passed along to the inner build as you noted. Ideally the repo-projects would have no/limited special repo settings.

[MichaelSimons]

You could easily validate the e2e in the VMR and either backport the change or if necessary, a patch is an option for Preview 1.

[directhex]

I'm going to mark this closed.

As best I can determine, it's doing the right and expected thing now:

Successfully installed workload(s) wasm-tools-net7

 directhex  /  tmp  ohboyihopethisworks  ./dotnet workload list

Installed Workload Id      Manifest Version                               Installation Source  
-----------------------------------------------------------------------------------------------
runtimes-windows-net6      8.0.0-preview.2.23107.1/8.0.100-preview.2      SDK 8.0.100-preview.2
wasm-tools-net7            8.0.0-preview.2.23107.1/8.0.100-preview.2      SDK 8.0.100-preview.2

[MichaelSimons]

@directhex - Please reference link to the new issue to source build the maui workloads as discussed in dotnet/installer#15485.

[directhex]

#3242