Add workloads build #47225

akoeplinger · 2025-03-03T18:50:43Z

This builds the emsdk and runtime workload manifests here instead of the original repos. It also adds the VS insertion .zip creation which is only built in the official build by default.

Closes dotnet/source-build#3904

marcpopMSFT · 2025-03-03T19:45:21Z

The goal of this is to simplify the implicit version management to one place, correct? With the VMR being used in 10, is there a simpler way to do this (since sdk and runtime will be built in the same repo)?

akoeplinger · 2025-03-03T23:24:09Z

Correct, it's both to simplify versioning and also because there is no good other way in the VMR due to join points. Not sure what you mean with simpler way?

PR Overview

This PR introduces updates to the CI and pipeline configurations to support workload builds while deprecating obsolete functionality.

Increase the CI build timeout and add a workload build parameter.
Add multiple target platforms for workload builds in the pipeline configuration.
Remove the OverrideWasmRuntimePackVersions task, which appears to be deprecated.

Reviewed Changes

File	Description
.vsts-ci.yml	Updated timeout and appended a workload build flag to build properties.
eng/pipelines/templates/stages/vmr-build.yml	Added new workload target platforms with potential naming inconsistencies.
src/Tasks/sdk-tasks/OverrideWasmRuntimePackVersions.cs	Removed obsolete task code possibly no longer needed.

Copilot reviewed 124 out of 124 changed files in this pull request and generated no comments.

Comments suppressed due to low confidence (3)

.vsts-ci.yml:122

Ensure that appending '/p:BuildWorkloads=true' to the build properties does not conflict with other MSBuild parameters, and that it integrates correctly with existing build configurations.

officialBuildProperties: $(_officialBuildProperties) /p:BuildWorkloads=true

eng/pipelines/templates/stages/vmr-build.yml:886

[nitpick] The platform name 'Wasi_Shortstack_wasm' uses a different capitalization compared to 'Browser_ShortStack_wasm'; consider standardizing the naming convention for clarity.

- Wasi_Shortstack_wasm

src/Tasks/sdk-tasks/OverrideWasmRuntimePackVersions.cs:1

The removal of OverrideWasmRuntimePackVersions.cs may impact dependent build steps if references to this task remain elsewhere; verify that all dependencies have been updated appropriately.

File removed

eng/Version.Details.xml

src/Workloads/VSInsertion/workloads.csproj

ViktorHofer · 2025-03-05T12:46:48Z

.vsts-ci.yml

@@ -113,13 +113,13 @@ extends:
          - powershell: New-Item -ItemType Directory -Path $(Build.SourcesDirectory)/artifacts/bin -Force
            displayName: Create artifacts/bin directory
          ${{ if and(eq(parameters.runTestBuild, false), ne(variables['Build.Reason'], 'PullRequest')) }}:
-            timeoutInMinutes: 90
+            timeoutInMinutes: 180


Do you have numbers on how much longer the build is now due to the addition of the workloads?

building the workloads themselves takes ~20mins, but signing is taking quite a lot more time. I believe we can optimize this since I see it signing all of the emsdk files again even though they are already signed.

Because it's unpacking the content from the MSIs? Is it actually signing the files or just unpacking and then checking to see they are signed and then skipping them?

I see them in the SigningRound* logs so I assume they're getting sent to ESRP. I haven't investigated yet why

And the source nupkgs we're using from runtime/EMSDK, those should already be signed as part of the runtime build, correct? Is it possible we're pulling down the unsigned packages?

The emsdk ones are always signed as part of its official build yes. For runtime they are signed either in it's current official build (when targetting a release branch) or in the VMR as part of one of the vertical builds.

So at least for emsdk it shouldn't sign the files again, ever.

Signtool is supposed to be quite good now at avoiding re-signing. But I would expect re-signing in a build off main, since the source emsdk bits are unsigned.

since the source emsdk bits are unsigned.

I changed that already since I expected this to be a problem but it didn't help. Need to dig deeper.

eng/Versions.props

joeloff · 2025-03-05T16:13:42Z

src/Layout/redist/targets/OverlaySdkOnLKG.targets

-      <WasmWorkloads Include="microsoft.net.workload.emscripten.net9" Version="$(EmscriptenWorkloadManifestVersion)/$(EmscriptenWorkloadFeatureBand)"/>
-    </ItemGroup>
-
-    <!-- Create a rollback file for installing workloads during the build  -->


Was the rollback JSON file generation moved?

yes it was removed, we don't need it anymore since the mono.toolchain/emscripten manifests are builtin to the sdk now

joeloff · 2025-03-06T17:16:55Z

So far this all looks good. I'm going to check out the PR locally and just get a feel for the dev build process before signing off in case more questions come up

joeloff · 2025-03-06T22:22:25Z

@akoeplinger I built the PR locally with -pack. I don't see the runtime and EMSDK manifest installers at all and they appear to be missing from the SDK installer.

When I look at artifacts\obj\redist-installer\Debug\WorkloadManifests.wxs, it's missing the runtime and EMSDK manifest installers. These form the baseline for SDK installs. I suspect because of the bundled manifest changes in the redist project, these may no longer be included:

        <PackageGroup Id="PG_WorkloadManifests">
            <MsiPackage SourceFile="C:\Users\joeloff\.nuget\packages\microsoft.net.sdk.android.manifest-10.0.100-preview.1.msi.x64\35.99.0-preview.1.140\data\f331437845925b4135e0726b180284de-x64.msi">
                <MsiProperty Name="DOTNETHOME" Value="[DOTNETHOME]" />
            </MsiPackage>
            <MsiPackage SourceFile="C:\Users\joeloff\.nuget\packages\microsoft.net.sdk.ios.manifest-10.0.100-preview.1.msi.x64\18.2.10322-net10-p1\data\14f56a757980cd4c1ec686afab1010d1-x64.msi">
                <MsiProperty Name="DOTNETHOME" Value="[DOTNETHOME]" />
            </MsiPackage>
            <MsiPackage SourceFile="C:\Users\joeloff\.nuget\packages\microsoft.net.sdk.maccatalyst.manifest-10.0.100-preview.1.msi.x64\18.2.10322-net10-p1\data\34241177f7938fc2a56f602ce51b5137-x64.msi">
                <MsiProperty Name="DOTNETHOME" Value="[DOTNETHOME]" />
            </MsiPackage>
            <MsiPackage SourceFile="C:\Users\joeloff\.nuget\packages\microsoft.net.sdk.macos.manifest-10.0.100-preview.1.msi.x64\15.2.10322-net10-p1\data\4a2028e00ad70923caf6c5e1331969ef-x64.msi">
                <MsiProperty Name="DOTNETHOME" Value="[DOTNETHOME]" />
            </MsiPackage>
            <MsiPackage SourceFile="C:\Users\joeloff\.nuget\packages\microsoft.net.sdk.maui.manifest-10.0.100-preview.1.msi.x64\10.0.0-preview.1.25122.6\data\4f0fe2a64d09849cb931c16a90ecade6-x64.msi">
                <MsiProperty Name="DOTNETHOME" Value="[DOTNETHOME]" />
            </MsiPackage>
            <MsiPackage SourceFile="C:\Users\joeloff\.nuget\packages\microsoft.net.sdk.tvos.manifest-10.0.100-preview.1.msi.x64\18.2.10322-net10-p1\data\7a7c674bd96d607fe7b50ea2e1f9998c-x64.msi">
                <MsiProperty Name="DOTNETHOME" Value="[DOTNETHOME]" />
            </MsiPackage>
            <MsiPackage SourceFile="C:\Users\joeloff\.nuget\packages\microsoft.net.sdk.aspire.manifest-8.0.100.msi.x64\8.2.2\data\7bdd89fa015679c6d39dbc54ab7f51c7-x64.msi">
                <MsiProperty Name="DOTNETHOME" Value="[DOTNETHOME]" />
            </MsiPackage>
        </PackageGroup>

Add workloads build

5e282c0

dotnet-issue-labeler bot added Area-Workloads untriaged labels Mar 3, 2025

build-analysis bot mentioned this pull request Mar 3, 2025

dotnet-watch tests failing #40006

Open

akoeplinger added 2 commits March 4, 2025 14:37

Fix wrong manifest name

5b28fb0

Fix sln

26108e9

akoeplinger marked this pull request as ready for review March 4, 2025 23:29

Copilot bot review requested due to automatic review settings March 4, 2025 23:29

akoeplinger requested review from a team as code owners March 4, 2025 23:29

Copilot AI reviewed Mar 4, 2025

View reviewed changes

akoeplinger requested a review from joeloff March 4, 2025 23:50

ViktorHofer reviewed Mar 5, 2025

View reviewed changes

eng/Version.Details.xml Outdated Show resolved Hide resolved

ViktorHofer reviewed Mar 5, 2025

View reviewed changes

src/Workloads/VSInsertion/workloads.csproj Outdated Show resolved Hide resolved

ViktorHofer reviewed Mar 5, 2025

View reviewed changes

joeloff reviewed Mar 5, 2025

View reviewed changes

eng/Versions.props Show resolved Hide resolved

joeloff reviewed Mar 5, 2025

View reviewed changes

akoeplinger added 3 commits March 6, 2025 12:41

Fix dependent job names

5c48269

Merge remote-tracking branch 'upstream/main' into workload-manifests

02391cd

Add manifest-packages.proj to sln

2448a5a

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add workloads build #47225

Add workloads build #47225

akoeplinger commented Mar 3, 2025 •

edited

Loading

marcpopMSFT commented Mar 3, 2025

akoeplinger commented Mar 3, 2025

ViktorHofer Mar 5, 2025

akoeplinger Mar 5, 2025

joeloff Mar 5, 2025

akoeplinger Mar 5, 2025

joeloff Mar 5, 2025

akoeplinger Mar 6, 2025 •

edited

Loading

mmitche Mar 7, 2025

akoeplinger Mar 8, 2025

joeloff Mar 5, 2025

akoeplinger Mar 5, 2025 •

edited

Loading

joeloff commented Mar 6, 2025

joeloff commented Mar 6, 2025

Add workloads build #47225

Are you sure you want to change the base?

Add workloads build #47225

Conversation

akoeplinger commented Mar 3, 2025 • edited Loading

marcpopMSFT commented Mar 3, 2025

akoeplinger commented Mar 3, 2025

Choose a reason for hiding this comment

PR Overview

Reviewed Changes

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

akoeplinger Mar 6, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

akoeplinger Mar 5, 2025 • edited Loading

Choose a reason for hiding this comment

joeloff commented Mar 6, 2025

joeloff commented Mar 6, 2025

akoeplinger commented Mar 3, 2025 •

edited

Loading

akoeplinger Mar 6, 2025 •

edited

Loading

akoeplinger Mar 5, 2025 •

edited

Loading