Official build failure trying to sign Microsoft.NET.Sdk.WebAssembly.Pack #84742

sbomer · 2023-04-12T23:53:34Z

Official builds started failing after #84082. Link to build: https://dev.azure.com/dnceng/internal/_build/results?buildId=2156574&view=logs&j=4d50a8bf-a143-51c7-5cc8-defff437e23b&t=24058bbf-eb7e-5a58-7620-a06c45604c2a

The publish stage step Download Intermediate Artifacts says it downloaded the package:

Downloaded IntermediateArtifacts/MonoRuntimePacks/Shipping/Microsoft.NET.Sdk.WebAssembly.Pack.8.0.0-preview.4.23212.5.nupkg to D:\a\_work\1\s\artifacts\PackageDownload\IntermediateArtifacts\MonoRuntimePacks\Shipping\Microsoft.NET.Sdk.WebAssembly.Pack.8.0.0-preview.4.23212.5.nupkg

But the next step Prepare artifacts and upload to build fails:

D:\a\_work\1\s\src\installer\prepare-artifacts.proj(146,5): error : Item to sign 'Microsoft.NET.Sdk.WebAssembly.Pack.8.0.0-preview.4.23212.5.nupkg' was not found in the artifacts
##[error]src\installer\prepare-artifacts.proj(146,5): error : (NETCORE_ENGINEERING_TELEMETRY=Build) Item to sign 'Microsoft.NET.Sdk.WebAssembly.Pack.8.0.0-preview.4.23212.5.nupkg' was not found in the artifacts
D:\a\_work\1\s\src\installer\prepare-artifacts.proj(146,5): error : Item to sign 'Microsoft.NET.Sdk.WebAssembly.Pack.8.0.0-preview.4.23212.5.symbols.nupkg' was not found in the artifacts
##[error]src\installer\prepare-artifacts.proj(146,5): error : (NETCORE_ENGINEERING_TELEMETRY=Build) Item to sign 'Microsoft.NET.Sdk.WebAssembly.Pack.8.0.0-preview.4.23212.5.symbols.nupkg' was not found in the artifacts
  Writing build manifest file 'D:\a\_work\1\s\artifacts\log\Release\AssetManifest/Manifest.xml'...

The failure comes from arcade here:
https://github.com/dotnet/arcade/blob/5ec6af6931967583ebedeeccc85c25769de902bc/src/Microsoft.DotNet.Build.Tasks.Feed/src/SigningInformationModelFactory.cs#L54-L60

I believe the PushToAzureDevOpsArtifacts invocation here:

runtime/src/installer/prepare-artifacts.proj

Lines 146 to 163 in 887c043

    
           <PushToAzureDevOpsArtifacts 
        
             AzureDevOpsCollectionUri="$(SYSTEM_TEAMFOUNDATIONCOLLECTIONURI)" 
        
             AzureDevOpsProject="$(SYSTEM_TEAMPROJECT)" 
        
             AzureDevOpsBuildId="$(BUILD_BUILDID)" 
        
             ItemsToSign="@(ItemsToSignPostBuild)" 
        
             StrongNameSignInfo="@(StrongNameSignInfo)" 
        
             FileSignInfo="@(FileSignInfo)" 
        
             FileExtensionSignInfo="@(FileExtensionSignInfo)" 
        
             ItemsToPush="@(ItemsToPush)" 
        
             ManifestBuildData="@(ManifestBuildData)" 
        
             ManifestRepoUri="$(BUILD_REPOSITORY_NAME)" 
        
             ManifestBranch="$(BUILD_SOURCEBRANCH)" 
        
             ManifestBuildId="$(BUILD_BUILDNUMBER)" 
        
             ManifestCommit="$(BUILD_SOURCEVERSION)" 
        
             IsStableBuild="$(IsStableBuild)" 
        
             AssetManifestPath="$(AssetManifestFile)" 
        
             AssetsTemporaryDirectory="$(TempWorkingDir)" 
        
             PublishingVersion ="3" />

includes the new package in ItemsToSign (which gets populated from the PackageDownload directory mentioned earlier), but not in ItemsToPush, and this is what causes the failure. I think the fix would be to add the new package to NupkgToPublishFile somewhere in this logic (I'm assuming the package should get published):

runtime/src/installer/prepare-artifacts.proj

Lines 188 to 259 in 887c043

    
                 <DownloadedArtifactFile Include="$(DownloadDirectory)**" /> 
        
                 <Error 
        
                   Condition="'@(DownloadedArtifactFile)' == ''" 
        
                   Text="No downloaded artifacts found." /> 
        
                 <DownloadedSymbolNupkgFile Include="$(DownloadDirectory)**\*.symbols.nupkg" /> 
        
                 <DownloadedWixPdbFile Include="$(DownloadDirectory)**\*.wixpdb" /> 
        
                 <DownloadedWorkloadsVSInsertionFile Include="$(DownloadDirectory)*\workloads-vs\**\*" /> 
        
                 <DownloadedNupkgFile 
        
                   Include="$(DownloadDirectory)**\*.nupkg" 
        
                   Exclude="@(DownloadedSymbolNupkgFile)" /> 
        
                 <!-- Add files that are not affected by filtering. --> 
        
                 <UploadToBlobStorageFile 
        
                   Include="@(DownloadedArtifactFile)" 
        
                   Exclude="@(DownloadedSymbolNupkgFile);@(DownloadedNupkgFile);@(DownloadedWixPdbFile);@(DownloadedWorkloadsVSInsertionFile)" /> 
        
                 <!-- 
        
                   Filter out the RID-specific (Runtime) nupkgs and RID-agnostic nupkgs. RID-specific packages 
        
                   are published from every job. RID-agnostic nupkgs are built with the same ID/version by 
        
                   every job, so one specific job's outputs must be picked to sign and publish. 
        
                 --> 
        
                 <!-- RID-specific framework packs. --> 
        
                 <RuntimeNupkgFile 
        
                   Include=" 
        
                     $(DownloadDirectory)**\Microsoft.*.Runtime.*.nupkg; 
        
                     $(DownloadDirectory)**\Microsoft.*.App.Host.*.nupkg; 
        
                     $(DownloadDirectory)**\Microsoft.*.App.Crossgen2.*.nupkg" 
        
                   Exclude="@(DownloadedSymbolNupkgFile)" /> 
        
                 <!-- VS insertion packages, carrying RID-specific installers. --> 
        
                 <RuntimeNupkgFile 
        
                   Include="$(DownloadDirectory)**\VS.Redist.Common.*.nupkg" 
        
                   Exclude="@(DownloadedSymbolNupkgFile)" /> 
        
                 <!-- 
        
                   Workloads VS insertion artifacts produced by src/workloads/workloads.csproj. Only grab 
        
                   the zip artifacts as they're grouped by SDK feature band which correlates with specific VS versions. 
        
                 --> 
        
                 <WorkloadsVSInsertionFile 
        
                   Include=" 
        
                     $(DownloadDirectory)*\workloads-vs\**\*.zip"/> 
        
                 <!-- 
        
                   Runtime packages associated with some identity packages. Need to exclude "runtime.native.*" 
        
                   because Libraries produces some "runtime.native.Foo" packages with 
        
                   "runtime.<rid>.runtime.native.Foo" identity packages. 
        
                 --> 
        
                 <RuntimeNupkgFile 
        
                   Include="$(DownloadDirectory)**\runtime.*.nupkg" 
        
                   Exclude=" 
        
                     $(DownloadDirectory)**\runtime.native.*.nupkg; 
        
                     @(DownloadedSymbolNupkgFile)" /> 
        
                 <!-- 
        
                   Packages that aren't matched above as RID-specific are considered RID-agnostic. Also include 
        
                   the AllConfigurations packages from the Libraries build. 
        
                 --> 
        
                 <RidAgnosticNupkgToPublishFile 
        
                   Include=" 
        
                     $(DownloadDirectory)**\Microsoft.NET.Workload.Mono.Toolchain.*Manifest-*.nupkg; 
        
                     $(DownloadDirectory)*\$(PublishRidAgnosticPackagesFromPlatform)\**\*.nupkg; 
        
                     $(DownloadDirectory)*\*AllConfigurations\**\*.nupkg" 
        
                   Exclude="@(RuntimeNupkgFile);@(DownloadedSymbolNupkgFile)" /> 
        
                 <TransportPackagesToPublishFile 
        
                   Include="$(DownloadDirectory)**\*Transport*.nupkg" 
        
                   Exclude="@(RuntimeNupkgFile);@(RidAgnosticNupkgToPublishFile);@(DownloadedSymbolNupkgFile)" /> 
        
                 <NupkgToPublishFile Include="@(RuntimeNupkgFile);@(RidAgnosticNupkgToPublishFile);@(TransportPackagesToPublishFile)" />

@maraf @ViktorHofer

The text was updated successfully, but these errors were encountered:

lewing · 2023-04-12T23:59:27Z

We can revert this to unblock if needed

lewing · 2023-04-13T00:07:17Z

@joeloff any thoughts on this one?

lewing · 2023-04-13T00:26:16Z

opened #84743 while we look into this

ghost · 2023-04-13T00:40:15Z

Tagging subscribers to 'arch-wasm': @lewing
See info in area-owners.md if you want to be subscribed.

Issue Details

Official builds started failing after #84082. Link to build: https://dev.azure.com/dnceng/internal/_build/results?buildId=2156574&view=logs&j=4d50a8bf-a143-51c7-5cc8-defff437e23b&t=24058bbf-eb7e-5a58-7620-a06c45604c2a

The publish stage step Download Intermediate Artifacts says it downloaded the package:

Downloaded IntermediateArtifacts/MonoRuntimePacks/Shipping/Microsoft.NET.Sdk.WebAssembly.Pack.8.0.0-preview.4.23212.5.nupkg to D:\a\_work\1\s\artifacts\PackageDownload\IntermediateArtifacts\MonoRuntimePacks\Shipping\Microsoft.NET.Sdk.WebAssembly.Pack.8.0.0-preview.4.23212.5.nupkg

But the next step Prepare artifacts and upload to build fails:

D:\a\_work\1\s\src\installer\prepare-artifacts.proj(146,5): error : Item to sign 'Microsoft.NET.Sdk.WebAssembly.Pack.8.0.0-preview.4.23212.5.nupkg' was not found in the artifacts
##[error]src\installer\prepare-artifacts.proj(146,5): error : (NETCORE_ENGINEERING_TELEMETRY=Build) Item to sign 'Microsoft.NET.Sdk.WebAssembly.Pack.8.0.0-preview.4.23212.5.nupkg' was not found in the artifacts
D:\a\_work\1\s\src\installer\prepare-artifacts.proj(146,5): error : Item to sign 'Microsoft.NET.Sdk.WebAssembly.Pack.8.0.0-preview.4.23212.5.symbols.nupkg' was not found in the artifacts
##[error]src\installer\prepare-artifacts.proj(146,5): error : (NETCORE_ENGINEERING_TELEMETRY=Build) Item to sign 'Microsoft.NET.Sdk.WebAssembly.Pack.8.0.0-preview.4.23212.5.symbols.nupkg' was not found in the artifacts
  Writing build manifest file 'D:\a\_work\1\s\artifacts\log\Release\AssetManifest/Manifest.xml'...

The failure comes from arcade here:
https://github.com/dotnet/arcade/blob/5ec6af6931967583ebedeeccc85c25769de902bc/src/Microsoft.DotNet.Build.Tasks.Feed/src/SigningInformationModelFactory.cs#L54-L60

I believe the PushToAzureDevOpsArtifacts invocation here:

runtime/src/installer/prepare-artifacts.proj

Lines 146 to 163 in 887c043

    
           <PushToAzureDevOpsArtifacts 
        
             AzureDevOpsCollectionUri="$(SYSTEM_TEAMFOUNDATIONCOLLECTIONURI)" 
        
             AzureDevOpsProject="$(SYSTEM_TEAMPROJECT)" 
        
             AzureDevOpsBuildId="$(BUILD_BUILDID)" 
        
             ItemsToSign="@(ItemsToSignPostBuild)" 
        
             StrongNameSignInfo="@(StrongNameSignInfo)" 
        
             FileSignInfo="@(FileSignInfo)" 
        
             FileExtensionSignInfo="@(FileExtensionSignInfo)" 
        
             ItemsToPush="@(ItemsToPush)" 
        
             ManifestBuildData="@(ManifestBuildData)" 
        
             ManifestRepoUri="$(BUILD_REPOSITORY_NAME)" 
        
             ManifestBranch="$(BUILD_SOURCEBRANCH)" 
        
             ManifestBuildId="$(BUILD_BUILDNUMBER)" 
        
             ManifestCommit="$(BUILD_SOURCEVERSION)" 
        
             IsStableBuild="$(IsStableBuild)" 
        
             AssetManifestPath="$(AssetManifestFile)" 
        
             AssetsTemporaryDirectory="$(TempWorkingDir)" 
        
             PublishingVersion ="3" />

includes the new package in ItemsToSign (which gets populated from the PackageDownload directory mentioned earlier), but not in ItemsToPush, and this is what causes the failure. I think the fix would be to add the new package to NupkgToPublishFile somewhere in this logic (I'm assuming the package should get published):

runtime/src/installer/prepare-artifacts.proj

Lines 188 to 259 in 887c043

    
                 <DownloadedArtifactFile Include="$(DownloadDirectory)**" /> 
        
                 <Error 
        
                   Condition="'@(DownloadedArtifactFile)' == ''" 
        
                   Text="No downloaded artifacts found." /> 
        
                 <DownloadedSymbolNupkgFile Include="$(DownloadDirectory)**\*.symbols.nupkg" /> 
        
                 <DownloadedWixPdbFile Include="$(DownloadDirectory)**\*.wixpdb" /> 
        
                 <DownloadedWorkloadsVSInsertionFile Include="$(DownloadDirectory)*\workloads-vs\**\*" /> 
        
                 <DownloadedNupkgFile 
        
                   Include="$(DownloadDirectory)**\*.nupkg" 
        
                   Exclude="@(DownloadedSymbolNupkgFile)" /> 
        
                 <!-- Add files that are not affected by filtering. --> 
        
                 <UploadToBlobStorageFile 
        
                   Include="@(DownloadedArtifactFile)" 
        
                   Exclude="@(DownloadedSymbolNupkgFile);@(DownloadedNupkgFile);@(DownloadedWixPdbFile);@(DownloadedWorkloadsVSInsertionFile)" /> 
        
                 <!-- 
        
                   Filter out the RID-specific (Runtime) nupkgs and RID-agnostic nupkgs. RID-specific packages 
        
                   are published from every job. RID-agnostic nupkgs are built with the same ID/version by 
        
                   every job, so one specific job's outputs must be picked to sign and publish. 
        
                 --> 
        
                 <!-- RID-specific framework packs. --> 
        
                 <RuntimeNupkgFile 
        
                   Include=" 
        
                     $(DownloadDirectory)**\Microsoft.*.Runtime.*.nupkg; 
        
                     $(DownloadDirectory)**\Microsoft.*.App.Host.*.nupkg; 
        
                     $(DownloadDirectory)**\Microsoft.*.App.Crossgen2.*.nupkg" 
        
                   Exclude="@(DownloadedSymbolNupkgFile)" /> 
        
                 <!-- VS insertion packages, carrying RID-specific installers. --> 
        
                 <RuntimeNupkgFile 
        
                   Include="$(DownloadDirectory)**\VS.Redist.Common.*.nupkg" 
        
                   Exclude="@(DownloadedSymbolNupkgFile)" /> 
        
                 <!-- 
        
                   Workloads VS insertion artifacts produced by src/workloads/workloads.csproj. Only grab 
        
                   the zip artifacts as they're grouped by SDK feature band which correlates with specific VS versions. 
        
                 --> 
        
                 <WorkloadsVSInsertionFile 
        
                   Include=" 
        
                     $(DownloadDirectory)*\workloads-vs\**\*.zip"/> 
        
                 <!-- 
        
                   Runtime packages associated with some identity packages. Need to exclude "runtime.native.*" 
        
                   because Libraries produces some "runtime.native.Foo" packages with 
        
                   "runtime.<rid>.runtime.native.Foo" identity packages. 
        
                 --> 
        
                 <RuntimeNupkgFile 
        
                   Include="$(DownloadDirectory)**\runtime.*.nupkg" 
        
                   Exclude=" 
        
                     $(DownloadDirectory)**\runtime.native.*.nupkg; 
        
                     @(DownloadedSymbolNupkgFile)" /> 
        
                 <!-- 
        
                   Packages that aren't matched above as RID-specific are considered RID-agnostic. Also include 
        
                   the AllConfigurations packages from the Libraries build. 
        
                 --> 
        
                 <RidAgnosticNupkgToPublishFile 
        
                   Include=" 
        
                     $(DownloadDirectory)**\Microsoft.NET.Workload.Mono.Toolchain.*Manifest-*.nupkg; 
        
                     $(DownloadDirectory)*\$(PublishRidAgnosticPackagesFromPlatform)\**\*.nupkg; 
        
                     $(DownloadDirectory)*\*AllConfigurations\**\*.nupkg" 
        
                   Exclude="@(RuntimeNupkgFile);@(DownloadedSymbolNupkgFile)" /> 
        
                 <TransportPackagesToPublishFile 
        
                   Include="$(DownloadDirectory)**\*Transport*.nupkg" 
        
                   Exclude="@(RuntimeNupkgFile);@(RidAgnosticNupkgToPublishFile);@(DownloadedSymbolNupkgFile)" /> 
        
                 <NupkgToPublishFile Include="@(RuntimeNupkgFile);@(RidAgnosticNupkgToPublishFile);@(TransportPackagesToPublishFile)" />

@maraf @ViktorHofer

Author:	sbomer
Assignees:	maraf
Labels:	`arch-wasm`, `packaging`, `needs-area-label`
Milestone:	8.0.0

maraf · 2023-04-13T05:28:50Z

This seems like the right pattern. I'll update it

runtime/src/installer/prepare-artifacts.proj

Lines 248 to 253 in 0ac097f

    
                 <RidAgnosticNupkgToPublishFile 
        
                   Include=" 
        
                     $(DownloadDirectory)**\Microsoft.NET.Workload.Mono.Toolchain.*Manifest-*.nupkg; 
        
                     $(DownloadDirectory)*\$(PublishRidAgnosticPackagesFromPlatform)\**\*.nupkg; 
        
                     $(DownloadDirectory)*\*AllConfigurations\**\*.nupkg" 
        
                   Exclude="@(RuntimeNupkgFile);@(DownloadedSymbolNupkgFile)" />

I'm curious, I ran the official build for the PR branch yesterday, and it passed. Does this build run something more?

maraf · 2023-04-13T07:06:58Z

How can I validate that #84754 really fixes the issue?

EDIT: I'll be able to validate that the package arrives in the PushToAzureDevOpsArtifacts

radical · 2023-04-13T07:20:48Z

Is this package meant to be downloaded by users? Or is it a transport package for getting some files to the sdk repo?

maraf · 2023-04-13T07:24:30Z

@radical It's kind of a transport package used by SDK

maraf · 2023-04-13T07:39:46Z

Found that the validation by ItemsToSignPostBuild is turned on by PostBuildSign=true, can I set it for my build?

radical · 2023-04-13T07:55:07Z

Naming it like the other transport packages should help at least partially. See

runtime/src/installer/prepare-artifacts.proj

Lines 255 to 257 in 1311b8a

    
           <TransportPackagesToPublishFile 
        
             Include="$(DownloadDirectory)**\*Transport*.nupkg" 
        
             Exclude="@(RuntimeNupkgFile);@(RidAgnosticNupkgToPublishFile);@(DownloadedSymbolNupkgFile)" />

ViktorHofer · 2023-04-13T08:05:32Z

Yes, if the package is a transport package please name it accordingly, i.e.: https://github.com/dotnet/runtime/blob/main/src/libraries/Microsoft.Internal.Runtime.AspNetCore.Transport/src/Microsoft.Internal.Runtime.AspNetCore.Transport.proj

But is the package really a transport package? Transport package means that the customer will never see the package in their local package cache. We use transport packages to transport assets from one repository to the other without every leaking that implementation detail to the customer.

maraf · 2023-04-13T08:26:42Z

But is the package really a transport package? Transport package means that the customer will never see the package in their local package cache. We use transport packages to transport assets from one repository to the other without every leaking that implementation detail to the customer.

No, that is not the case. The package is similar to Microsoft.NET.ILLink.Tasks.
It's registered here as the "Known*" https://github.com/dotnet/installer/blob/cb3333646b29a90f1777a28bafab779379a79b89/src/redist/targets/GenerateBundledVersions.targets#L409-L411 and used by the SDK

Sorry for confusion. I'm still not sure about the terminology here.

ViktorHofer · 2023-04-13T08:45:32Z

OK. That means that it's just a normal shipping package but the customer doesn't reference it directly as the SDK brings it in.

dotnet-issue-labeler bot added the needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners label Apr 12, 2023

ghost added the untriaged New issue has not been triaged by the area owner label Apr 12, 2023

lewing added the packaging Related to packaging label Apr 13, 2023

lewing assigned maraf Apr 13, 2023

lewing added this to the 8.0.0 milestone Apr 13, 2023

ghost removed the untriaged New issue has not been triaged by the area owner label Apr 13, 2023

lewing added the arch-wasm WebAssembly architecture label Apr 13, 2023

lewing added the area-Build-mono label Apr 13, 2023

lewing mentioned this issue Apr 13, 2023

Revert "[browser] Wasm SDK packed as a nuget package" #84743

Merged

maraf mentioned this issue Apr 13, 2023

[wasm] Fix publishing Microsoft.NET.Sdk.WebAssembly.Pack #84754

Closed

ghost added the in-pr There is an active PR which will close this issue when it is merged label Apr 13, 2023

maraf mentioned this issue Apr 13, 2023

[browser] Wasm SDK packed as a nuget package #84761

Merged

SingleAccretion removed the needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners label Apr 13, 2023

lewing closed this as completed in #84761 Apr 14, 2023

ghost removed the in-pr There is an active PR which will close this issue when it is merged label Apr 14, 2023

ghost locked as resolved and limited conversation to collaborators May 14, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Official build failure trying to sign Microsoft.NET.Sdk.WebAssembly.Pack #84742

Official build failure trying to sign Microsoft.NET.Sdk.WebAssembly.Pack #84742

sbomer commented Apr 12, 2023

lewing commented Apr 12, 2023

lewing commented Apr 13, 2023

lewing commented Apr 13, 2023

ghost commented Apr 13, 2023

maraf commented Apr 13, 2023 •

edited

Loading

maraf commented Apr 13, 2023 •

edited

Loading

radical commented Apr 13, 2023

maraf commented Apr 13, 2023 •

edited

Loading

maraf commented Apr 13, 2023

radical commented Apr 13, 2023

ViktorHofer commented Apr 13, 2023 •

edited

Loading

maraf commented Apr 13, 2023 •

edited

Loading

ViktorHofer commented Apr 13, 2023

Official build failure trying to sign Microsoft.NET.Sdk.WebAssembly.Pack #84742

Official build failure trying to sign Microsoft.NET.Sdk.WebAssembly.Pack #84742

Comments

sbomer commented Apr 12, 2023

lewing commented Apr 12, 2023

lewing commented Apr 13, 2023

lewing commented Apr 13, 2023

ghost commented Apr 13, 2023

maraf commented Apr 13, 2023 • edited Loading

maraf commented Apr 13, 2023 • edited Loading

radical commented Apr 13, 2023

maraf commented Apr 13, 2023 • edited Loading

maraf commented Apr 13, 2023

radical commented Apr 13, 2023

ViktorHofer commented Apr 13, 2023 • edited Loading

maraf commented Apr 13, 2023 • edited Loading

ViktorHofer commented Apr 13, 2023

maraf commented Apr 13, 2023 •

edited

Loading

maraf commented Apr 13, 2023 •

edited

Loading

maraf commented Apr 13, 2023 •

edited

Loading

ViktorHofer commented Apr 13, 2023 •

edited

Loading

maraf commented Apr 13, 2023 •

edited

Loading