PublicSign with full key pair produces assembly with malformed public key blob on Linux

[steveisok]

Summary

The fix in #19242 (merged Mar 2) for signatureSize with full RSA key pairs on Linux unblocked the signing path, but the resulting assembly has a malformed public key blob that fails MSBuild's IsValidPublicKey validation during ResolveAssemblyReference.

Repro

Any F# project compiled on Linux/macOS with:

• SignAssembly=true
• Full RSA key pair (.snk file, not public-only)
• PublicSign=true (implicit on Unix for non-official builds)

The compiled assembly's public key blob has an invalid header — the blob type byte at offset 12 is 0x07 (PRIVATEKEYBLOB) instead of 0x06 (PUBLICKEYBLOB), causing downstream consumers to reject it.

Error

MSB4018: The "ResolveAssemblyReference" task failed unexpectedly.
System.Security.SecurityException: Invalid assembly public key.
   at System.Reflection.AssemblyNameHelpers.ComputePublicKeyToken(Byte[] publicKey)
   at System.Reflection.AssemblyName.get_FullName()
   at Microsoft.Build.Tasks.ReferenceTable.GetReferenceItems(...)

Impact

This blocks the dotnet/runtime rolling CI pipeline on all Unix legs. The only F# project in the repo (System.Formats.Cbor.Tests.DataModel.fsproj) triggers the failure across 15 build legs per run.

• Affected codeflow PR: [main] Source code updates from dotnet/dotnet runtime#125593
• AzDO build: https://dev.azure.com/dnceng-public/public/_build/results?buildId=1336533
• Windows legs pass (real signing, not PublicSign)

Root Cause Analysis

Before #19242, signatureSize threw CryptographicException on Linux with full key pairs, causing the F# compiler to fall back to producing an unsigned assembly. After #19242, signatureSize succeeds, so the compiler proceeds to embed the public key — but getPublicKeyForKeyPair (or the PE writer) embeds a key blob with an invalid format that fails the runtime's IsValidPublicKey check (source).

The IsValidPublicKey check requires:

1. Blob header: valid sigAlgID, hashAlgID, cbPublicKey
2. publicKey[12] == 0x06 (PUBLICKEYBLOB)

The assembly produced by the fixed F# compiler fails one of these checks.

Existing Workaround in dotnet/runtime

The affected project already has a partial workaround:

<!-- Public signing is broken for f#: https://github.com/dotnet/fsharp/issues/17451 -->
<SignAssembly Condition="'$(DotNetBuildSourceOnly)' == 'true'">false</SignAssembly>

This only applies to source-build. Expanding to all non-Windows platforms (or unconditionally, since this is a test project) would unblock CI.

Related

• PublicSign does not work when supplied with a full private key #17451 (original issue, closed by Fix StrongNameSignatureSize failure on Linux when using full RSA keys #19242)
• Fix StrongNameSignatureSize failure on Linux when using full RSA keys #19242 (the fix that introduced this regression)
• [main] Source code updates from dotnet/dotnet runtime#125593 (blocked codeflow PR)