Packages in FallbackFolders installed by .NET Core SDK are custom installed, and fail signature validation.

[livarcocc]

From @AArnott on September 10, 2018 23:18

As discussed on this appveyor forum, AppVeyor builds recently started breaking for me. Feodor Fitsner kindly came up with a minimal repro that suggests that the .NET Core 2.1 packages on nuget.org have bad signatures.

Repro

Create a test-nuget.csproj file with these contents:

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <TargetFrameworks>netcoreapp1.0;netcoreapp2.0;netcoreapp2.1</TargetFrameworks>
  </PropertyGroup>
</Project>

Rename or remove your %userprofile%\.nuget\packages folder.

Restore packages for test-nuget.csproj using msbuild:

msbuild /t:restore /p:RestoreDisableParallel=true,TreatWarningsAsErrors=true

This produces these errors:

  Restoring packages for D:\temp\test-nuget\test-nuget.csproj...
<snip/>
  Installing Microsoft.NETCore.DotNetAppHost 2.0.0.
  Installing Microsoft.NETCore.App 2.1.0.
D:\temp\test-nuget\test-nuget.csproj : error NU3005: The package signature file entry is invalid. The central directory header field 'compression method' has an in
valid value (8).
  Installing Microsoft.NETCore.DotNetHostPolicy 2.1.0.
D:\temp\test-nuget\test-nuget.csproj : error NU3005: The package signature file entry is invalid. The central directory header field 'compression method' has an in
valid value (8).
  Installing Microsoft.NETCore.Platforms 2.1.0.
D:\temp\test-nuget\test-nuget.csproj : error NU3005: The package signature file entry is invalid. The central directory header field 'compression method' has an in
valid value (8).
  Installing Microsoft.NETCore.Targets 2.1.0.
D:\temp\test-nuget\test-nuget.csproj : error NU3005: The package signature file entry is invalid. The central directory header field 'compression method' has an in
valid value (8).
  Installing NETStandard.Library 2.0.3.
  Installing Microsoft.NETCore.DotNetHostResolver 2.1.0.
D:\temp\test-nuget\test-nuget.csproj : error NU3005: The package signature file entry is invalid. The central directory header field 'compression method' has an in
valid value (8).
  Installing Microsoft.NETCore.DotNetAppHost 2.1.0.
D:\temp\test-nuget\test-nuget.csproj : error NU3005: The package signature file entry is invalid. The central directory header field 'compression method' has an in
valid value (8).
  Generating MSBuild file D:\temp\test-nuget\obj\test-nuget.csproj.nuget.g.props.
  Generating MSBuild file D:\temp\test-nuget\obj\test-nuget.csproj.nuget.g.targets.
  Restore failed in 32.27 sec for D:\temp\test-nuget\test-nuget.csproj.

Interestingly, this only repros while there are multiple target frameworks specified. Reducing it to just netcoreapp2.1 eliminates the error.

Copied from original issue: dotnet/sdk#2523

[livarcocc]

@rohit21agrawal @nkolev92 any idea what would cause these errors?

[livarcocc]

Also adding @mmitche @leecow and @vitek-karas in case there is something wrong with the packages we pushed to nuget.org.

[livarcocc]

From @rohit21agrawal on September 10, 2018 23:21

CC: @dtivel @PatoBeltran for signing issues

[livarcocc]

From @nkolev92 on September 11, 2018 1:18

Did some digging cause this looks weird, if you do a NuGet.exe verify -signatures on the package, for example Microsoft.NETCore.App 2.1.0, you get the same error message.

This seems to suggest that there is something wrong with this package.

The reason why it fails with multitargetting is because of the fallback folders.
When you cross-target 2.x and 1.x, the new fallback folder from C:\Program Files\dotnet\sdk\NuGetFallbackFolder is not used.

When you target 2.x only, then the package is consumed from the fallback folder, instead of downloading it and installing in the global packages folder.

tl;dr;

• The multi-targeting scenario is by design because NuGet needs to download that package.
• Rest of it suggests that the package is not signed correctly.

@dtivel @PatoBeltran will be able to give more info on the 2nd one.

[livarcocc]

From @mmitche on September 11, 2018 15:35

This is really odd. Why would it start now? 2.1.0 is ages old. Was there a change in nuget?

[livarcocc]

From @mmitche on September 11, 2018 15:50

WIth nuget.exe 4.6.2.5055 I get a successful verification

[livarcocc]

From @PatoBeltran on September 12, 2018 0:23

Did some investigation in this with @nkolev92. It seems like the fallback folder in C:\Program Files\dotnet\sdk\NuGetFallbackFolder is being used as a source and the package in there has the signature entry compressed.

This repros with any version of nuget greater than 4.6.2 (4.6.2 has a worse error experience, but the package errors out). If you test this with the same package from nuget.org the verification succeeds, therefore this tells us that the packages in the fallback folder are the ones that are bad.

NuGet tools do not create any compressed signatures, which tells us that there was something that modified the packages after they where signed.

@livarcocc is sdk doing anything with the packages after signing that could have modified the compression metadata of the signature entry in the packages?

[livarcocc]

The fallback folder is used as a source if your application targets netcoreapp 1.1 or 1.0. Yes, the lzma compresses all files, however, we have been doing that for quite some time now.

@PatoBeltran we are going to need help to figure out what we are doing wrong here. Is it a file that we are not expanding correctly?

[livarcocc]

From @jainaashish on September 12, 2018 18:41

We also ran an additional exercise to validate packages hash from extracted SHA512 file vs generating SHA512 hash from nupkg in dotnet fallback folder and all these package validation failed which suggest either dotnet is not extracting nupkg correctly in fallback folder or they are creating a new nupkg in fallback folder.

The right expectation here is that dotnet has single nupkg for each of their package, which is extracted into fallback folder using NuGet and keep the same nupkg there or publish the same nupkg to NuGet.org or any other feed. At no point in time, there should be another nupkg generated for the same package (on the fly or anything) after the original nupkg has been extracted in fallback folder.

[livarcocc]

From @AArnott on October 19, 2018 0:19

What's happening here? Now I'm seeing this error in VS itself, blocking intellisense and builds:

Error occurred while restoring NuGet packages: The package signature file entry is invalid. The central directory header field 'compression method' has an invalid value (8).

[livarcocc]

I am going to move this issue over to NuGet, where we can get quicker and better answers to this.

[mmitche]

@livarcocc Let me know if you need anything from me.

[dtivel]

Somewhere something modified the package after it was signed. It's now considered a tampered package.

[AArnott]

From the nuget side, can the error message be improved to actually mention:

1. what package ID and version failed?
2. where the package was being read from?

Then as a user I might be able to speculate as to how to workaround the problem.

[dtivel]

I think this has been done already for NuGet 4.9 / VS 15.9. See NuGet/NuGet.Client#2360

[AArnott]

@livarcocc I'm not sure why this issue was moved to github considering github isn't the team that deployed the faulty packages with the VS or dotnet SDK installer. Who "owns" this issue at this point?

[jainaashish]

@AArnott there are two parts of the issue with the same root cause.

First part is when you started seeing this as AppVeyor builds which was failing to verify some packages signature which was a legit issue since dotnet lzma tool recompress and generates a new nupkg which failed the signature verification.

Second part is when you started seeing this inside VS as well, this is happening because NuGet introduced a new nupkg metadata file called .nupkg.metadata to contain the package's content hash to differentiate between existing SHA512 file which contains .nupkg file hash. So when NuGet tries to generate this file for an already extracted package in global packages folder, it fails to get the content hash for the signed package (this is the same failure as in the first case).

For first part, we've already improved the error message to show the corresponding package id, version, source to provide more details about the failure. And also, once lzma tool refresh the packages from NuGet.org, then these failures will be fixed (possibly in 15.9 preview5).

Second part is bit tricky and needs to resolve the upgrade scenario of moving from invalid signed package to signed package. We're still discussing and exploring multiple ways to improve this and get the right experience. But at least will make sure to add package id, version details with this exception as well. For now, one workaround is to clean global packages folder and run restore.

[AArnott]

For now, one workaround is to clean global packages folder and run restore.

Thanks, @jainaashish. By "global packages folder" are you referring to %userprofile%\.nuget\packages?

[jainaashish]

Yes. If you aren't overwriting it in your project.

[tillig]

Just updated VS to 15.9.3 which came with .NET SDK 2.1.500 and started getting this error. Even clearing %userprofile%\.nuget\packages didn't solve it for me; I had to rename the C:\Program Files\dotnet\sdk\NuGetFallbackFolder to something else and create a new, empty C:\Program Files\dotnet\sdk\NuGetFallbackFolder. After that and running restore... finally I could build again.

[nkolev92]

@tillig
You would need to somehow get a valid version of the faulty package into the global packages folder before trying to restore again.

Simply deleting the global packages folder, and restoring the same project again would get you in the same situation.

[mikkelbu]

Hi @PatoBeltran. In the NUnit project we have run into this problem on AppVeyor (see error in the end of this comment), see nunit/nunit-console#556 for more context.

My guess is that it is due to an update of nuget to version 4.9.2 (from version 4.7.1, if https://www.appveyor.com/updates/ is correct). We have prior solved the same issue on AppVeyor by introducing a separate nuget restore step (instead of calling the MSBuild restore target), see nunit/nunit-console#525. But now this new step is also failing.

We still, however, support netcore 1.x, so I've been trying to disable/remove the fallbackfolder when building on AppVeyor, but so far I've not succeeded. I've tried

• Specifying the following in our NuGet.config

  <fallbackPackageFolders>
    <clear />
  </fallbackPackageFolders>

• Specifying the following in our csproj

<RestoreFallbackFolders>clear</RestoreFallbackFolders>

Do you know any way we can disable/remove the fallbackfolder (perhaps as separate nuget operation before the nuget restore)?

Example failing build

...
Errors in C:\projects\nunit-console\src\NUnitEngine\mock-assembly\mock-assembly.csproj
    NU3005: Package 'Microsoft.NETCore.Platforms 2.0.0' from source 'C:\Program Files\dotnet\sdk\NuGetFallbackFolder': The package signature file entry is invalid. The central directory header field 'compression method' has an invalid value (8).
...

[mikkelbu]

Ps. We solved the problem by inserting the following element in the project file

<DisableImplicitNuGetFallbackFolder>true</DisableImplicitNuGetFallbackFolder>

[ctaggart]

This continues to be problematic for those of us trying to use NuGet lock files with dotnet 2.x. I think the best solution is to:

1. delete the NuGetFallbackFolder
2. disable recreation of the NuGetFallbackFolder
3. clear out all NuGet caches
4. clear our your project
5. recreate lock files

On Windows, I did this with:

1. deleted C:\Program Files\dotnet\sdk\NuGetFallbackFolder
2. [Environment]::SetEnvironmentVariable("DOTNET_SKIP_FIRST_TIME_EXPERIENCE", "1", "User")
3. dotnet nuget locals all --clear
4. ⚠️ git clean -xfd (commit or back up anything needed but not committed)
5. dotnet restore --force-evaluate

For Linux, it may be something like:

1. rm -rf /usr/share/dotnet/sdk/NuGetFallbackFolder/ (use dotnet --info to find the sdk)
2. put export DOTNET_SKIP_FIRST_TIME_EXPERIENCE=1 in ~/.bash_profile
3. (same)
4. (same)
5. (same)

Nate has details at https://natemcmaster.com/blog/2018/02/02/dotnet-global-tool/ for DOTNET_SKIP_FIRST_TIME_EXPERIENCE gotchas.

[forki]

I still see this error with .NET SDK 3.1.100

[rrelyea]

@forki - would love details of that.
Does that machine have 3.1.100 and 2.x.xxx both installed?
Does the repro machine use any global.json?
Does the problem only occur with TargetFrameworks that are supported by 2.x?

[forki]

All our Linux dotnet core builds stopped working working over holidays on all build agents. No idea what changed, definitely not the code or any deps references. We cant even build stuff that was already building fine before Christmas. It seems to be related to privileges since sudo dotnet restore works, but without sudo this error message is thrown (unfortunately it does not say which file can't be accessed). Rob Relyea <notifications@github.com> schrieb am Do., 2. Jan. 2020, 18:41:

…

@forki <https://github.com/forki> - would love details of that. Does that machine have 3.1.100 and 2.x.xxx both installed? Does the repro machine use any global.json? Does the problem only occur with TargetFrameworks that are supported by 2.x? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#7414?email_source=notifications&email_token=AAAOANCVSFHNJNY4OLVP5E3Q3YRL7A5CNFSM4F56CEZKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEH65FNY#issuecomment-570282679>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAOANDN4UF56KCIM367XUTQ3YRL7ANCNFSM4F56CEZA> .

[forki]

Looks like I'm not the only one. https://twitter.com/TBHolschen/status/1212816843863404546?s=19

But the error message is really from hell. It should tell me where the issue is.