New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Packages in FallbackFolders installed by .NET Core SDK are custom installed, and fail signature validation. #7414

Open
livarcocc opened this Issue Oct 19, 2018 · 28 comments

Comments

Projects
None yet
8 participants
@livarcocc
Copy link

livarcocc commented Oct 19, 2018

From @AArnott on September 10, 2018 23:18

As discussed on this appveyor forum, AppVeyor builds recently started breaking for me. Feodor Fitsner kindly came up with a minimal repro that suggests that the .NET Core 2.1 packages on nuget.org have bad signatures.

Repro

Create a test-nuget.csproj file with these contents:

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <TargetFrameworks>netcoreapp1.0;netcoreapp2.0;netcoreapp2.1</TargetFrameworks>
  </PropertyGroup>
</Project>

Rename or remove your %userprofile%\.nuget\packages folder.

Restore packages for test-nuget.csproj using msbuild:

msbuild /t:restore /p:RestoreDisableParallel=true,TreatWarningsAsErrors=true

This produces these errors:

  Restoring packages for D:\temp\test-nuget\test-nuget.csproj...
<snip/>
  Installing Microsoft.NETCore.DotNetAppHost 2.0.0.
  Installing Microsoft.NETCore.App 2.1.0.
D:\temp\test-nuget\test-nuget.csproj : error NU3005: The package signature file entry is invalid. The central directory header field 'compression method' has an in
valid value (8).
  Installing Microsoft.NETCore.DotNetHostPolicy 2.1.0.
D:\temp\test-nuget\test-nuget.csproj : error NU3005: The package signature file entry is invalid. The central directory header field 'compression method' has an in
valid value (8).
  Installing Microsoft.NETCore.Platforms 2.1.0.
D:\temp\test-nuget\test-nuget.csproj : error NU3005: The package signature file entry is invalid. The central directory header field 'compression method' has an in
valid value (8).
  Installing Microsoft.NETCore.Targets 2.1.0.
D:\temp\test-nuget\test-nuget.csproj : error NU3005: The package signature file entry is invalid. The central directory header field 'compression method' has an in
valid value (8).
  Installing NETStandard.Library 2.0.3.
  Installing Microsoft.NETCore.DotNetHostResolver 2.1.0.
D:\temp\test-nuget\test-nuget.csproj : error NU3005: The package signature file entry is invalid. The central directory header field 'compression method' has an in
valid value (8).
  Installing Microsoft.NETCore.DotNetAppHost 2.1.0.
D:\temp\test-nuget\test-nuget.csproj : error NU3005: The package signature file entry is invalid. The central directory header field 'compression method' has an in
valid value (8).
  Generating MSBuild file D:\temp\test-nuget\obj\test-nuget.csproj.nuget.g.props.
  Generating MSBuild file D:\temp\test-nuget\obj\test-nuget.csproj.nuget.g.targets.
  Restore failed in 32.27 sec for D:\temp\test-nuget\test-nuget.csproj.

Interestingly, this only repros while there are multiple target frameworks specified. Reducing it to just netcoreapp2.1 eliminates the error.

Copied from original issue: dotnet/sdk#2523

@livarcocc

This comment has been minimized.

Copy link

livarcocc commented Oct 19, 2018

@rohit21agrawal @nkolev92 any idea what would cause these errors?

@livarcocc

This comment has been minimized.

Copy link

livarcocc commented Oct 19, 2018

Also adding @mmitche @leecow and @vitek-karas in case there is something wrong with the packages we pushed to nuget.org.

@livarcocc

This comment has been minimized.

Copy link

livarcocc commented Oct 19, 2018

From @rohit21agrawal on September 10, 2018 23:21

CC: @dtivel @PatoBeltran for signing issues

@livarcocc

This comment has been minimized.

Copy link

livarcocc commented Oct 19, 2018

From @nkolev92 on September 11, 2018 1:18

Did some digging cause this looks weird, if you do a NuGet.exe verify -signatures on the package, for example Microsoft.NETCore.App 2.1.0, you get the same error message.

This seems to suggest that there is something wrong with this package.

The reason why it fails with multitargetting is because of the fallback folders.
When you cross-target 2.x and 1.x, the new fallback folder from C:\Program Files\dotnet\sdk\NuGetFallbackFolder is not used.

When you target 2.x only, then the package is consumed from the fallback folder, instead of downloading it and installing in the global packages folder.

tl;dr;

  • The multi-targeting scenario is by design because NuGet needs to download that package.
  • Rest of it suggests that the package is not signed correctly.

@dtivel @PatoBeltran will be able to give more info on the 2nd one.

@livarcocc

This comment has been minimized.

Copy link

livarcocc commented Oct 19, 2018

From @mmitche on September 11, 2018 15:35

This is really odd. Why would it start now? 2.1.0 is ages old. Was there a change in nuget?

@livarcocc

This comment has been minimized.

Copy link

livarcocc commented Oct 19, 2018

From @mmitche on September 11, 2018 15:50

WIth nuget.exe 4.6.2.5055 I get a successful verification

@livarcocc

This comment has been minimized.

Copy link

livarcocc commented Oct 19, 2018

From @PatoBeltran on September 12, 2018 0:23

Did some investigation in this with @nkolev92. It seems like the fallback folder in C:\Program Files\dotnet\sdk\NuGetFallbackFolder is being used as a source and the package in there has the signature entry compressed.

This repros with any version of nuget greater than 4.6.2 (4.6.2 has a worse error experience, but the package errors out). If you test this with the same package from nuget.org the verification succeeds, therefore this tells us that the packages in the fallback folder are the ones that are bad.

NuGet tools do not create any compressed signatures, which tells us that there was something that modified the packages after they where signed.

@livarcocc is sdk doing anything with the packages after signing that could have modified the compression metadata of the signature entry in the packages?

@livarcocc

This comment has been minimized.

Copy link

livarcocc commented Oct 19, 2018

The fallback folder is used as a source if your application targets netcoreapp 1.1 or 1.0. Yes, the lzma compresses all files, however, we have been doing that for quite some time now.

@PatoBeltran we are going to need help to figure out what we are doing wrong here. Is it a file that we are not expanding correctly?

@livarcocc

This comment has been minimized.

Copy link

livarcocc commented Oct 19, 2018

From @jainaashish on September 12, 2018 18:41

We also ran an additional exercise to validate packages hash from extracted SHA512 file vs generating SHA512 hash from nupkg in dotnet fallback folder and all these package validation failed which suggest either dotnet is not extracting nupkg correctly in fallback folder or they are creating a new nupkg in fallback folder.

The right expectation here is that dotnet has single nupkg for each of their package, which is extracted into fallback folder using NuGet and keep the same nupkg there or publish the same nupkg to NuGet.org or any other feed. At no point in time, there should be another nupkg generated for the same package (on the fly or anything) after the original nupkg has been extracted in fallback folder.

@livarcocc

This comment has been minimized.

Copy link

livarcocc commented Oct 19, 2018

From @AArnott on October 19, 2018 0:19

What's happening here? Now I'm seeing this error in VS itself, blocking intellisense and builds:

Error occurred while restoring NuGet packages: The package signature file entry is invalid. The central directory header field 'compression method' has an invalid value (8).

@livarcocc

This comment has been minimized.

Copy link

livarcocc commented Oct 19, 2018

I am going to move this issue over to NuGet, where we can get quicker and better answers to this.

@mmitche

This comment has been minimized.

Copy link

mmitche commented Oct 19, 2018

@livarcocc Let me know if you need anything from me.

@dtivel

This comment has been minimized.

Copy link
Contributor

dtivel commented Oct 19, 2018

Somewhere something modified the package after it was signed. It's now considered a tampered package.

@AArnott

This comment has been minimized.

Copy link
Contributor

AArnott commented Oct 22, 2018

From the nuget side, can the error message be improved to actually mention:

  1. what package ID and version failed?
  2. where the package was being read from?

Then as a user I might be able to speculate as to how to workaround the problem.

@dtivel

This comment has been minimized.

Copy link
Contributor

dtivel commented Oct 22, 2018

I think this has been done already for NuGet 4.9 / VS 15.9. See NuGet/NuGet.Client#2360

@AArnott

This comment has been minimized.

Copy link
Contributor

AArnott commented Oct 22, 2018

@livarcocc I'm not sure why this issue was moved to github considering github isn't the team that deployed the faulty packages with the VS or dotnet SDK installer. Who "owns" this issue at this point?

@jainaashish

This comment has been minimized.

Copy link
Contributor

jainaashish commented Oct 22, 2018

@AArnott there are two parts of the issue with the same root cause.

First part is when you started seeing this as AppVeyor builds which was failing to verify some packages signature which was a legit issue since dotnet lzma tool recompress and generates a new nupkg which failed the signature verification.

Second part is when you started seeing this inside VS as well, this is happening because NuGet introduced a new nupkg metadata file called .nupkg.metadata to contain the package's content hash to differentiate between existing SHA512 file which contains .nupkg file hash. So when NuGet tries to generate this file for an already extracted package in global packages folder, it fails to get the content hash for the signed package (this is the same failure as in the first case).

For first part, we've already improved the error message to show the corresponding package id, version, source to provide more details about the failure. And also, once lzma tool refresh the packages from NuGet.org, then these failures will be fixed (possibly in 15.9 preview5).

Second part is bit tricky and needs to resolve the upgrade scenario of moving from invalid signed package to signed package. We're still discussing and exploring multiple ways to improve this and get the right experience. But at least will make sure to add package id, version details with this exception as well. For now, one workaround is to clean global packages folder and run restore.

@AArnott

This comment has been minimized.

Copy link
Contributor

AArnott commented Oct 22, 2018

For now, one workaround is to clean global packages folder and run restore.

Thanks, @jainaashish. By "global packages folder" are you referring to %userprofile%\.nuget\packages?

@nkolev92 nkolev92 added this to the 4.9 milestone Oct 22, 2018

@jainaashish

This comment has been minimized.

Copy link
Contributor

jainaashish commented Oct 22, 2018

Yes. If you aren't overwriting it in your project.

@rrelyea rrelyea modified the milestones: 4.9, 4.9.x Nov 20, 2018

@rrelyea rrelyea added the Type:Bug label Nov 27, 2018

@rrelyea rrelyea changed the title Builds broken because package signature file entry is invalid Packages in FallbackFolders installed by .NET Core SDK are custom installed, and fail signature validation. Nov 27, 2018

@tillig

This comment has been minimized.

Copy link

tillig commented Nov 29, 2018

Just updated VS to 15.9.3 which came with .NET SDK 2.1.500 and started getting this error. Even clearing %userprofile%\.nuget\packages didn't solve it for me; I had to rename the C:\Program Files\dotnet\sdk\NuGetFallbackFolder to something else and create a new, empty C:\Program Files\dotnet\sdk\NuGetFallbackFolder. After that and running restore... finally I could build again.

@nkolev92

This comment has been minimized.

Copy link
Member

nkolev92 commented Nov 29, 2018

@tillig
You would need to somehow get a valid version of the faulty package into the global packages folder before trying to restore again.

Simply deleting the global packages folder, and restoring the same project again would get you in the same situation.

@tillig

This comment has been minimized.

Copy link

tillig commented Nov 29, 2018

It could be that I'm just not following the workaround then.

For now, one workaround is to clean global packages folder and run restore.

Thanks, @jainaashish. By "global packages folder" are you referring to %userprofile%\.nuget\packages?

Yes. If you aren't overwriting it in your project.

Simply cleaning the global packages folder as noted - literally deleting everything out of it - doesn't fix anything for me. I had to both clean that and nuke everything from NuGetFallbackFolder. Did I miss that step? Is there something else I should be looking at?

@nkolev92

This comment has been minimized.

Copy link
Member

nkolev92 commented Nov 29, 2018

There are 2 different issues here.

One is getting bad packages in the global packages folder. That's the whole multi targeting scenario.
That was a problem in 15.8 and it's equivalent SDK release already.

To workaround that, you'd need to clean the global packages folder and have the packages that would be used from the fallback downloaded from nuget.org. You achieved that by creating an empty fallback folder(that's used as source in multi targeting scenarios).
Once the packages are in the global packages folder, even if you rename back the fallback folder, NuGet would favor those.

Now the second issue is that once you have a bad package in the global packages folder (using older versions of the tools etc, can get it there), due to the new feature that was added, we try to read the nupkg again, and we fail there. For that specific scenario, deleting the global packages folder solves it.

Hope this makes sense.

@tillig

This comment has been minimized.

Copy link

tillig commented Nov 29, 2018

I think it was this bit that I missed (emphasis mine):

To workaround that, you'd need to clean the global packages folder and have the packages that would be used from the fallback downloaded from nuget.org. You achieved that by creating an empty fallback folder(that's used as source in multi targeting scenarios).

Of course, I didn't actually put anything back into the fallback folder, so I'm not sure if anything else will be broken or if that just means it'll force everything to be downloaded and put into the global packages folder. If it means everything will just be re-downloaded... why is the NuGetFallbackFolder even there? Wouldn't it be better to have everything just downloaded clean?

@jainaashish

This comment has been minimized.

Copy link
Contributor

jainaashish commented Nov 29, 2018

@tillig by your description so far, it seems you're experiencing first part of this issue, which was:

First part is when you started seeing this as AppVeyor builds which was failing to verify some packages signature which was a legit issue since dotnet lzma tool recompress and generates a new nupkg which failed the signature verification.

And this part only happens when you're multi targetting where one target being netcoreapp 1.0 or 1.1. Please confirm your scenario matches this otherwise you shouldn't see this issue altogether.

Now, if this is indeed your case, then only way to fix it to remove all these packages (which threw signing error) from fallback folder so that it gets the right package from nuget.org. There are only handful of packages which throw this signing error, which is why we dont recommend clearing whole fallback folder. Other than targetframework being netcoreapp1.1, fallback folder is very critical for offline scenarios as well as performance for file new projects scenarios.

@tillig

This comment has been minimized.

Copy link

tillig commented Nov 29, 2018

Yes, I am building something targeting netcoreapp1.0. I will have to get more surgical about which things I remove from the fallback folder; there were something like 184 errors across all the projects, some of which were duplicate, but in just trying to unblock myself I cleared the whole thing. I copied the original content off to another location so I can be more precise later.

What is the process to update the contents of the whole NuGetFallbackFolder to version(s) that are supported properly? Is there such a thing?

@jainaashish

This comment has been minimized.

Copy link
Contributor

jainaashish commented Nov 29, 2018

there is no process to update the content of fallback folders, its suppose to be meant static and just for consumption purpose. Only dotnet team (who owns it) could update it with any new dotnet release but since there wont be any support for netcoreapp1.0 going forward so they aren't planning to update their lzma tool to fix this fallback folder issue. So I'd recommend if you can move to nercoreapp1.2 or later

@tillig

This comment has been minimized.

Copy link

tillig commented Nov 29, 2018

Well, I'm working on a PR for code I don't own, so moving isn't an option. I'll muddle through.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment