Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

error NU1403: The package Microsoft.CSharp.4.0.1 sha512 validation failed. #7921

Closed
ctaggart opened this issue Mar 27, 2019 · 24 comments
Closed

error NU1403: The package Microsoft.CSharp.4.0.1 sha512 validation failed. #7921

ctaggart opened this issue Mar 27, 2019 · 24 comments

Comments

@ctaggart
Copy link

@ctaggart ctaggart commented Mar 27, 2019

I'm trying to add nuget 4.9 lock files, but I get the error:

/src/Serialization.Test/Froto.Serialization.Test.fsproj : error NU1403: The package Microsoft.CSharp.4.0.1 sha512 validation failed. The package is different than the last restore. [/src/Froto.sln]

Steps to reproduce:
Checkout the lock branch from ctaggart/froto#104, then restore using docker:

docker run -it --rm -v ${PWD}:/src -w /src mcr.microsoft.com/dotnet/core/sdk:2.2.105 bash
dotnet restore Froto.sln

My hunch is that the nuget is NuGetFallbackFolder is causing problems. It includes nuget packages that don't match NuGet.org. WHY DOES IT HAVE PACKAGES THAT DON'T MATCH NUGET.ORG?

The Froto.Serialization.Test.fsproj has a lock to:

      "Microsoft.CSharp": {
        "type": "Transitive",
        "resolved": "4.0.1",
        "contentHash": "eEgt/ktABPBa/1xBex3M6b7C4dYsGQ0HWuwe9Xhflgu4cvn4a5VvFSLj5vyTKkp+hKrVpTRdqTXDTAxsIL1Reg==",

It matches what is downloaded from nuget.org:

C:\Users\taggac\.nuget\packages\microsoft.csharp\4.0.1\.nupkg.metadata
{
  "version": 1,
  "contentHash": "eEgt/ktABPBa/1xBex3M6b7C4dYsGQ0HWuwe9Xhflgu4cvn4a5VvFSLj5vyTKkp+hKrVpTRdqTXDTAxsIL1Reg=="
}

The docker image NuGetFallbackFolder contentHash does not match:

root@0366cb41b8b4:/src# cat /usr/share/dotnet/sdk/NuGetFallbackFolder/microsoft.csharp/4.0.1/.nupkg.metadata
{
  "version": 1,
  "contentHash": "17h8b5mXa87XYKrrVqdgZ38JefSUqLChUQpXgSnpzsM0nDOhE40FTeNWOJ/YmySGV6tG6T8+hjz6vxbknHJr6A=="
}

At least the NuGetFallbackFolder on my Windows PC matches the docker image:

C:\Program Files\dotnet\sdk\NuGetFallbackFolder\microsoft.csharp\4.0.1\.nupkg.metadata
{
  "version": 1,
  "contentHash": "17h8b5mXa87XYKrrVqdgZ38JefSUqLChUQpXgSnpzsM0nDOhE40FTeNWOJ/YmySGV6tG6T8+hjz6vxbknHJr6A=="
}

Is there a way to disable NuGetFallbackFolder from the lock check? I tried this without luck:

<DisableImplicitNuGetFallbackFolder>true</DisableImplicitNuGetFallbackFolder>

If you are going to continue shipping NuGet packages in NuGetFallbackFolder, can you make sure they match NuGet.org?

@ctaggart

This comment has been minimized.

Copy link
Author

@ctaggart ctaggart commented Mar 27, 2019

It looks like it may not be the NuGetFallbackFolder, but the contentHash in docker is not producing the same thing as on Windows for all packages. It is different on these packages:

ctaggart/froto@83f9bd3

I think contentHash implementation has a bug.

https://github.com/NuGet/Home/wiki/Nupkg-Metadata-File
Ashish Jain @jainaashish

@nkolev92

This comment has been minimized.

Copy link
Member

@nkolev92 nkolev92 commented Mar 28, 2019

The fallback folder issues are due to #7414.

It's a bug on Dotnet CLI side with the LZMA tool. The bug summarizes the findings the actionable. Unfortunately there will be no changes to the LZMA tool as it's being completely done away with in .NET Core 3.0.

What you are articulating sounds like the bug in #7682, which should have been fixed in 2.2.104. Are you using the same version everywhere?

The property you are using to disable the fallback folder seems right:

<DisableImplicitNuGetFallbackFolder>true</DisableImplicitNuGetFallbackFolder>

But you need to make sure the lock file is generated with that property.

Also, are you using the same version of the dotnet sdk on all instances?

@ctaggart

This comment has been minimized.

Copy link
Author

@ctaggart ctaggart commented Mar 28, 2019

No matter what I do the lock file that is generated while in a docker sdk container contains 17h8b5mXa87XYKrrVqdgZ38JefSUqLChUQpXgSnpzsM0nDOhE40FTeNWOJ/YmySGV6tG6T8+hjz6vxbknHJr6A== as the contentHash when it should be eEgt/ktABPBa/1xBex3M6b7C4dYsGQ0HWuwe9Xhflgu4cvn4a5VvFSLj5vyTKkp+hKrVpTRdqTXDTAxsIL1Reg==.

I created a little app to troubleshoot this:
https://github.com/ctaggart/NuGet.Client/blob/PrintContentHash/PrintContentHash/Program.cs

PS C:\Users\taggac\github\NuGet.Client> docker run -it --rm -v ${PWD}:/src -w /src mcr.microsoft.com/dotnet/core/sdk:2.2.105 bash
root@fd158f92f5d6:/src# dotnet run -p PrintContentHash -- testpackages Microsoft.CSharp 4.0.1
eEgt/ktABPBa/1xBex3M6b7C4dYsGQ0HWuwe9Xhflgu4cvn4a5VvFSLj5vyTKkp+hKrVpTRdqTXDTAxsIL1Reg==
eEgt/ktABPBa/1xBex3M6b7C4dYsGQ0HWuwe9Xhflgu4cvn4a5VvFSLj5vyTKkp+hKrVpTRdqTXDTAxsIL1Reg==
@nkolev92

This comment has been minimized.

Copy link
Member

@nkolev92 nkolev92 commented Mar 28, 2019

@ctaggart
If possible can you provide the docker file you used for testing, so we don't have to go through the same exercise?

Thanks

@ctaggart

This comment has been minimized.

Copy link
Author

@ctaggart ctaggart commented Mar 28, 2019

@nkolev92, steps to reproduce:

git clone git@github.com:ctaggart/froto.git
git checkout lock
docker run -it --rm -v ${PWD}:/src -w /src mcr.microsoft.com/dotnet/core/sdk:2.2.105 bash
git rm **/packages.lock.json
dotnet restore Froto.sln

Various other troubleshooting:

git clean -xdf

dotnet nuget locals all --clear

export NUGET_FALLBACK_PACKAGES=$HOME/.nuget/fallbackpackages/
mkdir -p $NUGET_FALLBACK_PACKAGES

export DOTNET_SKIP_FIRST_TIME_EXPERIENCE=1
rm -rf /usr/share/dotnet/sdk/NuGetFallbackFolder/

Nothing has worked. I've also tried sdk:3.0.100-preview3.

Worth noting that the Directory.Build.props contains:

    <RestorePackagesWithLockFile>true</RestorePackagesWithLockFile>
    <RestoreLockedMode>true</RestoreLockedMode>
@ctaggart

This comment has been minimized.

Copy link
Author

@ctaggart ctaggart commented Mar 28, 2019

I found this pretty strange. I was expecting these to hashes to match.

root@fd158f92f5d6:/src# dotnet run -p PrintContentHash -- /usr/share/dotnet/sdk/NuGetFallbackFolder Microsoft.CSharp 4.0.1
17h8b5mXa87XYKrrVqdgZ38JefSUqLChUQpXgSnpzsM0nDOhE40FTeNWOJ/YmySGV6tG6T8+hjz6vxbknHJr6A==
lIwCKwiI4K/KZHVAMki2zuq9MpwOP/HP2vcbpe1pKo6KHKC8R8zXfA2Z8VNcqXs0DDPMJfrk7JBcMNXLK3nJLg==

root@fd158f92f5d6:/src# cat /usr/share/dotnet/sdk/NuGetFallbackFolder/microsoft.csharp/4.0.1/.nupkg.metadata
{
  "version": 1,
  "contentHash": "17h8b5mXa87XYKrrVqdgZ38JefSUqLChUQpXgSnpzsM0nDOhE40FTeNWOJ/YmySGV6tG6T8+hjz6vxbknHJr6A=="
}

On Windows:

PS C:\users\taggac\github\NuGet.Client> dotnet run -p PrintContentHash -- 'C:\Program Files\dotnet\sdk\NuGetFallbackFolder' Microsoft.CSharp 4.0.1
17h8b5mXa87XYKrrVqdgZ38JefSUqLChUQpXgSnpzsM0nDOhE40FTeNWOJ/YmySGV6tG6T8+hjz6vxbknHJr6A==
b9YcH7cOd8y0zsL8UqGh633QqxHf8A4Ehlia9HXvVBKvpURwF8iCYCllucq07uDQEYiSR95gZStAszIXYoO43Q==
@jainaashish

This comment has been minimized.

Copy link
Contributor

@jainaashish jainaashish commented Mar 28, 2019

@ctaggart it will also be worth checking what's the value in *.nupkg.sha512 file on the system.

And btw what's this value lIwCKwiI4K/KZHVAMki2zuq9MpwOP/HP2vcbpe1pKo6KHKC8R8zXfA2Z8VNcqXs0DDPMJfrk7JBcMNXLK3nJLg==?? and b9YcH7cOd8y0zsL8UqGh633QqxHf8A4Ehlia9HXvVBKvpURwF8iCYCllucq07uDQEYiSR95gZStAszIXYoO43Q==

@nkolev92

This comment has been minimized.

Copy link
Member

@nkolev92 nkolev92 commented Mar 29, 2019

@ctaggart
Those wouldn't match because of the bug in the LZMA.

The first hash value is the one that's written in the fallback folder prior to the LZMA is run.
The 2nd hash value is different because the package itself is different due to the LZMA scrambling it around.

@nkolev92

This comment has been minimized.

Copy link
Member

@nkolev92 nkolev92 commented Mar 29, 2019

The way that the GenerateNupkgMetadataFile method is the following.

public static void GenerateNupkgMetadataFile(string nupkgPath, string installPath, string hashPath, string nupkgMetadataPath)
  1. If the package is signed, generate a content hash.
  2. If it's not, fallback & use the value from the hashPath.

Now prior to the LZMA zips them, the package is signed.
The contentHash and the sha512 will be different.

@nkolev92

This comment has been minimized.

Copy link
Member

@nkolev92 nkolev92 commented Mar 29, 2019

It seems to me like the .nupkg.metadata file in the fallback folder has the same value as the sha512.

Do you see any issues if you completely disable the fallback folder?

@slang25

This comment has been minimized.

Copy link

@slang25 slang25 commented Mar 29, 2019

I'm seeing this for Microsoft.NETCore.Targets.1.1.0 (and other packages), I've disabled the fallback folder, and still have the issue. I've also pinned the dotnet sdk to 2.2.105 in global.json.

Locally I'm running macOS (and generating the lock files), it's building and failing on a docker container.

@ctaggart

This comment has been minimized.

Copy link
Author

@ctaggart ctaggart commented Mar 29, 2019

The expectation is that I am using that same dotnet sdk version, such as 2.2.105, I can lock the depdendencies on one system (Windows) and then restore the depdendencies on another system (Docker). That doesn't work and I haven't found a work-a-round.

dotnet sdk 2.2.105 on Windows produces this:
ctaggart/froto@2c6837c#diff-ce2cbfcdedae8d0b839697df83de2102R64
Serialization.Test/packages.lock.json

      "Microsoft.CSharp": {
        "type": "Transitive",
        "resolved": "4.0.1",
        "contentHash": "eEgt/ktABPBa/1xBex3M6b7C4dYsGQ0HWuwe9Xhflgu4cvn4a5VvFSLj5vyTKkp+hKrVpTRdqTXDTAxsIL1Reg=="

That contentHash ends up matching the sha512sum of the file. I go restore on a docker sdk 2.2.105 docker image using the locked files and it fails:

PS C:\Users\taggac\github\froto> docker run -it --rm -v ${PWD}:/src -w /src mcr.microsoft.com/dotnet/core/sdk:2.2.105 bash
root@d731321ca5cf:/src# dotnet restore Froto.sln /p:DisableImplicitNuGetFallbackFolder=true
  Restoring packages for /src/Parser.Test/Froto.Parser.Test.fsproj...
  Restoring packages for /src/Compiler/Froto.Compiler.fsproj...
  Installing Microsoft.NETCore.DotNetAppHost 2.1.0.
  Installing Microsoft.NETCore.DotNetHostResolver 2.1.0.
  Installing Microsoft.NETCore.DotNetHostPolicy 2.1.0.
  Installing NETStandard.Library 2.0.3.
  Installing Microsoft.NETCore.Targets 2.1.0.
  Installing Microsoft.NETCore.Platforms 2.1.0.
  Installing Microsoft.Build.Tasks.Git 1.0.0-beta-63127-02.
  Installing Microsoft.SourceLink.Common 1.0.0-beta-63127-02.
  Installing FParsec 1.0.3.
  Installing Microsoft.NETCore.App 2.1.0.
  Installing Microsoft.SourceLink.GitHub 1.0.0-beta-63127-02.
  Installing Nerdbank.GitVersioning 2.3.138.
  Installing FSharp.Core 4.5.2.
  Installing Argu 4.0.0.
/src/Compiler/Froto.Compiler.fsproj : error NU1403: The package Argu.4.0.0 sha512 validation failed. The package is different than the last restore. [/src/Froto.sln]
  Generating MSBuild file /src/Compiler/obj/Froto.Compiler.fsproj.nuget.g.props.
  Generating MSBuild file /src/Compiler/obj/Froto.Compiler.fsproj.nuget.g.targets.
  Restore failed in 5.64 sec for /src/Compiler/Froto.Compiler.fsproj.
  Restoring packages for /src/Parser/Froto.Parser.fsproj...
  Installing Microsoft.NETCore.Platforms 1.1.0.
/src/Parser/Froto.Parser.fsproj : error NU1403: The package Microsoft.NETCore.Platforms.1.1.0 sha512 validation failed. The package is different than the last restore. [/src/Froto.sln]
  Generating MSBuild file /src/Parser/obj/Froto.Parser.fsproj.nuget.g.props.
  Generating MSBuild file /src/Parser/obj/Froto.Parser.fsproj.nuget.g.targets.
  Restore failed in 641.24 ms for /src/Parser/Froto.Parser.fsproj.
  Restoring packages for /src/Serialization.Test/Froto.Serialization.Test.fsproj...
  Installing System.Xml.XmlSerializer 4.0.11.
  Installing System.Collections.Concurrent 4.0.12.
  Installing System.Globalization.Extensions 4.0.1.
  Installing System.Private.DataContractSerialization 4.1.1.
  Installing System.Security.Cryptography.Encoding 4.0.0.
  Installing System.Runtime.Numerics 4.0.1.
  Installing System.Security.Cryptography.Primitives 4.0.0.
  Installing runtime.native.System.Security.Cryptography 4.0.0.
  Installing System.Collections.Specialized 4.0.1.
  Installing System.Collections.NonGeneric 4.0.1.
  Installing System.ComponentModel.Primitives 4.1.0.
  Installing System.ComponentModel 4.0.1.
  Installing System.Xml.XmlDocument 4.0.1.
  Installing System.Xml.XPath 4.0.1.
  Installing Microsoft.Win32.Primitives 4.0.1.
  Installing Microsoft.Win32.Registry 4.0.0.
  Installing Unquote 4.0.0.
  Installing FsUnit.xUnit 3.2.0.
  Installing System.Threading.ThreadPool 4.0.10.
  Installing Microsoft.NETCore.App 2.0.0.
  Installing System.Runtime.Handles 4.0.1.
  Installing runtime.native.System 4.0.0.
  Installing System.Collections.Immutable 1.2.0.
  Installing System.Threading.Tasks.Extensions 4.0.0.
  Installing System.IO.FileSystem.Primitives 4.0.1.
  Installing System.Reflection.Primitives 4.0.1.
  Installing System.Reflection.Emit.ILGeneration 4.0.1.
  Installing System.Reflection.Emit.Lightweight 4.0.1.
  Installing System.Reflection.Emit 4.0.1.
  Installing System.Diagnostics.Tools 4.0.1.
  Installing Microsoft.NETCore.Targets 1.0.1.
  Installing System.Runtime.Serialization.Json 4.0.2.
  Installing System.Security.Cryptography.Algorithms 4.2.0.
  Installing xunit 2.4.0.
  Installing xunit.runner.visualstudio 2.4.0.
  Installing System.ComponentModel.TypeConverter 4.1.0.
  Installing System.Xml.XPath.XmlDocument 4.0.1.
  Installing NHamcrest 2.0.1.
  Installing Microsoft.NETCore.Platforms 2.0.0.
  Installing System.IO.FileSystem 4.0.1.
  Installing Microsoft.NETCore.DotNetHostPolicy 2.0.0.
  Installing xunit.analyzers 0.10.0.
  Installing xunit.assert 2.4.0.
  Installing System.Diagnostics.Process 4.1.0.
  Installing xunit.core 2.4.0.
  Installing Microsoft.NET.Test.Sdk 15.0.0.
  Installing Microsoft.NETCore.DotNetHostResolver 2.0.0.
  Installing xunit.extensibility.execution 2.4.0.
  Installing System.Runtime.InteropServices 4.1.0.
  Installing xunit.extensibility.core 2.4.0.
  Installing Microsoft.TestPlatform.TestHost 15.0.0.
  Installing System.Reflection.TypeExtensions 4.1.0.
  Installing Microsoft.NETCore.DotNetAppHost 2.0.0.
  Installing xunit.abstractions 2.0.2.
  Installing Newtonsoft.Json 9.0.1.
  Installing System.Runtime.InteropServices.RuntimeInformation 4.0.0.
  Installing System.Diagnostics.TraceSource 4.0.0.
  Installing System.ComponentModel.EventBasedAsync 4.0.11.
  Installing System.Diagnostics.TextWriterTraceListener 4.0.0.
  Installing Microsoft.TestPlatform.ObjectModel 15.0.0.
  Installing System.Runtime.Loader 4.0.0.
  Installing System.Reflection.Metadata 1.3.0.
  Installing System.Diagnostics.Tracing 4.1.0.
  Installing System.ObjectModel 4.0.12.
  Installing System.Reflection.Extensions 4.0.1.
  Installing System.Threading.Thread 4.0.0.
  Installing System.Resources.ResourceManager 4.0.1.
  Installing System.Xml.ReaderWriter 4.0.11.
  Installing System.Diagnostics.Debug 4.0.11.
  Installing System.Collections 4.0.11.
  Installing Microsoft.CSharp 4.0.1.
  Installing System.Linq 4.1.0.
  Installing System.Reflection 4.1.0.
  Installing System.Runtime.Serialization.Primitives 4.1.1.
  Installing System.Globalization 4.0.11.
  Installing System.Dynamic.Runtime 4.0.11.
  Installing System.Linq.Expressions 4.1.0.
  Installing System.Text.RegularExpressions 4.1.0.
  Installing System.Text.Encoding.Extensions 4.0.11.
  Installing System.Text.Encoding 4.0.11.
  Installing System.Runtime 4.1.0.
  Installing System.IO 4.1.0.
  Installing System.Threading 4.0.11.
  Installing System.Threading.Tasks 4.0.11.
  Installing System.Runtime.Extensions 4.1.0.
  Installing System.Xml.XDocument 4.0.11.
/src/Parser.Test/Froto.Parser.Test.fsproj : error NU1403: The package Microsoft.CSharp.4.0.1 sha512 validation failed. The package is different than the last restore. [/src/Froto.sln]
/src/Serialization.Test/Froto.Serialization.Test.fsproj : error NU1403: The package Microsoft.CSharp.4.0.1 sha512 validation failed. The package is different than the last restore. [/src/Froto.sln]
  Generating MSBuild file /src/Parser.Test/obj/Froto.Parser.Test.fsproj.nuget.g.props.
  Generating MSBuild file /src/Serialization.Test/obj/Froto.Serialization.Test.fsproj.nuget.g.props.
  Generating MSBuild file /src/Parser.Test/obj/Froto.Parser.Test.fsproj.nuget.g.targets.
  Generating MSBuild file /src/Serialization.Test/obj/Froto.Serialization.Test.fsproj.nuget.g.targets.
  Restore failed in 10.84 sec for /src/Parser.Test/Froto.Parser.Test.fsproj.
  Restore failed in 4.3 sec for /src/Serialization.Test/Froto.Serialization.Test.fsproj.
  Restoring packages for /src/Serialization/Froto.Serialization.fsproj...
/src/Serialization/Froto.Serialization.fsproj : error NU1403: The package Microsoft.NETCore.Platforms.1.1.0 sha512 validation failed. The package is different than the last restore. [/src/Froto.sln]
  Generating MSBuild file /src/Serialization/obj/Froto.Serialization.fsproj.nuget.g.props.
  Generating MSBuild file /src/Serialization/obj/Froto.Serialization.fsproj.nuget.g.targets.
  Restore failed in 13.86 ms for /src/Serialization/Froto.Serialization.fsproj.

I begin troubleshooting by recreating the lock files on the docker image and it says the contentHash is:

ctaggart/froto@83f9bd3#diff-ce2cbfcdedae8d0b839697df83de2102R67

      "Microsoft.CSharp": {
        "type": "Transitive",
        "resolved": "4.0.1",
        "contentHash": "17h8b5mXa87XYKrrVqdgZ38JefSUqLChUQpXgSnpzsM0nDOhE40FTeNWOJ/YmySGV6tG6T8+hjz6vxbknHJr6A==",

Global Packages in Windows

NUPKG=/mnt/c/Users/taggac/.nuget/packages/microsoft.csharp/4.0.1/microsoft.csharp.4.0.1.nupkg
SHA512=`cat $NUPKG.sha512`
echo $SHA512
eEgt/ktABPBa/1xBex3M6b7C4dYsGQ0HWuwe9Xhflgu4cvn4a5VvFSLj5vyTKkp+hKrVpTRdqTXDTAxsIL1Reg==
SHA512=`sha512sum "$NUPKG" | cut -d " " -f 1 | xxd -r -p | base64 -w 0`
echo $SHA512
eEgt/ktABPBa/1xBex3M6b7C4dYsGQ0HWuwe9Xhflgu4cvn4a5VvFSLj5vyTKkp+hKrVpTRdqTXDTAxsIL1Reg==
dotnet run -p PrintContentHash -- 'C:\Users\taggac\.nuget\packages' Microsoft.CSharp 4.0.1
nupkgMetadata.ContentHash
eEgt/ktABPBa/1xBex3M6b7C4dYsGQ0HWuwe9Xhflgu4cvn4a5VvFSLj5vyTKkp+hKrVpTRdqTXDTAxsIL1Reg==
packageReader.GetContentHash
eEgt/ktABPBa/1xBex3M6b7C4dYsGQ0HWuwe9Xhflgu4cvn4a5VvFSLj5vyTKkp+hKrVpTRdqTXDTAxsIL1Reg==

NuGetFallbackFolder in Windows

NUPKG=/mnt/c/Program\ Files/dotnet/sdk/NuGetFallbackFolder/microsoft.csharp/4.0.1/microsoft.csharp.4.0.1.nupkg
SHA512=`cat $NUPKG.sha512`
echo $SHA512
17h8b5mXa87XYKrrVqdgZ38JefSUqLChUQpXgSnpzsM0nDOhE40FTeNWOJ/YmySGV6tG6T8+hjz6vxbknHJr6A==
SHA512=`sha512sum "$NUPKG" | cut -d " " -f 1 | xxd -r -p | base64 -w 0`
echo $SHA512
b9YcH7cOd8y0zsL8UqGh633QqxHf8A4Ehlia9HXvVBKvpURwF8iCYCllucq07uDQEYiSR95gZStAszIXYoO43Q==
dotnet run -p PrintContentHash -- 'C:\Program Files\dotnet\sdk\NuGetFallbackFolder' Microsoft.CSharp 4.0.1
nupkgMetadata.ContentHash
17h8b5mXa87XYKrrVqdgZ38JefSUqLChUQpXgSnpzsM0nDOhE40FTeNWOJ/YmySGV6tG6T8+hjz6vxbknHJr6A==
packageReader.GetContentHash
b9YcH7cOd8y0zsL8UqGh633QqxHf8A4Ehlia9HXvVBKvpURwF8iCYCllucq07uDQEYiSR95gZStAszIXYoO43Q==

NuGetFallbackFolder in Docker

NUPKG=/usr/share/dotnet/sdk/NuGetFallbackFolder/microsoft.csharp/4.0.1/microsoft.csharp.4.0.1.nupkg
SHA512=`cat $NUPKG.sha512`
echo $SHA512
17h8b5mXa87XYKrrVqdgZ38JefSUqLChUQpXgSnpzsM0nDOhE40FTeNWOJ/YmySGV6tG6T8+hjz6vxbknHJr6A==
SHA512=`sha512sum "$NUPKG" | cut -d " " -f 1 | xxd -r -p | base64 -w 0`
echo $SHA512
lIwCKwiI4K/KZHVAMki2zuq9MpwOP/HP2vcbpe1pKo6KHKC8R8zXfA2Z8VNcqXs0DDPMJfrk7JBcMNXLK3nJLg==
dotnet run -p PrintContentHash -- '/usr/share/dotnet/sdk/NuGetFallbackFolder' Microsoft.CSharp 4.0.1
nupkgMetadata.ContentHash
17h8b5mXa87XYKrrVqdgZ38JefSUqLChUQpXgSnpzsM0nDOhE40FTeNWOJ/YmySGV6tG6T8+hjz6vxbknHJr6A==
packageReader.GetContentHash
lIwCKwiI4K/KZHVAMki2zuq9MpwOP/HP2vcbpe1pKo6KHKC8R8zXfA2Z8VNcqXs0DDPMJfrk7JBcMNXLK3nJLg==

How do I get the contentHash?

I was expecting packageReader.GetContentHash to return the contentHash value that see in the nupkgMetadata, but looks like it is the hash of the entire nuget package. Any idea how I modify this to get the value I'm looking for?

https://github.com/ctaggart/NuGet.Client/blob/PrintContentHash/PrintContentHash/Program.cs#L46-L50

            using (var stream = File.OpenRead(zipPath))
            using (var packageReader = new PackageArchiveReader(stream, leaveStreamOpen: true))
            {
                var contentHash = packageReader.GetContentHash(CancellationToken.None);
Console.WriteLine(contentHash);
@slang25

This comment has been minimized.

Copy link

@slang25 slang25 commented Mar 29, 2019

Update from me:

I generate a lock file that then goes on to work in CI (docker for me) after deleting the package cache folder.

@ctaggart

This comment has been minimized.

Copy link
Author

@ctaggart ctaggart commented Mar 29, 2019

My mistake was not locking down my NuGet sources with a NuGet.Config, which had this one enabled: Microsoft Visual Studio Offline Packages, but probably better to lock down your the sources with NuGet.Config:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <packageSources>
    <clear />
    <add key="nuget.org" value="https://api.nuget.org/v3/index.json" protocolVersion="3" />
  </packageSources>
</configuration>

The DisableImplicitNuGetFallbackFolder of true is required for the restore to work correctly in docker. Here is my working Directory.Build.props:

<Project>
  <PropertyGroup>
    <RestorePackagesWithLockFile>true</RestorePackagesWithLockFile>
    <RestoreLockedMode>true</RestoreLockedMode>
    <NoWarn>NU1603</NoWarn>
    <DisableImplicitNuGetFallbackFolder>true</DisableImplicitNuGetFallbackFolder>
  </PropertyGroup>
</Project>

I then best to clear all the local nuget caches and regenerate the lock files.

dotnet nuget locals all --clear
git clean -xfd
git rm **/packages.lock.json -f
dotnet restore Froto.sln

It is working now for me. Some of the hash's don't make sense to me still, but at least it is consistent across platforms when you take the NuGetFallbackFolder's completely out of the picture.

@ctaggart ctaggart closed this Mar 29, 2019
@nkolev92

This comment has been minimized.

Copy link
Member

@nkolev92 nkolev92 commented Mar 29, 2019

Thanks for following up @ctaggart

We are sorry that we could not make this easier.

Why did the Microsoft Visual Studio Offline Packages cause issues for you?

@ctaggart

This comment has been minimized.

Copy link
Author

@ctaggart ctaggart commented Mar 29, 2019

@nkolev92 I just tested with it enabled and it doesn't appear to have been a park of the issue. Thanks for clearing that up.

@nkolev92

This comment has been minimized.

Copy link
Member

@nkolev92 nkolev92 commented Mar 29, 2019

Oh, perfect. That's good then.
Just wanted to make sure there are not any lingering issues that we might miss.

lpraz added a commit to lpraz/stryker-net that referenced this issue Oct 29, 2019
May be encountering this issue:
microsoft/azure-pipelines-tasks#11449
Suggested workarounds taken from this comment:
NuGet/Home#7921 (comment)
Mobrockers added a commit to stryker-mutator/stryker-net that referenced this issue Nov 4, 2019
…ibman (#788)

* build(pipeline): Added NuGet package caching

Add CacheBeta task to azure-pipelines.yml to enable NuGet package caching
and hopefully speed up CI builds.
Enable creation of packages.lock.json for projects built in pipeline to
enable above.

* build(pipeline): Added NuGet cache tasks to other stages

* build(pipeline): Added LibMan caching

* build(pipeline): Fixed "pipeline[s]" typos

* build(csproj): Generated packages.lock.json

* build(cache): Fixed libman cache key

* build(cache): Fixed cache step not using template

* build(cache): Changed libman cache key

* build(pipeline): Fixed cache template path in prepare-integration-test-steps.yml

* pipeline(cache): Made NuGet cache unique per OS

* build(nuget): Workaround for failed restore

May be encountering this issue:
microsoft/azure-pipelines-tasks#11449
Suggested workarounds taken from this comment:
NuGet/Home#7921 (comment)

* build(cache): Added OS-specific libman cache paths

* build(pipeline): Moved variables to main yml file

* Remove RestorePackagesWithLockFile from csproj's because that is handles by Directory.Build.Props

* Remove RestoreLockedMode from Directory.Build.Props because you want to be able to update the package lock files as a developer

* Delete not needed nuget.config files

* Regenerate package lock files

* Set restore locked mode true on azure pipelines

* Never RestoreLockedMode for integration test projects because they are used in an integrationtest scenario

* Only use nuget package folders if they contain an entry for our OS, otherwise try deploying embedded vstest binaries

* Remove zero check on _vstestPaths

* Check if paths exists

* ! <> !!

* Install dotnet core before running unit tests

* fix sonarqube major issue

* Update package lock with McMaster.Extensions.CommandLineUtils 2.4.3

* switch back unit test run step

* populate cache before running unit tests

* Revert "populate cache before running unit tests"

This reverts commit a9c5923.

* restore caches as first steps everywhere
@RehanSaeed

This comment has been minimized.

Copy link

@RehanSaeed RehanSaeed commented Nov 6, 2019

If I'm using the latest .NET Core 3.0.100 SDK, should this all work as advertised with just this in my Directory.Build.props?

<Project>
  <PropertyGroup>
    <RestorePackagesWithLockFile>true</RestorePackagesWithLockFile>
  </PropertyGroup>
</Project>
@slang25

This comment has been minimized.

Copy link

@slang25 slang25 commented Nov 6, 2019

It should, but you may need to clear out any packages from you cache that would have been restored with an old nuget. Going forward, you won't get into this state.

@RehanSaeed

This comment has been minimized.

Copy link

@RehanSaeed RehanSaeed commented Nov 6, 2019

This does not seem to be the case. I get different hashes when I build with:

  • .NET Core 3.0.100 CLI
  • Visual Studio 16.3.8

It seems you still require:

    <DisableImplicitNuGetFallbackFolder>true</DisableImplicitNuGetFallbackFolder>
enricosada pushed a commit to enricosada/nosln that referenced this issue Dec 2, 2019
JeremyTCD added a commit to JeringTech/Javascript.NodeJS that referenced this issue Dec 6, 2019
- Fallback folder packages don't have the same hash as packages from NuGet.org: NuGet/Home#7921 (comment)
@mattwoberts

This comment has been minimized.

Copy link

@mattwoberts mattwoberts commented Dec 11, 2019

Hi,

I'm using dotnet SDK 3.0, and I'm introducing a lock file so that I can cache the nuget restore on azure pipelines. I've added <RestorePackagesWithLockFile>true</RestorePackagesWithLockFile> to my project files, and checked the resulting package.lock.json files into source control. All works locally.

When I push this, the azure pipelines restore task fails, and I see lots of issues e.g.:

error NU1403: Package content hash validation failed for runtime.native.System.IO.Compression.4.3.0.

This was running previously.

My nuget restore task is configured to use feeds in my nuget.config file, which I do have in the root of the project:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
	<activePackageSource>
	  <add key="All" value="(Aggregate source)" />
	</activePackageSource>
	<packageSources>
	  <add key="nuget.org" value="https://www.nuget.org/api/v2/" />
	  <add key="myget" value="https://www.myget.org/F/[redacted]/index.json" />
	</packageSources>
	<disabledPackageSources />
</configuration>

Can anyone advise?

@NicolasDorier

This comment has been minimized.

Copy link

@NicolasDorier NicolasDorier commented Dec 17, 2019

Had this problem with 3.1.100.

I deleted my nuget cache dotnet nuget locals all --clear, then deleted the lock file, and regenerated it.

This fixed the issue.

@NicolasDorier

This comment has been minimized.

Copy link

@NicolasDorier NicolasDorier commented Dec 18, 2019

Reporting back:
Had this problem again building my app on 3.1.100 on appveyor where setting DisableImplicitNuGetFallbackFolder to true fixed. I thought this problem was fixed for good in 3.x...

DisableImplicitNuGetFallbackFolder was not needed when building inside docker container. Only on appveyor. (win64)

@shantanu1singh

This comment has been minimized.

Copy link

@shantanu1singh shantanu1singh commented Mar 2, 2020

The fix suggested by @ctaggart worked for me too but doesn't deleting the packages.lock.json file while building the docker image defeat the purpose of trying to enforce the same package versions everywhere?

Wouldn't NuGet pull new versions of packages (if they were available) once the lock file is deleted?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
8 participants
You can’t perform that action at this time.