Intermittent package restore failures depending on client version

[maartenba]

NuGet.exe seems to have trouble installing a package from dotnet.myget.org. The issue occurs with NuGet 3.5. The 4.0 nightlies always install the package correctly.

Is there any way to find out the root cause of this issue?

Affected: NuGet version: 3.5
Not affected: NuGet version: 4.x
Not affected: Visual Studio 2015 Update 3 with latest NuGet 3.5

Detailed repro steps so we can see the same problem

1. Use NuGet.exe version 3.5
2. Run NuGet.exe install System.Net.Http -Version 4.4.0-beta-24913-02 -Verbosity Detailed -Source https://dotnet.myget.org/F/dotnet-core/api/v3/index.json
3. If no error is shown, clear local caches (nuget locals all -clear) and run the above again. It sometimes fails, sometimes works.

The error is always one dependency not being found, for example:

System.AggregateException: One or more errors occurred. ---> System.InvalidOperationException: Unable to resolve dependency 'System.Security.Cryptography.X509Certificates'. Source(s) used: 'nuget.org', 'dotnet-core', 'Microsoft Visual Studio Offline Packages'. ---> NuGet.Resolver.NuGetResolverConstraintException: Unable to resolve dependency 'System.Security.Cryptography.X509Certificates'. Source(s) used: 'nuget.org', 'dotnet-core', 'Microsoft Visual Studio Offline Packages'.
   at NuGet.Resolver.PackageResolver.Resolve(PackageResolverContext context, CancellationToken token)
   at NuGet.PackageManagement.NuGetPackageManager.<PreviewInstallPackageAsync>d__57.MoveNext()
   --- End of inner exception stack trace ---
   at NuGet.PackageManagement.NuGetPackageManager.<PreviewInstallPackageAsync>d__57.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at NuGet.PackageManagement.NuGetPackageManager.<InstallPackageAsync>d__41.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at NuGet.CommandLine.InstallCommand.<InstallPackage>d__31.MoveNext()
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
   at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
   at NuGet.CommandLine.Command.Execute()
   at NuGet.CommandLine.Program.MainCore(String workingDirectory, String[] args)
---> (Inner Exception #0) System.InvalidOperationException: Unable to resolve dependency 'System.Security.Cryptography.X509Certificates'. Source(s) used: 'nuget.org', 'dotnet-core', 'Microsoft Visual Studio Offline Packages'. ---> NuGet.Resolver.NuGetResolverConstraintException: Unable to resolve dependency 'System.Security.Cryptography.X509Certificates'. Source(s) used: 'nuget.org', 'dotnet-core', 'Microsoft Visual Studio Offline Packages'.
   at NuGet.Resolver.PackageResolver.Resolve(PackageResolverContext context, CancellationToken token)
   at NuGet.PackageManagement.NuGetPackageManager.<PreviewInstallPackageAsync>d__57.MoveNext()
   --- End of inner exception stack trace ---
   at NuGet.PackageManagement.NuGetPackageManager.<PreviewInstallPackageAsync>d__57.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at NuGet.PackageManagement.NuGetPackageManager.<InstallPackageAsync>d__41.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at NuGet.CommandLine.InstallCommand.<InstallPackage>d__31.MoveNext()<---

Verbose Logs

See out.txt

Succesful logs (with 4.0): out2.txt

[emgarten]

The 4.0 nightlies always install the package correctly.

There haven't been a lot of changes in this area. But there has been some general clean up around the http calls in 4.0.

@joelverhagen do you know of any changes that may have fixed this? I'm wondering if it could still exist in 4.0.

[joelverhagen]

I can't think of a particular change that would fix this. @maartenba, could you provide a Fiddler trace and a full -Verbosity detailed log of the failure? The log.txt you provided above cuts off at:

Attempting to resolve dependencies for package 'System.Net.Http.4.4.0-beta-24913-02' with DependencyBehavior 'Lowest'

[maartenba]

Which is where the exception is thrown :-( Fun fact: if Fiddler sits in between, I'm unable to reproduce this issue (suspecting @emgarten's comment on HTTP-call clean-up in 4.x is probably what fixed this).

Giving up on trying to capure Fiddler (tried 14 times, no repro, dropped Fiddler, repo after 2 tries).

For reference, a (succesful) Fiddler trace is available here, may be useful.

Feel free to close if you're unable to repro.

[joelverhagen]

Could a Wireshark capture be made against an HTTP (not HTTPS) package source? That is a less invasive form of network monitoring.

[maartenba]

Have been unable to copy this to an HTTP repo and repro 😢

Wondering: have there been changes to SSL settings that could be the reason for v4 working? (e.g. in ServicePointManager)

And from a final test for the day: -DisableParallelProcessing seems to also help in succesfully restoring (but is very slow due to # of requests).

[dend]

Encountering the same issue with another package with NuGet 4.3.0.4406:

.\nuget install Microsoft.VisualStudio.Services.ExtensionManagement.WebApi -Verbosity Detailed -Pre

@rrelyea @emgarten - any thoughts on what might be the reason behind this?

[dend]

@rrelyea @emgarten bringing this to your attention again, as it still repros.

[emgarten]

@dend the issue you are hitting doesn't look intermittent to me. A dependency package appears to have an invalid nuspec which causes the v3 feed for nuget.org to be invalid for that package. The result is that zero versions of that package are found by the client.

https://api.nuget.org/v3/registration3-gz-semver2/microsoft.visualstudio.services.gallery.webapi/index.json contains a version range [15.106.0.preview] which should be [15.106.0-preview]

To workaround this you will need to either install Microsoft.VisualStudio.Services.Gallery.WebApi directly, or copy the package you are trying to use to a local folder where it can avoid the invalid one.

I've opened NuGet/NuGetGallery#4684 to track this for nuget.org

From the client side it would help if these issues were logged at a verbose level to help track this down.
https://github.com/NuGet/NuGet.Client/blob/3b388e6eed7ea4464da2769943e20a677a8b6d90/src/NuGet.Core/NuGet.PackageManagement/Resolution/ResolverGather.cs#L521

Potentially @maartenba was hitting a similar problem where the message for a transitive dependency was being ignored, possibly due to an intermittent problem with the feed.