NuGet intermittently hangs while restoring packages on VSTS

[MatthewSteeples]

I have a case open with Microsoft Support on this (116011113569374) who have advised me that my problem is NuGet related so I need to open it here.

Our solution references over 100 packages (probably close to 150) totalling over 100mb of nupkg files. One of these is phantomjs (2.0.0 and 2.1.1 both had the same issue) which is 18mb itself. Occasionally (sometimes a few times per day), this file seems to cause the nuget client to hang while restoring. Remoting into the box shows that NuGet has an open request to the Azure CDN, but that's all we've been able to determine so far.

VSTS allows the build to run for 6 hours before killing it, and the task does not complete within this time.

I'm not sure what other information I can provide here, but I'm keen to help to get to the bottom of it so it can be resolved.

[yishaigalatzer]

@MatthewSteeples Can you provide more details:

1. Does it ever hang anywhere outside of VSTS?
2. Does it hang if you pass -disableparallelprocessing
3. Do you run NuGet 3.3 or another version?
4. Can you get more information on the machine you run on? Network speed, fiddler trace, etc?

[MatthewSteeples]

Hi @yishaigalatzer

1. I've not witnessed it hang outside of VSTS
2. It does still hang when parallel processing is disabled
3. Whichever version it is it doesn't appear to output it's version number (even on Verbosity Detailed) when restoring packages so I'll see what I can find out. It's whatever comes pre-installed with VSTS
4. It's an Azure D2 instance machine running in the West Europe datacentre. Not sure I'll be able to run a Fiddler trace as it's very much a non-interactive process

[MatthewSteeples]

Update to 3. The version of NuGet 3.2.1. I can't update the version there but I can deploy and run a custom one to see if that makes a difference

[apacha]

I can answer on 3. Yes it happens with NuGet Version 3.3.0 (see issue on Bitrise) and works with NuGet 2.8.6.

[yishaigalatzer]

What is the feed URL you are using?

[MatthewSteeples]

Feeds used:
C:\Users\buildguest\AppData\Local\NuGet\Cache
https://api.nuget.org/v3/index.json

The local cache is always empty because it's a fresh VM

[joelverhagen]

Hey Matthew, as we investigate the root cause here, a mitigation we can provide is a custom build of nuget.exe that supports a download timeout (if the a download takes longer than a certain period, the typically retry logic will be invoked). When I have that build ready, can I send you the NuGet executable to the email listed on your GitHub profile? Does that work for you?

[MatthewSteeples]

Hi Joel,

That would be great thanks. It forwards to a Gmail address which is sometimes funny about receiving exe files so I'll let you know if it fails to arrive

[yishaigalatzer]

@MatthewSteeples Can you please also give us information on the machine - What OS, what version of .NET, and do try to get both MTR logs and a Fiddler trace, it should be doable you will have to log in on the machine with a Remote desktop for that.

You can find information on MTR & Fiddler on this doc - https://docs.nuget.org/consume/nuget-faq

[joelverhagen]

@MatthewSteeples additionally, could you provide us with your packages.config file so we can simulate the restore that you are performing?

[yishaigalatzer]

And another idea, can you please move your user profile on the machine to the SSD drive rather than the spinning rust and see if you get a different result?

[joelverhagen]

Hey @MatthewSteeples, I was wondering if you are still having problems with your restore.

[MatthewSteeples]

@yishaigalatzer I'm afraid I don't know the specific details. It's whatever VSTS is using for hosted builds. It's mostly a black box as far as we're concerned. We have no form of access to the server, unless you want to go down the path of writing a custom TFS build step that would reverse connect an RDP session or something like that.

According to the docs (https://msdn.microsoft.com/library/vs/alm/build/services/hosted-agent-pool) it's Windows Server 2012 R2 64 bit with .NET going all the way up to 4.6.

I can't move the user profile to a different disk, although to be honest I would hope that the VMs are set up to be using the SSDs for build anyway.

@joelverhagen I'll email you through a copy of our packages.config files. It's quite an extensive list. I haven't noticed the issue happen again since you emailed me an updated NuGet, but our CI has been quiet this past week and a bit. We've only done 10 builds since then (and we normally get near that per day).

[yishaigalatzer]

Ok, will follow up with the vsts guys.

Sent from Outlook Mobilehttp://aka.ms/xp9y6l

On Fri, Mar 4, 2016 at 9:57 AM -0800, "Matthew Steeples" <notifications@github.commailto:notifications@github.com> wrote:

@yishaigalatzerhttps://github.com/yishaigalatzer I'm afraid I don't know the specific details. It's whatever VSTS is using for hosted builds. It's mostly a black box as far as we're concerned. We have no form of access to the server, unless you want to go down the path of writing a custom TFS build step that would reverse connect an RDP session or something like that.

According to the docs (https://msdn.microsoft.com/library/vs/alm/build/services/hosted-agent-poolhttps://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2fmsdn.microsoft.com%2flibrary%2fvs%2falm%2fbuild%2fservices%2fhosted-agent-pool&data=01%7c01%7cyigalatz%40microsoft.com%7c7df6c26d317e4d846bc708d344566c4e%7c72f988bf86f141af91ab2d7cd011db47%7c1&sdata=niez8372bu9UaZZsXq74P1KF3GgX9cZjV4ouqeSuHkM%3d) it's Windows Server 2012 R2 64 bit with .NET going all the way up to 4.6.

I can't move the user profile to a different disk, although to be honest I would hope that the VMs are set up to be using the SSDs for build anyway.

@joelverhagenhttps://github.com/joelverhagen I'll email you through a copy of our packages.config files. It's quite an extensive list. I haven't noticed the issue happen again since you emailed me an updated NuGet, but our CI has been quiet this past week and a bit. We've only done 10 builds since then (and we normally get near that per day).

Reply to this email directly or view it on GitHubhttps://github.com//issues/2150#issuecomment-192381709.

[joelverhagen]

This is fixed with a change to add timeout to downloads (fail is no bytes received from the network in X seconds).
Duplicate of #2269

[joelverhagen]

Matthew, please open if this issue is still blocking you.

[benjamin-goldman]

we are also seeing this issue