(Future?) performance issue with the NuGet v3 protocol in NuGet client

[maartenba]

Based on https://github.com/dotnet/corefx/issues/11100#issuecomment-275857983, which describes a Visual Studio (and command line NuGet.exe) crash when installing System.Net.Http 4.4.0-beta-24913-021 from https://dotnet.myget.org/F/dotnet-core/api/v3/index.json, we (MyGet) found a (Future?) performance issue with the NuGet v3 protocol in NuGet client.

At the time of writing this issue, at MyGet we don't implement server-side paging for V3 registrations. In other words, a URL like https://dotnet.myget.org/F/dotnet-core/api/v3/registration1/system.net.http/index.json could potentially return 25 MB of JSON data which the client has to parse.

Dependencies may have such big metadata sizes as well. All in all, NuGet.exe (and NuGet in VS) consume over 500 MB of memory at a given point in this restore, essentially slowing down the machine tremendously. The install works (if your machine has enough memory), but takes a while.

Obviously, that is an issue that can be fixed on the server-side (at MyGet we will be deploying a fix after the weekend, on Feb 6, 2017), but it could occur in the future with packages on NuGet.org that have many different versions (think FAKE, RavenDB.Client, ...). Consider this issue not-super-urgent, but as something to look at while NuGet is gaining more and more traction.

The more versions of a package exist, the more likely this will happen for that package. Paging will go a while to circumvent this, but it may be knocking on NuGet's door at some point...

Environment

We tested this issue with Visual Studio 2015 + NuGet 3.5 extension, as well as NuGet.exe 3.5.0.1996.

Repro

As we are fixing MyGet to page results, there is no "live" repro. Nevertheless we have mirrored all downloaded metadata files so a restore should be possible using this point-in-time copy of URLs:

NuGet.exe install System.Net.Http -Version 4.4.0-beta-24913-02 -Verbosity Detailed -Source https://stagingmyget.blob.core.windows.net/repro/index.json

Installing this package in VS2015 leads to VS crashing (most probably due to an out of memory condition, but we did not investigate this further).

Restore/install logs are meaningless: download speeds are normal, parsing in the client is taking excessive time (and is not visible in the logs). We do have some other supporting data, see below.

Supporting data

We have also uploaded a memory profiling dump (using JetBrains dotMemory) where the excessive memory usage and traffic is very visible.

Workspace export (which captured all allocations) is available at: NuGet-Install-Memory-Usage.dmw

A Fiddler request log of our initial investigation is also available: Restore system-net-http.saz

[chadwackerman]

Awesome detail. Would you consider posting average daily bandwidth egress before and after the fix? That would be interesting.

[maartenba]

Would be very interesting indeed :-) Will try to capture that.

[maartenba]

Fix on MyGet has been deployed.

We did observe a few things (@rrelyea / @emgarten):

• The client handles paged data much more comfortably in terms of memory usage. That's a clear win!

• The client does seem to fetch all pages for all dependencies during the dependency resolution stage, which for the above still means there's roughly 200 MB of package metadata going over the wire. Could it be optimized to fetch only "likely" candidate pages and then fetch some more if no valid dependency can be found? This would reduce the metadata going over the wire. Imho this is a more critical thing to look at as it may bring the (Future?) from the issue title closer to the present (also for NuGet.org, think RavenDB.Client and FAKE). Fiddler trace available here

• Long package descriptions are still adding a lot of data into the individual pages, we may reduce the size to 32 packages per page instead of 64 after observing this for a bit. Although given the client downloads all pages anyway this optimization would not help reducing bandwidth usage.

• (Intermittent package restore failures depending on client version (see Intermittent package restore failures depending on client version #4512))

[nkolev92]

Related to #8478 which is tackling search resource changes.
fyi @erdembayar

This is an effective dup of #8058.

[nkolev92]

Lowering the priority here.
#8058 deals with the loading of registration pages in the PM UI. PackageReference restore itself does not use registration and uses the flat container instead and as such this is not a problem at this point in time.