Paket is slow on HDD because of creating the packages folder

[teo-tsirpanis]

Description

I am creating a simple .NET Stnandard library. It only contains FAKE, Chessie and the usual F# SDK libraries. However, probably due to .NET Standard's assembly file fragmentation, Paket downloads a significant amount of libraries, slowing down the build process. This thing does not happen with .NET Core console apps.

paket.dependencies

framework: netstandard1.6

source https://api.nuget.org/v3/index.json

nuget Chessie
nuget FAKE
nuget FSharp.Core
nuget FSharp.NET.Sdk

paket.lock

paket --version

Paket version 5.0.0-alpha020

Repro steps

1. dotnet new classlib -lang F#

2. 'paket convert-from nuget`

3. 'paket add nuget Chessie`

4. paket install

5. Wait... 😫

Expected behavior

Paket would download the dependencies quickly and without any hassle (like what it does so well when working on the good old .NET Framework).

Actual behavior

• Paket downloads everything (including the Linux and macOS runtimes).

• The packages folder becomes quite big (450 MB).

Known workarounds

• Wait every time you run paket install... 😫

• Target the .NET framework (and tolerate a lack of portability for your library 😢)

• Do not use Paket 😨.

[forki]

/cc @matthid regarding runtime libs

[matthid]

I get
54 seconds - ready.
on paket install

[matthid]

It shouldn't actually download that much because we use the nuget cache as well. Do you have a slow hard drive and creating the packages folder itself is taking long for you?

[matthid]

The reason because its not happening in your .NET Core console app is because something is broken in paket. That lockfile looks broken :)

[cwe1ss]

I tried paket as well a few weeks ago with an internal repository that contains a mix of netstandard1.x, netcoreapp1.x and net46 libraries and it took forever to do the migration from csproj/nuget. After more than an hour of waiting, I cancelled it and gave up. I could run it again and provide more information if you want?!

[cwe1ss]

I just tried it again and looked for file accesses with procmon.exe - the step "Resolving packages for group Main" is downloading files into my AppData\Local\NuGet\Cache folder. I have a bunch of ".json" and also a bunch of ".failed" files in that folder.

As you can see, the first few packages were fast (and without .failed files) and then it gets ridiculously slow with just a few packages per minute and a lot of .failed files. I tried this a few times with the same pattern so it can't be my internet connection.

[forki]

Can you please check the specified sources in paket.dependencies? Maybe there are more sources than you actually need? Am 17.05.2017 21:30 schrieb "Christian Weiss" <notifications@github.com>:

…

I just tried it again and looked for file accesses with procmon.exe - the step "Resolving packages for group Main" is downloading files into my AppData\Local\NuGet\Cache folder. I have a bunch of ".json" and also a bunch of ".failed" files in that folder. As you can see, the first few packages were fast (and without .failed files) and then it gets ridiculously slow with just a few packages per minute and a lot of .failed files. I tried this a few times with the same pattern so it can't be my internet connection. [image: unbenannt1] <https://cloud.githubusercontent.com/assets/4581460/26172150/b41e157a-3b47-11e7-9265-e0c5509e0108.PNG> [image: unbenannt2] <https://cloud.githubusercontent.com/assets/4581460/26172151/b73e10e8-3b47-11e7-9d01-48f3952b17fb.PNG> — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#2333 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AADgNGF2KYlANZmYE-iDL7XAWLIKqifkks5r60rBgaJpZM4NZB81> .

[matthid]

This thing does not happen

After #2336 this will pull netcore dependencies properly as well (it was actually a bug):
https://gist.github.com/matthid/dc0a1e9599b340dac289fb8d6a3241e0

Now we need to figure out why its so slow for you...

[teo-tsirpanis]

(it was actually a bug)

Oh god! Really? 🙀🙁

Now we need to figure out why its so slow for you...

I guess it's something with the NuGet cache, but i have no idea.

[matthid]

Without any cache I get:

Performance:
 - Resolver: 1 minute, 14 seconds (2 runs)
    - Runtime: 2 seconds
    - Runtime Paket 4 (estimated ~500ms respose*): 3 minutes, 26 seconds
      * See http://stats.pingdom.com/aqicaf2upspo/1265300 for average response times.
    - Blocked (retrieving package details): 38 seconds (246 times)
    - Blocked (retrieving package versions): 32 seconds (163 times)
 - Disk IO: 32 seconds
 - Average Download Time: 358 milliseconds
 - Number of downloads: 84
 - Average Request Time: 7 seconds
 - Number of Requests: 544
 - Runtime: 2 minutes, 8 seconds

Do you maybe have a slow internet connection?

@teo-tsirpanis Where is it slow for you, when resolving (lists of version numbers showing up) or when downloading (Downloading ... appears)

[teo-tsirpanis]

While building paket.lock, Paket is spending a lot of time Calculating the runtime graph....
This time, I used version 5.0.0-beta005, and a different slightly more complex configuration.

paket.dependencies

source https://api.nuget.org/v3/index.json

framework: netstandard1.6

nuget FAKE
nuget FSharp.Core
nuget SoftWx.Numerics.Portable

group Test
    source https://api.nuget.org/v3/index.json

    framework: netcoreapp1.1
    
    nuget FSharp.Core
    nuget fscheck.xunit
    nuget xunit
    nuget xunit.runner.visualstudio
    nuget Microsoft.NET.Test.Sdk

paket install: 4 minutes, 44 seconds - ready.

@matthid How can I make a more detailed performance report like the previous comment? And why are packages copied again to a local project-specific directory? They are already in `%UserProfile%, so why not use them from there without copying? I guess that's why Paket is so slow in comparison with the .NET CLI.

[matthid]

@teo-tsirpanis I'm trying to release the performance report, but anti-maleware is blocking me, hopefully I can resolve this soon and release a new beta. I will report back (hopefully soon)

[matthid]

We need to keep the packages folder for compatibility reasons, but we maybe can make that opt-out?
We could even add a warning and make it opt-in in the future.
@forki what do you think?

[teo-tsirpanis]

IMHO, that's a good idea. A better proposal is that Paket should put only the direct references in the packages directory by default and allow the user to configure this behavior by introducing a new option like copy_to_packages: always | directonly | never in paket.dependencies.

[matthid]

@teo-tsirpanis yes that would work as well
To be honest I don't think that's the biggest performance problem at the moment.

And @cwe1ss reported hours of waiting, you report 4 minutes. I think we need to be more systematic here to solve this. First step is using beta006 which will report performance more detailed. I managed to release it just right now: https://www.nuget.org/packages/Paket/5.0.0-beta006 might take a moment for nuget to index it.

[cwe1ss]

I'm currently on holiday so I'll try this again next week.

[teo-tsirpanis]

Here it is. With the paket.dependencies file I submitted earlier and after deleting paket.lock and the packages directory, this is the performance breakdown of a paket install:

Performance:
 - Resolver: 45 seconds (6 runs)
    - Runtime: 4 seconds
    - Runtime Paket 4 (estimated ~500ms respose*): 7 minutes, 39 seconds
      * See http://stats.pingdom.com/aqicaf2upspo/1265300 for average response times.
    - Blocked (retrieving package versions): 28 seconds (354 times)
    - Blocked (retrieving package details): 12 seconds (556 times)
 - Disk IO: 3 minutes, 15 seconds
 - Average Request Time: 4 seconds
 - Number of Requests: 580
 - Runtime: 4 minutes, 14 seconds

[forki]

3min disk io? Am 25.05.2017 5:31 nachm. schrieb "Theodore Tsirpanis" < notifications@github.com>: Here it is. With the paket.dependencies file I submitted earlier and after deleting paket.lock and the packages directory, this is the performance breakdown of a paket install: Performance: - Resolver: 45 seconds (6 runs) - Runtime: 4 seconds - Runtime Paket 4 (estimated ~500ms respose*): 7 minutes, 39 seconds * See http://stats.pingdom.com/aqicaf2upspo/1265300 for average response times. - Blocked (retrieving package versions): 28 seconds (354 times) - Blocked (retrieving package details): 12 seconds (556 times) - Disk IO: 3 minutes, 15 seconds - Average Request Time: 4 seconds - Number of Requests: 580 - Runtime: 4 minutes, 14 seconds — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2333 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AADgNCHohRCCccqHjlHeh3DWBGe4N-dyks5r9Z7ggaJpZM4NZB81> .

[matthid]

I guess we have no ssd there and it's in fact the time to build the packages folder...

[teo-tsirpanis]

I guess we have no ssd there and it's in fact the time to build the packages folder...

Most probably, the HDD speed is the botleneck. FYI, this is the laptop I use.

[matthid]

@teo-tsirpanis See #2369 where we track the problem of slow resolution
This issue will be about the disk IO (will change the title appropriately).

@cwe1ss I guess you have the other problem and not this one?

[cwe1ss]

@matthid yes, it seems like my problem happens due to #2369 - thank you!

[matthid]

dulicate of #2141

[matthid]

This will be available soon and is tracked in #2638, please test the PR (or soonish alpha). If it doesn't fix the problem let us know.