paket install slow #2823

TheAngryByrd · 2017-10-04T22:35:12Z

Description

paket install seems to take a long time with dotnet core projects. On my machine this repo took over 10 minutes. Maybe my ssd is dying, maybe my internet is throttled?

Paket version 5.108.0
...
Performance:
 - Resolver: 10 minutes, 28 seconds (2 runs)
    - Runtime: 10 minutes, 23 seconds
    - Blocked (retrieving package details): 891 milliseconds (21 times)
    - Blocked (retrieving package versions): 4 seconds (6 times)
    - Not Blocked (retrieving package details): 174 times
    - Not Blocked (retrieving package versions): 189 times
 - Disk IO: 450 milliseconds
 - Average Download Time: 2 seconds
 - Number of downloads: 1
 - Average Request Time: 159 milliseconds
 - Number of Requests: 212
 - Runtime: 10 minutes, 38 seconds

Repro steps

https://github.com/TheAngryByrd/paket-install-slow

Expected behavior

[Insert Sonic]

Actual behavior

[Insert not Sonic]

Known workarounds

Obtain a hobby between calls to install

Other

Mono JIT compiler version 5.2.0.215 (d15-3/da80840 Thu Jul 20 16:43:07 EDT 2017)
Copyright (C) 2002-2014 Novell, Inc, Xamarin Inc and Contributors. www.mono-project.com
	TLS:           normal
	SIGSEGV:       altstack
	Notification:  kqueue
	Architecture:  amd64
	Disabled:      none
	Misc:          softdebug 
	LLVM:          yes(3.6.0svn-mono-master/8b1520c)
	GC:            sgen (concurrent by default)

.NET Command Line Tools (2.0.0)

Product Information:
 Version:            2.0.0
 Commit SHA-1 hash:  cdcd1928c9

Runtime Environment:
 OS Name:     Mac OS X
 OS Version:  10.12
 OS Platform: Darwin
 RID:         osx.10.12-x64
 Base Path:   /usr/local/share/dotnet/sdk/2.0.0/

Microsoft .NET Core Shared Framework Host

  Version  : 2.0.0
  Build    : e8b8861ac7faf042c87a5c2f9f2d04c98b69f28d

The text was updated successfully, but these errors were encountered:

sandeepc24 · 2017-10-04T22:51:50Z

I have this issue on Windows 10. I avoid upgrading my packages but when I have to add a new package to my project it takes a long time as for some reason I see all projects listed in console.

forki · 2017-10-04T23:15:34Z

Is this a regression? Is it faster with older (recent) versions? Am 05.10.2017 00:51 schrieb "Sandeep Chandra" <notifications@github.com>:

…

I have this issue on Windows 10. I avoid upgrading my packages but when I have to add a new package to my project it takes a long time as for some reason I see all projects listed in console. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#2823 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AADgNHV4xoaPKD5AlWF0SxWfWSnce4Ruks5spAwHgaJpZM4PuVME> .

sandeepc24 · 2017-10-04T23:20:05Z

If I create a new project and download packages using nuget it is much quicker than converting the new project to use paket and then to download the packages.

MadelineRitchie · 2017-10-04T23:36:51Z

I reproduced the issue as well

 - src/paket-slow/paket.references -> src/paket-slow/paket-slow.fsproj
 - tests/paket-slow.Tests/paket.references -> tests/paket-slow.Tests/paket-slow.Tests.fsproj
Performance:
 - Resolver: 12 minutes, 17 seconds (2 runs)
    - Runtime: 11 minutes, 41 seconds
    - Blocked (retrieving package details): 19 seconds (49 times)
    - Blocked (retrieving package versions): 16 seconds (5 times)
    - Not Blocked (retrieving package details): 146 times
    - Not Blocked (retrieving package versions): 190 times
 - Disk IO: 4 seconds
 - Average Download Time: 14 seconds
 - Number of downloads: 66
 - Average Request Time: 406 milliseconds
 - Number of Requests: 648
 - Runtime: 12 minutes, 50 seconds

Deets

Mono JIT compiler version 5.0.1.1 (2017-02/5077205 Wed May 31 14:47:04 BST 2017)
Copyright (C) 2002-2014 Novell, Inc, Xamarin Inc and Contributors. www.mono-project.com
	TLS:           normal
	SIGSEGV:       altstack
	Notification:  kqueue
	Architecture:  amd64
	Disabled:      none
	Misc:          softdebug
	LLVM:          supported, not enabled.
	GC:            sgen (concurrent by default)


.NET Command Line Tools (2.0.0)

Product Information:
 Version:            2.0.0
 Commit SHA-1 hash:  cdcd1928c9

Runtime Environment:
 OS Name:     Mac OS X
 OS Version:  10.12
 OS Platform: Darwin
 RID:         osx.10.12-x64
 Base Path:   /usr/local/share/dotnet/sdk/2.0.0/

Microsoft .NET Core Shared Framework Host

  Version  : 2.0.0
  Build    : e8b8861ac7faf042c87a5c2f9f2d04c98b69f28d

forki · 2017-10-05T07:45:52Z

first install: 5min
second update: 4:32 (all packages already downloaded)

@matthid something is seriously weird. It "hangs" for a while at some runtime packages

matthid · 2017-10-05T08:15:19Z

I cannot take a look the next couple of days. maybe I broke something with my perf changes. Only a profiler will tell

forki · 2017-10-05T08:15:51Z

don't even know where to start here

forki · 2017-10-05T14:04:31Z

ok @matthid just told me that he has an idea and wants to investigate that during the weekend

matthid · 2017-10-05T14:21:38Z

Profiler says the problem might be that simplify is too expensive.
I think there was a bug where simplify was not doing the expected thing in every case which was fixed previously. That might have been the cause of #2364.

I think the "fix" is to only do simplify on import and when writing the lockfile or even better after the resolution.

TheAngryByrd · 2017-10-05T14:31:12Z

For reference:

Paket version 5.101.0
Performance:
 - Resolver: 8 minutes, 32 seconds (2 runs)
    - Runtime: 8 minutes, 21 seconds
    - Blocked (retrieving package details): 1 second (32 times)
    - Blocked (retrieving package versions): 8 seconds (4 times)
    - Not Blocked (retrieving package details): 163 times
    - Not Blocked (retrieving package versions): 191 times
 - Disk IO: 2 seconds
 - Average Request Time: 227 milliseconds
 - Number of Requests: 212
 - Runtime: 8 minutes, 42 seconds

Paket version 5.100.2
Performance:
 - Resolver: 2 minutes, 36 seconds (2 runs)
    - Runtime: 2 minutes, 25 seconds
    - Blocked (retrieving package details): 3 seconds (57 times)
    - Blocked (retrieving package versions): 8 seconds (9 times)
    - Not Blocked (retrieving package details): 138 times
    - Not Blocked (retrieving package versions): 186 times
 - Disk IO: 688 milliseconds
 - Average Request Time: 223 milliseconds
 - Number of Requests: 212
 - Runtime: 2 minutes, 55 seconds

matthid · 2017-10-05T14:34:47Z

@TheAngryByrd Thanks, yes that agrees with my quick findings with the profiler :)

zaaack · 2017-10-12T22:56:56Z

Hit this problem, too. I need to waiting for 17minute with an install.

forki · 2017-10-13T09:12:23Z

@zaaack so you have the deps file for us?

zaaack · 2017-10-13T09:21:12Z

@forki Here is my deps file, not the newest, but I think the speed won't change too much. In my debian labtop, most time it just takes 100% CPU and doesn't cause much network traffic.

source https://nuget.org/api/v2
storage:none

nuget FSharp.Core redirects: force
nuget Fable.JsonConverter
nuget Suave
nuget Microsoft.Extensions.Logging.Console
nuget System.Net.NetworkInformation
nuget Newtonsoft.Json
nuget jose-jwt

nuget Fable.Core
nuget Fable.Elmish.Browser
nuget Fable.Elmish.Debugger
nuget Fable.Elmish.React
nuget Fable.Elmish.HMR
nuget WindowsAzure.Storage
nuget Nett.Coma
nuget Microsoft.EntityFrameworkCore.Tools
nuget Pomelo.EntityFrameworkCore.MySql
nuget Z.EntityFramework.Plus.EFCore
nuget SmtpServer
nuget Foundatio
nuget Humanizer

clitool dotnet-fable
clitool Microsoft.DotNet.Watcher.Tools

group ServerTests
  source https://nuget.org/api/v2
  storage:none
  clitool Microsoft.DotNet.Watcher.Tools
  nuget FSharp.Core redirects: force
  nuget Expecto

group UITests
  source https://nuget.org/api/v2
  framework >= net461
  nuget FSharp.Core redirects: force
  nuget Canopy
  nuget Expecto
  nuget phantomjs

group Build
  source https://nuget.org/api/v2
  framework >= net461
  nuget FAKE
  nuget Newtonsoft.Json

forki · 2017-10-13T13:12:53Z

first trial: #2846

matthid · 2017-10-14T16:04:59Z

Please test with https://github.com/fsprojects/Paket/releases/tag/5.114.5 sorry for the delay :/

forki · 2017-10-14T17:43:30Z

And please report back if it helps. Thanks Am 14.10.2017 18:05 schrieb "Matthias Dittrich" <notifications@github.com>:

…

Closed #2823 <#2823>. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2823 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AADgNK_-4RekxIwd2hJC9wQGx-vwc5Vaks5ssNuugaJpZM4PuVME> .

TheAngryByrd · 2017-10-16T14:38:30Z

Much better! 🎸

Paket version 5.114.5
Performance:
 - Resolver: 2 minutes, 44 seconds (2 runs)
    - Runtime: 2 minutes, 33 seconds
    - Blocked (retrieving package details): 5 seconds (45 times)
    - Blocked (retrieving package versions): 6 seconds (3 times)
    - Not Blocked (retrieving package details): 150 times
    - Not Blocked (retrieving package versions): 192 times
 - Disk IO: 1 second
 - Average Download Time: 4 seconds
 - Number of downloads: 1
 - Average Request Time: 146 milliseconds
 - Number of Requests: 551
 - Runtime: 2 minutes, 53 seconds

matthid · 2017-10-16T14:40:43Z

Thanks for reporting back.

MadelineRitchie · 2017-10-16T17:10:52Z

Paket version 5.114.5
Performance:
 - Resolver: 4 minutes, 17 seconds (2 runs)
    - Runtime: 3 minutes, 39 seconds
    - Blocked (retrieving package details): 25 seconds (51 times)
    - Blocked (retrieving package versions): 12 seconds (5 times)
    - Not Blocked (retrieving package details): 144 times
    - Not Blocked (retrieving package versions): 190 times
 - Disk IO: 2 seconds
 - Average Request Time: 310 milliseconds
 - Number of Requests: 734
 - Runtime: 4 minutes, 28 seconds

+1 🎸

somdoron · 2017-11-04T08:46:57Z

I'm using latest paket (5.120.1) and paket install is taking a lot of time (10 minutes) for every package I'm adding.

TheAngryByrd · 2017-11-30T19:40:05Z

Seems like this is starting to creep back up:

Performance:
 - Resolver: 4 minutes, 6 seconds (2 runs)
    - Runtime: 3 minutes, 38 seconds
    - Blocked (retrieving package details): 20 seconds (52 times)
    - Blocked (retrieving package versions): 7 seconds (4 times)
    - Not Blocked (retrieving package details): 143 times
    - Not Blocked (retrieving package versions): 191 times
 - Disk IO: 947 milliseconds
 - Average Download Time: 4 seconds
 - Number of downloads: 15
 - Average Request Time: 209 milliseconds
 - Number of Requests: 688
 - Runtime: 4 minutes, 16 seconds

forki · 2017-12-01T07:32:59Z

Guys I need deps + lock file to help Am 30.11.2017 20:40 schrieb "Jimmy Byrd" <notifications@github.com>:

…

Seems like this is starting to creep back up: Performance: - Resolver: 4 minutes, 6 seconds (2 runs) - Runtime: 3 minutes, 38 seconds - Blocked (retrieving package details): 20 seconds (52 times) - Blocked (retrieving package versions): 7 seconds (4 times) - Not Blocked (retrieving package details): 143 times - Not Blocked (retrieving package versions): 191 times - Disk IO: 947 milliseconds - Average Download Time: 4 seconds - Number of downloads: 15 - Average Request Time: 209 milliseconds - Number of Requests: 688 - Runtime: 4 minutes, 16 seconds — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2823 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AADgNFxXJHkqc0nuL60JXrgIsR9zKcKjks5s7wSWgaJpZM4PuVME> .

TheAngryByrd · 2017-12-01T15:31:49Z

Sorry I wasn't clear. I was using the same github repo as stated in the original issue: https://github.com/TheAngryByrd/paket-install-slow

forki · 2017-12-02T06:20:02Z

Ok. Thanks. Will investigate on Monday Am 01.12.2017 16:31 schrieb "Jimmy Byrd" <notifications@github.com>:

…

Sorry I wasn't clear. I was using the same github repo as stated in the original issue: https://github.com/TheAngryByrd/paket-install-slow — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2823 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AADgNM9c1TbrwgHVWahzFs79MyxwMYxBks5s8BvlgaJpZM4PuVME> .

blair55 · 2018-02-06T16:40:50Z

I experience slow performance with mono paket.exe install using the below dependencies & lock

https://gist.github.com/blair55/fcbb276fc04e17517fd92beb37f33d11

Performance:
 - Resolver: 11 minutes, 32 seconds (1 runs)
    - Runtime: 10 minutes, 48 seconds
    - Blocked (retrieving package details): 40 seconds (63 times)
    - Blocked (retrieving package versions): 4 seconds (5 times)
    - Not Blocked (retrieving package details): 216 times
    - Not Blocked (retrieving package versions): 11 times
 - Disk IO: 743 milliseconds
 - Average Download Time: 307 milliseconds
 - Number of downloads: 9
 - Average Request Time: 107 milliseconds
 - Number of Requests: 1054
 - Runtime: 11 minutes, 50 seconds

blair55 · 2018-02-06T19:44:34Z

p.s. Paket version 5.133.0

forki · 2018-02-07T06:58:51Z

Sorry I'm very busy the next two weeks. Will look into this after that. Am 06.02.2018 20:44 schrieb "Nick" <notifications@github.com>:

…

p.s. Paket version 5.133.0 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2823 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AADgNNOO_qrSV0GdTf6s-iHGgqAeLhjcks5tSKuigaJpZM4PuVME> .

forki · 2018-02-11T10:54:00Z

@blair55 how fast is second update run after the first run?

blair55 · 2018-02-11T19:04:47Z

@forki Normally there is no significant difference.

forki · 2018-02-12T05:30:43Z

Ok. I Have a version in preparation that needs 3min. Which seems plausible given that this retrieves the internet. But need to verify correctness Am 11.02.2018 20:04 schrieb "Nick" <notifications@github.com>:

…

@forki <https://github.com/forki> Normally there is no significant difference. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2823 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AADgNA08pOrRrDX74W0SsdW16c7ARmBzks5tTznPgaJpZM4PuVME> .

forki · 2018-02-12T11:57:29Z

@blair55 #3037 is solving your case in 45s

forki · 2018-02-12T12:46:04Z

@blair55 can you please retry?

forki · 2018-02-12T12:56:17Z

@TheAngryByrd tried git@github.com:TheAngryByrd/paket-install-slow.git and it's now at 52s

TheAngryByrd · 2018-02-12T13:35:02Z

Awesome! I'll yet when I get in the office

blair55 · 2018-02-12T14:10:46Z

Yep, can confirm:

 - Resolver: 45 seconds (1 runs)
    - Runtime: 19 seconds
    - Blocked (retrieving package details): 11 seconds (96 times)
    - Blocked (retrieving package versions): 14 seconds (16 times)
    - Not Blocked (retrieving package details): 183 times
    - Not Blocked (retrieving package versions): 263 times
 - Disk IO: 123 milliseconds
 - Average Request Time: 124 milliseconds
 - Number of Requests: 282
 - Runtime: 51 seconds

👍

TheAngryByrd · 2018-02-12T16:28:22Z

🎸

Performance:
 - Resolver: 23 seconds (2 runs)
    - Runtime: 5 seconds
    - Blocked (retrieving package details): 3 seconds (42 times)
    - Blocked (retrieving package versions): 14 seconds (12 times)
    - Not Blocked (retrieving package details): 164 times
    - Not Blocked (retrieving package versions): 192 times
 - Disk IO: 591 milliseconds
 - Average Download Time: 167 milliseconds
 - Number of downloads: 17
 - Average Request Time: 88 milliseconds
 - Number of Requests: 227
 - Runtime: 28 seconds

matthid mentioned this issue Oct 5, 2017

Framework restrictions: Simplify (>= net45) && (== net46) to (== net46) #2364

Closed

forki added a commit that referenced this issue Oct 13, 2017

memoize simplify - references #2823

bdca8a1

forki mentioned this issue Oct 13, 2017

memoize simplify #2846

Closed

matthid closed this as completed Oct 14, 2017

paket install slow #2823

paket install slow #2823

Comments

TheAngryByrd commented Oct 4, 2017

Description

Repro steps

Expected behavior

Actual behavior

Known workarounds

Other

sandeepc24 commented Oct 4, 2017

forki commented Oct 4, 2017 via email

sandeepc24 commented Oct 4, 2017

MadelineRitchie commented Oct 4, 2017

forki commented Oct 5, 2017

matthid commented Oct 5, 2017

forki commented Oct 5, 2017

forki commented Oct 5, 2017

matthid commented Oct 5, 2017

TheAngryByrd commented Oct 5, 2017

matthid commented Oct 5, 2017

zaaack commented Oct 12, 2017

forki commented Oct 13, 2017

zaaack commented Oct 13, 2017 • edited

forki commented Oct 13, 2017

matthid commented Oct 14, 2017

forki commented Oct 14, 2017 via email

TheAngryByrd commented Oct 16, 2017 • edited

matthid commented Oct 16, 2017

MadelineRitchie commented Oct 16, 2017

somdoron commented Nov 4, 2017

TheAngryByrd commented Nov 30, 2017

forki commented Dec 1, 2017 via email

TheAngryByrd commented Dec 1, 2017

forki commented Dec 2, 2017 via email

blair55 commented Feb 6, 2018

blair55 commented Feb 6, 2018

forki commented Feb 7, 2018 via email

forki commented Feb 11, 2018

blair55 commented Feb 11, 2018

forki commented Feb 12, 2018 via email

forki commented Feb 12, 2018

forki commented Feb 12, 2018

forki commented Feb 12, 2018 • edited

TheAngryByrd commented Feb 12, 2018

blair55 commented Feb 12, 2018 • edited

TheAngryByrd commented Feb 12, 2018

zaaack commented Oct 13, 2017 •

edited

TheAngryByrd commented Oct 16, 2017 •

edited

forki commented Feb 12, 2018 •

edited

blair55 commented Feb 12, 2018 •

edited