New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"unlisted" spammed over and over during package resolution/restore #2149

Closed
optical opened this Issue Feb 23, 2017 · 10 comments

Comments

Projects
None yet
2 participants
@optical

optical commented Feb 23, 2017

Description

Not sure why paket logs "unlisted" when doing a restore, but after some dependencies I've noticed paket goes abit off the chain when doing the initial resolution of the dependency graph, spamming "unlisted" thousands of times, taking a very very long time (10 min+) to eventually succeed.

Repro steps

Please provide the steps required to reproduce the problem

  1. Download the sample project here
  2. run the runKlondike.bat batch file to startup the nuget repository
  3. Run the paket bootstrapper
  4. Run paket install

Expected behavior

Paket eventually succeeds with a meaningful log output

Actual behavior

Paket succeeds after a very long time and thousands upon thousands of noisy, seemingly meaningless log lines

Known workarounds

None known

@optical optical changed the title from "unlisted" spammed over and over during paket restore to "unlisted" spammed over and over during package resolution/restore Feb 23, 2017

@forki forki closed this in a1700d4 Feb 23, 2017

@forki

This comment has been minimized.

Show comment
Hide comment
@forki

forki Feb 23, 2017

Member

@optical I fixed the spamming in Paket 3 (master branch). I think slow resolution is basically because of yesterday's hotfix where I disabled a heuristic if "lowest matching" is used. I assume with "normal" resolution it would be much faster. I didn't take to much time for yesterday's fix since Paket 4 didn't had that bug.

@cloudRoutine this is an interesting case. As describe above I fixed it in master. Interstingly Paket 4 resolver runs into StackOverflow on this.

Member

forki commented Feb 23, 2017

@optical I fixed the spamming in Paket 3 (master branch). I think slow resolution is basically because of yesterday's hotfix where I disabled a heuristic if "lowest matching" is used. I assume with "normal" resolution it would be much faster. I didn't take to much time for yesterday's fix since Paket 4 didn't had that bug.

@cloudRoutine this is an interesting case. As describe above I fixed it in master. Interstingly Paket 4 resolver runs into StackOverflow on this.

@forki forki reopened this Feb 23, 2017

@forki

This comment has been minimized.

Show comment
Hide comment
@forki

forki Feb 23, 2017

Member

Ok scratch that. There is no StackOverflow. I forgot to use Release mode.

Regarding slowness: I now think it's because of the very tight restrictions. But using max resolver seems a bit faster.

Member

forki commented Feb 23, 2017

Ok scratch that. There is no StackOverflow. I forgot to use Release mode.

Regarding slowness: I now think it's because of the very tight restrictions. But using max resolver seems a bit faster.

@forki forki closed this Feb 23, 2017

forki added a commit that referenced this issue Feb 23, 2017

@optical

This comment has been minimized.

Show comment
Hide comment
@optical

optical Feb 23, 2017

Thanks for the fix. Unfortunately the speed is too slow to be practically useful in my current setup. I've worked around this by pinning some of the packages paket appears to be having a rough time with, which isn't ideal but does work. For some context I am currently migrating a very large, monolithic source shared class library from using checked in binaries over to Nuget packages using paket to resolve. This assembly is shared amongst many applications via a subrepository and they include it as a csproj. This results in a paket.lock for each application, so we do end up doing these initial resolves a fair bit. Its worth noting, as you probably saw yourself the bulk of the restore is spent with the CPU pegged at 100%, so its not the sheer number of network requests that make it slow.

Is this something that paket 4 will eventually cope with better? Admittedly I don't know much about V4, but just dropping the latest release into my solution and running it didn't work so well, looks like some of the resolver & framework settings are either being ignored or renamed, so I wasn't able to see if it was any faster.

optical commented Feb 23, 2017

Thanks for the fix. Unfortunately the speed is too slow to be practically useful in my current setup. I've worked around this by pinning some of the packages paket appears to be having a rough time with, which isn't ideal but does work. For some context I am currently migrating a very large, monolithic source shared class library from using checked in binaries over to Nuget packages using paket to resolve. This assembly is shared amongst many applications via a subrepository and they include it as a csproj. This results in a paket.lock for each application, so we do end up doing these initial resolves a fair bit. Its worth noting, as you probably saw yourself the bulk of the restore is spent with the CPU pegged at 100%, so its not the sheer number of network requests that make it slow.

Is this something that paket 4 will eventually cope with better? Admittedly I don't know much about V4, but just dropping the latest release into my solution and running it didn't work so well, looks like some of the resolver & framework settings are either being ignored or renamed, so I wasn't able to see if it was any faster.

@forki

This comment has been minimized.

Show comment
Hide comment
@forki

forki Feb 23, 2017

Member
Member

forki commented Feb 23, 2017

@forki

This comment has been minimized.

Show comment
Hide comment
@forki

forki Feb 23, 2017

Member
Member

forki commented Feb 23, 2017

@optical

This comment has been minimized.

Show comment
Hide comment
@optical

optical Feb 23, 2017

Thanks for the reply. The advice to not pin anything seems to be a winning strategy, I dropped all the pins and retained the min strategies and things are going about twice as fast as they were before, while resolving the same graph. Its taking less than a minute and not pegging my CPU, so I'll stick with this approach and see how it pans out once I shift all our dependencies over.

I retested with paket4 and found I had made a mistake in my setup which caused it to behave differently - sorry for the confusion.

optical commented Feb 23, 2017

Thanks for the reply. The advice to not pin anything seems to be a winning strategy, I dropped all the pins and retained the min strategies and things are going about twice as fast as they were before, while resolving the same graph. Its taking less than a minute and not pegging my CPU, so I'll stick with this approach and see how it pans out once I shift all our dependencies over.

I retested with paket4 and found I had made a mistake in my setup which caused it to behave differently - sorry for the confusion.

@forki

This comment has been minimized.

Show comment
Hide comment
@forki

forki Feb 23, 2017

Member

Thaks for the feedback. What's the real reason for min strategy?

Member

forki commented Feb 23, 2017

Thaks for the feedback. What's the real reason for min strategy?

@optical

This comment has been minimized.

Show comment
Hide comment
@optical

optical Feb 23, 2017

It was driven by several factors, the tipping point being #2134 stinging us during deployment due to the missing transitive dependency. The other major factor is that this is a project shared by probably 50+ other devs, and since we're switching from checked in binaries (I know...) which provide absolute guarantees around what version you're getting, over to potentially floating versions, there was a lot of fear about potentially getting bumped up to a version which introduces a regression etc. Additionally we have a few dependencies, some of which are internal that aren't following proper SemVer, so that is why we had some pinned packages. Given #2134 came up we decided to go with min to try minimize risk of aggressively floating up, and try ensure all applications sharing this monolith end up with as close to the same versions of its dependencies. The fact that Nuget utilises min by default was also an argument that came into play.

Hopefully that makes some sense into the decision for min over max.

optical commented Feb 23, 2017

It was driven by several factors, the tipping point being #2134 stinging us during deployment due to the missing transitive dependency. The other major factor is that this is a project shared by probably 50+ other devs, and since we're switching from checked in binaries (I know...) which provide absolute guarantees around what version you're getting, over to potentially floating versions, there was a lot of fear about potentially getting bumped up to a version which introduces a regression etc. Additionally we have a few dependencies, some of which are internal that aren't following proper SemVer, so that is why we had some pinned packages. Given #2134 came up we decided to go with min to try minimize risk of aggressively floating up, and try ensure all applications sharing this monolith end up with as close to the same versions of its dependencies. The fact that Nuget utilises min by default was also an argument that came into play.

Hopefully that makes some sense into the decision for min over max.

@forki

This comment has been minimized.

Show comment
Hide comment
@forki

forki Feb 23, 2017

Member
Member

forki commented Feb 23, 2017

@optical

This comment has been minimized.

Show comment
Hide comment
@optical

optical Feb 23, 2017

I am not sure what the solution with #2134 is, but adding binding redirects is one. I need to setup a test scenario which causes Nuget to float the transitive dep up to v15 and see how it copes with the situation, but I haven't got around to it, mostly since we decided to just go with the Min strategy. I'll see if I can get some time this weekend to setup that situation.

optical commented Feb 23, 2017

I am not sure what the solution with #2134 is, but adding binding redirects is one. I need to setup a test scenario which causes Nuget to float the transitive dep up to v15 and see how it copes with the situation, but I haven't got around to it, mostly since we decided to just go with the Min strategy. I'll see if I can get some time this weekend to setup that situation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment