Resolving dependencies takes too long #1524

Closed
Stift opened this Issue Mar 17, 2016 · 38 comments

Projects

None yet

3 participants

@Stift
Contributor
Stift commented Mar 17, 2016

We have around 90+ direct dependencies and like ~200 dependencies in total in our project and since v2.52.9 the resolving takes 'forever'. It seems for me, that the fix of #1520 has caused that problem. Before an paket install took like 7-10s and now we are waiting +15min at least (I had to stop this and did not measure the correct timing).

@forki
Member
forki commented Mar 17, 2016

yeah looks like a massive perf issue. I will investigate
/cc @isaacabraham

@forki forki added the bug label Mar 17, 2016
@forki
Member
forki commented Mar 17, 2016

I just reverted that change. Can just please check if perf is restored? Seems I need to investigate deeper

@forki
Member
forki commented Mar 17, 2016

pushed 2.52.12 - I think I verschlimmbessert it in 2.52.11

@Stift
Contributor
Stift commented Mar 17, 2016

Tested it with 2.52.12. and it's back to normal. Thx.

@forki
Member
forki commented Mar 17, 2016

OK now I need to find a proper fix for #1520
@Stift I guess I need to push another version soon and ask you again if perf is still good ;-)

@Stift
Contributor
Stift commented Mar 17, 2016

Just do so. I'm glad to test it.
But today is St. Patricks Day and I'm off and out in a few hours:

@forki
Member
forki commented Mar 17, 2016

2.52.13 will be online in 2min. Please give it a try

@Stift
Contributor
Stift commented Mar 17, 2016

that will fit

@forki
Member
forki commented Mar 17, 2016

so it's online

@Stift
Contributor
Stift commented Mar 17, 2016

I'm already testing it. I get some warnings that I cannot specifcally say what they are about:

Could not resolve package Npgsql:
 - Dependencies file requested: 2.2.5-custom
 - <customPackage1> 1.0.11044 requested: >= 0
 - <customPackage2> 1.0.11149 requested: >= 2.2.5
 - <customPackage3> 1.1.10087 requested: >= 2.0.0 < 3.0.0

Also some of the numbers do not fit to the package itself. Is something mixing up here?

@forki
Member
forki commented Mar 17, 2016

shit. soo I'm not done

@Stift
Contributor
Stift commented Mar 17, 2016

Also after a second run of install downloads now all packages again and this again takes a while.
First run: 26s
Second run (downloading?): +5min

@Stift
Contributor
Stift commented Mar 17, 2016

sorry for that

@forki
Member
forki commented Mar 17, 2016

ok this one is really driving me nuts.

I released another version. Can you give me your deps file by any chance?

@forki forki closed this in 73715db Mar 17, 2016
@Stift
Contributor
Stift commented Mar 17, 2016

I'll send it to you, but it contains many of our own packages and we use also a Klondike server as backend. i'll send it to you via mail.

@forki
Member
forki commented Mar 17, 2016

i guess this means 2.52.14 is still broken...

@Stift
Contributor
Stift commented Mar 17, 2016

It has the same strange warning as before, but this time it does not tries to redownload.

@Stift
Contributor
Stift commented Mar 17, 2016

I have sent you our dep file per mail.

@forki
Member
forki commented Mar 17, 2016

just trying to reproduce with your deps file, but I assume in order to make it really hard the conflict is only in your custome nugets ;-)

@Stift
Contributor
Stift commented Mar 17, 2016

no worries. I can pin paket for now. For today I'm out. I have some assumption on that as well. Before monday I will not get into that. Would be glad if you can give me a hint what I should look at. Lately I saw on the verbose level that one package was repeatedly tried to "retrieve information".

@forki forki reopened this Mar 17, 2016
@forki
Member
forki commented Mar 17, 2016

so the public part of your deps file resolves without error. I guess we need to look into this next week.

@tsibelman
Contributor

@forki what do you think to separate releases into stable and beta channels because these frequent changes with high potential for regression, are affecting people day to day work.

@forki
Member
forki commented Mar 18, 2016

we usally do that, but the v3 branch is blocking the alpha channel.
I aslo starting to add property based test around this part.

@tsibelman
Contributor

Maybe you need 3 channels, stable, alpha for bigger longer changes and beta for smaller one

@Stift
Contributor
Stift commented Mar 18, 2016

Ok. I think I found the error in our package that was causing the warning:

Could not resolve package Npgsql:
 - Dependencies file requested: 2.2.5-custom
 - <customPackage1> 1.0.11044 requested: >= 0
 - <customPackage2> 1.0.11149 requested: >= 2.2.5
 - <customPackage3> 1.1.10087 requested: >= 2.0.0 < 3.0.0

The <customPackage2> had its dependency too strict and therefore the Npgsql.2.2.5-custom was not fitting in its constraint. It was obvious after I saw it. The first time I misread the warning and thought that the customPackages were requested, but this morning I just discovered that it was meant in a way that this package requested npgsql of this version.

@forki
Member
forki commented Mar 18, 2016

So what paket says is correct?

@Stift
Contributor
Stift commented Mar 18, 2016

Yes. I misunderstood the message! Sorry for that confusion.

If I read the message like this, I would maybe understand this earlier.
Could not resolve package Npgsql:
- Dependencies file requested package Npgsql: 2.2.5-custom
- 1.0.11044 requested package Npgsql: >= 0
- 1.0.11149 requested package Npgsql: >= 2.2.5
- 1.1.10087 requested package Npgsql: >= 2.0.0 < 3.0.0

Should I make a pr for this?

@forki
Member
forki commented Mar 18, 2016

Yes everything that improves error messages is good.
On Mar 18, 2016 18:03, "Christian Fiebrig" notifications@github.com wrote:

Yes. I misunderstood the message! Sorry for that confusion.

If I read the message like this, I would maybe understand this earlier.
Could not resolve package Npgsql:

  • Dependencies file requested package Npgsql: 2.2.5-custom
  • 1.0.11044 requested package Npgsql: >= 0
  • 1.0.11149 requested package Npgsql: >= 2.2.5
  • 1.1.10087 requested package Npgsql: >= 2.0.0 < 3.0.0

Should I make a pr for this?


You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#1524 (comment)

@forki forki closed this Mar 19, 2016
@forki
Member
forki commented Mar 22, 2016

@stift seems that I need your help again. Property based testing revealed another issue around that code. I was able to fix that but I want to know if how bad the perf is. Can you please check 3.0.0-alpha079? Thanks mate

@Stift
Contributor
Stift commented Mar 22, 2016

No problem.
stats for paket install

Commit v2.54.0 v3.0.0-alpha079
Unclean (with error in nupkg) 24s 2h+
Clean (paket.lock is fine) 8s 8s
Clean (paket.lock has to be updated) 11s 13s

and for a paket update nuget package only on a clean commit

v2.54.0 v3.0.0-alpha079
161s 161s

The unclean commit was the one that originally made problems and where we had a package which was not configured well.

I mean the only issue that I see is that with this unclean package we get into a situation where paket does not behave that well. When I looked in the verbose output of 2.52.10 (where the original problem started) I had the feeling paket was running in circles.

@forki
Member
forki commented Mar 22, 2016

So apart from a strange state it was ok?

@Stift
Contributor
Stift commented Mar 22, 2016

yes. Seems like current stable and alpha are similiar.

@Stift
Contributor
Stift commented Mar 22, 2016

I just fear a bad user experience when paket cannot detect that it runs in a regression.

@forki
Member
forki commented Mar 22, 2016

Yes we need to find that error. Question is how to reproduce on my machine.
On Mar 22, 2016 8:56 PM, "Christian Fiebrig" notifications@github.com
wrote:

I just fear a bad user experience when paket cannot detect that it runs in
a regression.


You are receiving this because you modified the open/close state.
Reply to this email directly or view it on GitHub
#1524 (comment)

@forki
Member
forki commented Mar 23, 2016

ok I added another 3.0alpha package. I hope it solves that 2h hour thing

@Stift
Contributor
Stift commented Mar 23, 2016

I tested with 3.0.0-alpha80 and with the unclean package paket install takes about 27 seconds. On a clean repo without the failing package we still have around 15 seconds. For me this looks definetely better and acceptable (from a user perspective).

Update is also fine on the clean one.

@forki
Member
forki commented Mar 23, 2016

thanks buddy.

@Stift
Contributor
Stift commented Mar 23, 2016

no problem. Glad to serve and give something back.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment