New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve preloading-request logic #2362

Merged
merged 17 commits into from Jun 10, 2017

Conversation

Projects
None yet
1 participant
@matthid
Member

matthid commented May 25, 2017

  • try to improve error handling in nuget
  • do only a maximum of 7 requests in parallel.
  • re-prioritize work to keep the blocking as low as possible.
  • fixes msbuild 15 support

This hopefully fixes the flawky test.

matthid added some commits May 25, 2017

let blockNum = blockedRaw |> Seq.sumBy (fun (_, num, _) -> num)
let blockPaket4 = TimeSpan.FromMilliseconds(500.0 * float blockNum)
tracefn " - Runtime Paket 4 (estimated ~500ms respose*): %s" (Utils.TimeSpanToReadableString (realTime + blockPaket4))
tracefn " * See http://stats.pingdom.com/aqicaf2upspo/1265300 for average response times."

This comment has been minimized.

@matthid

matthid May 25, 2017

Member

I don't think its useful going forward.

@matthid

matthid May 25, 2017

Member

I don't think its useful going forward.

tracefn " - Blocked (%s): %s (%d times)" reason (Utils.TimeSpanToReadableString elapsed) num
| Profile.Category.ResolverAlgorithmNotBlocked b ->
let reason = reason b
tracefn " - Not Blocked (%s): %d times" reason num

This comment has been minimized.

@matthid

matthid May 25, 2017

Member

This is an indication in how well pre-loading requests (or caching - but we should track that differently) work

@matthid

matthid May 25, 2017

Member

This is an indication in how well pre-loading requests (or caching - but we should track that differently) work

@matthid

This comment has been minimized.

Show comment
Hide comment
@matthid

matthid May 25, 2017

Member

This is 50% slower for me...

Member

matthid commented May 25, 2017

This is 50% slower for me...

work.Priority <- prio
member x.Task =
let { TaskSource = task } = x
task.Task

This comment has been minimized.

@matthid

matthid May 26, 2017

Member

@tpetricek Do you have an idea how I can achieve something like the above.
I basically want a queue but it's critical to re-prioritize some work of it on-demand.

Basically the paket resolver is queuing up work before it actually needs it, but when it needs the info the resolver blocks and therefore the work should be started immediately if not started already. So instead of re-prioritizing another way would be to take the work out of the queue and start it on the resolver directly.

Re-prioritizing generally has the nice property that we can further improve that in the future (ie the more insights the resolver has the more it changes the priority of some work, or even cancels it).

I think the above is a good start, but has some things I don't like:

  • It needs to use sleep so in worst case everything is delayed all the time (I think that's the reason why it's slower than before)
  • When the queue is close to empty a lot of workers go into this bad looking fold (minBy with throw errors...)

If you know other people who might be able to help please ping them. I have ideas but most of them will result in several hundred lines of code and are very hard to get right :/

@matthid

matthid May 26, 2017

Member

@tpetricek Do you have an idea how I can achieve something like the above.
I basically want a queue but it's critical to re-prioritize some work of it on-demand.

Basically the paket resolver is queuing up work before it actually needs it, but when it needs the info the resolver blocks and therefore the work should be started immediately if not started already. So instead of re-prioritizing another way would be to take the work out of the queue and start it on the resolver directly.

Re-prioritizing generally has the nice property that we can further improve that in the future (ie the more insights the resolver has the more it changes the priority of some work, or even cancels it).

I think the above is a good start, but has some things I don't like:

  • It needs to use sleep so in worst case everything is delayed all the time (I think that's the reason why it's slower than before)
  • When the queue is close to empty a lot of workers go into this bad looking fold (minBy with throw errors...)

If you know other people who might be able to help please ping them. I have ideas but most of them will result in several hundred lines of code and are very hard to get right :/

This comment has been minimized.

@matthid

matthid May 29, 2017

Member

Ok nevermind, I think I found a better approach...

@matthid

matthid May 29, 2017

Member

Ok nevermind, I think I found a better approach...

matthid added some commits May 29, 2017

@matthid matthid referenced this pull request May 30, 2017

Merged

downgrade to tooling 1.0 #2380

matthid added some commits Jun 10, 2017

@matthid matthid referenced this pull request Jun 10, 2017

Merged

Refactor nuget #2410

disable 'Failed to satisfy' something is not correct with it as it pr…
…ints 60k lines for not resolving dependency files.
@matthid

This comment has been minimized.

Show comment
Hide comment
@matthid

matthid Jun 10, 2017

Member

Sorry @zzdavid I couldn't figure out why the test explodes. I disabled this code place for now, because it doesn't seem to report the correct thing while the resolver is trying to resolve conflicts...

Member

matthid commented Jun 10, 2017

Sorry @zzdavid I couldn't figure out why the test explodes. I disabled this code place for now, because it doesn't seem to report the correct thing while the resolver is trying to resolve conflicts...

@matthid

This comment has been minimized.

Show comment
Hide comment
@matthid

matthid Jun 10, 2017

Member

Only red because of mono crash :/

Member

matthid commented Jun 10, 2017

Only red because of mono crash :/

@matthid matthid merged commit 56e3d8c into master Jun 10, 2017

3 of 4 checks passed

continuous-integration/travis-ci/push The Travis CI build failed
Details
continuous-integration/appveyor/branch AppVeyor build succeeded
Details
continuous-integration/appveyor/pr AppVeyor build succeeded
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details

@matthid matthid deleted the fix_retry_logic branch Jun 15, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment