New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Artifact resolution is unreasonably slow for multi-module projects #413

Closed
jwebb opened this Issue Mar 28, 2012 · 76 comments

Comments

Projects
None yet
@jwebb

jwebb commented Mar 28, 2012

note from sbt team

This issue is now solved using cached resolution feature.

updateOptions := updateOptions.value.withConsolidatedResolution(true)

If anyone has further issues with dependency resolution performance, please provide more context and open a new Github issue.

original report

So I have a multi-module project - dozens of modules - in which many of the modules have very similar (transitive) dependencies. Each day, sbt (v0.11.2) decides it needs to re-check all those dependencies, and it does so independently for each module. So if I have, say, 20 modules depending on slf4j, sbt appears to hit the repository 20 times to re-download exactly the same pom.xml. Multiply that by a hundred-odd dependencies and the whole process is taking a couple of minutes. Seems like some caching or up-front reconciliation would be helpful here.

@nau

This comment has been minimized.

Show comment
Hide comment
@nau

nau Jan 30, 2013

Contributor

Confirm, we have the same problem. It's the most annoying thing using sbt. We spend unreasonable amount of time just waiting for dependencies resolution. Any suggestion on how to improve this would be really appreciated.

Contributor

nau commented Jan 30, 2013

Confirm, we have the same problem. It's the most annoying thing using sbt. We spend unreasonable amount of time just waiting for dependencies resolution. Any suggestion on how to improve this would be really appreciated.

@manuelbernhardt

This comment has been minimized.

Show comment
Hide comment
@manuelbernhardt

manuelbernhardt Feb 5, 2013

Same issue here. It takes 5-10 minutes to resolve artifacts, and it is really quite frustrating sometimes.

manuelbernhardt commented Feb 5, 2013

Same issue here. It takes 5-10 minutes to resolve artifacts, and it is really quite frustrating sometimes.

@manuelbernhardt

This comment has been minimized.

Show comment
Hide comment
@manuelbernhardt

manuelbernhardt Mar 9, 2013

I made a screen capture to illustrate how slow it is: http://vimeo.com/61423682

It often is much slower, when on a slower network connection. This is SBT 0.12.2

manuelbernhardt commented Mar 9, 2013

I made a screen capture to illustrate how slow it is: http://vimeo.com/61423682

It often is much slower, when on a slower network connection. This is SBT 0.12.2

@everson

This comment has been minimized.

Show comment
Hide comment
@everson

everson Jul 3, 2013

I have the same issue. It is consistently slow, no matter how much times I run it. It takes more time resolving than building itself, even on a fast network connection. Using SBT 0.12.4.

everson commented Jul 3, 2013

I have the same issue. It is consistently slow, no matter how much times I run it. It takes more time resolving than building itself, even on a fast network connection. Using SBT 0.12.4.

@nau

This comment has been minimized.

Show comment
Hide comment
@nau

nau Sep 14, 2013

Contributor

Guys, any progress on this? We spend horrifying amount of time just resolving same hundred dependencies for each of 20 sub-projects. It can take 10 minutes to resolve the same artifacts that didn't change. I tired of hearing "Maven does it better" and hearing suggestions to move back to Maven.
Currently, for us, this is the most annoying, hence the most important issue. Please, help :)

Contributor

nau commented Sep 14, 2013

Guys, any progress on this? We spend horrifying amount of time just resolving same hundred dependencies for each of 20 sub-projects. It can take 10 minutes to resolve the same artifacts that didn't change. I tired of hearing "Maven does it better" and hearing suggestions to move back to Maven.
Currently, for us, this is the most annoying, hence the most important issue. Please, help :)

@mtgto

This comment has been minimized.

Show comment
Hide comment
@mtgto

mtgto Sep 29, 2013

I affected same problem in my project using sbt multi-modules (more than 10).
To solve this problem, I traced how HTTP requests called by sbt.
I found below remarkable situations:

  • When I use _SNAPSHOT VERSION library_, sbt might access to the resolver per every building of sub-projects.
  • sbt might not memory any HTTP response (include 404 Not Found) on library resolution of another sub-project.

Now, I am using squid to cache sbt's wasteful requests.
I successfully speed-up build-time 310 sec to 244 sec by squid.
(write -Dhttp.proxyHost=<squid host> -Dhttp.proxyPort=<squid port> to SBT_OPTS in ~/.sbtconfig)

In addition, I shall recommend to avoid using _SNAPSHOT VERSION_ library ;-p

mtgto commented Sep 29, 2013

I affected same problem in my project using sbt multi-modules (more than 10).
To solve this problem, I traced how HTTP requests called by sbt.
I found below remarkable situations:

  • When I use _SNAPSHOT VERSION library_, sbt might access to the resolver per every building of sub-projects.
  • sbt might not memory any HTTP response (include 404 Not Found) on library resolution of another sub-project.

Now, I am using squid to cache sbt's wasteful requests.
I successfully speed-up build-time 310 sec to 244 sec by squid.
(write -Dhttp.proxyHost=<squid host> -Dhttp.proxyPort=<squid port> to SBT_OPTS in ~/.sbtconfig)

In addition, I shall recommend to avoid using _SNAPSHOT VERSION_ library ;-p

@diwa-zz

This comment has been minimized.

Show comment
Hide comment
@diwa-zz

diwa-zz Oct 6, 2013

+1 for a solution
We have a play application split into multiple projects and encounter this behavior all the time

diwa-zz commented Oct 6, 2013

+1 for a solution
We have a play application split into multiple projects and encounter this behavior all the time

@dilbert69

This comment has been minimized.

Show comment
Hide comment
@dilbert69

dilbert69 Oct 21, 2013

+1 for a solution
We also have a small play application split into multiple small projects ... this gets especially annoying when fixing small issues in short iterations ... seems not too many people face this issue

dilbert69 commented Oct 21, 2013

+1 for a solution
We also have a small play application split into multiple small projects ... this gets especially annoying when fixing small issues in short iterations ... seems not too many people face this issue

@brikis98

This comment has been minimized.

Show comment
Hide comment
@brikis98

brikis98 Nov 22, 2013

It's still slow for us with SBT 0.13. We have several large multi-project setups and they have become unusably slow.

brikis98 commented Nov 22, 2013

It's still slow for us with SBT 0.13. We have several large multi-project setups and they have become unusably slow.

@jsuereth

This comment has been minimized.

Show comment
Hide comment
@jsuereth

jsuereth Nov 22, 2013

Member

This is kind of a known issue with -SNAPSHOT and Ivy. We'll see what we can do to workaround. I'm totally aware of the slowdown, sorry guys. Patches more than welcome.

Member

jsuereth commented Nov 22, 2013

This is kind of a known issue with -SNAPSHOT and Ivy. We'll see what we can do to workaround. I'm totally aware of the slowdown, sorry guys. Patches more than welcome.

@harrah

This comment has been minimized.

Show comment
Hide comment
@harrah

harrah Nov 22, 2013

Member

Redownloading the metadata for -SNAPSHOTs and other dynamic revisions will only be a problem if explicitly running update as documented here: http://www.scala-sbt.org/release/docs/Detailed-Topics/Dependency-Management-Flow.html. When explicitly runningn update, there are many possible reasons for slow downloading, such as misconfigured repository managers or Ivy just being serial. Again, downloading should only happen as described by the above document. Things are already cached fairly well for stable revisions.

The hardest to address would be if it is slow even if there is no network access. No network access can be forced by having everything in the cache and using offline := true. (offline really tells Ivy not to update dynamic revisions.) In that case, it is likely Ivy's resolution computation itself which is slow.

Member

harrah commented Nov 22, 2013

Redownloading the metadata for -SNAPSHOTs and other dynamic revisions will only be a problem if explicitly running update as documented here: http://www.scala-sbt.org/release/docs/Detailed-Topics/Dependency-Management-Flow.html. When explicitly runningn update, there are many possible reasons for slow downloading, such as misconfigured repository managers or Ivy just being serial. Again, downloading should only happen as described by the above document. Things are already cached fairly well for stable revisions.

The hardest to address would be if it is slow even if there is no network access. No network access can be forced by having everything in the cache and using offline := true. (offline really tells Ivy not to update dynamic revisions.) In that case, it is likely Ivy's resolution computation itself which is slow.

@brikis98

This comment has been minimized.

Show comment
Hide comment
@brikis98

brikis98 Nov 26, 2013

We aren't using any -SNAPSHOT versions - we even have a precommit hook to prevent them. One thing we noticed with multi-project setups is that adding the following speeds things up:

aggregate in update := false

If you aggregate several projects, is SBT redundantly (er, sequentially) updating all of them every single time?

That said, updating even a single project seems particularly slow: on the order of 1 minute for a few hundred dependencies, all in the local cache. Not sure what the issue is there.

brikis98 commented Nov 26, 2013

We aren't using any -SNAPSHOT versions - we even have a precommit hook to prevent them. One thing we noticed with multi-project setups is that adding the following speeds things up:

aggregate in update := false

If you aggregate several projects, is SBT redundantly (er, sequentially) updating all of them every single time?

That said, updating even a single project seems particularly slow: on the order of 1 minute for a few hundred dependencies, all in the local cache. Not sure what the issue is there.

@harrah

This comment has been minimized.

Show comment
Hide comment
@harrah

harrah Nov 26, 2013

Member

If project root aggregates other projects, it just means when you run update on the root project from the command line, it also runs each aggregated project's update. If you depend on root/update programmatically, only root/update is called. It is strictly a command line convenience.

Each project's update runs once. There shouldn't be any redundant resolution done in the absence of snapshots. Ivy's not really safe for concurrent cache access (maybe if you tried hard you could make it so, but in practice it isn't), so yes, only one update runs at a time.

I assume a few hundred dependencies is the number of ultimately selected dependencies. That's useful to know and is correlated to the next number, but the direct variable is probably the total number of considered dependencies. (Probably not a major distinction, but figured I'd mention it.) Typically, I'd expect < 10 s for that size, so perhaps something is off. It sounds like it might be in the resolution algorithm, so it is probably hard to say anything without inspecting the actual project.

Member

harrah commented Nov 26, 2013

If project root aggregates other projects, it just means when you run update on the root project from the command line, it also runs each aggregated project's update. If you depend on root/update programmatically, only root/update is called. It is strictly a command line convenience.

Each project's update runs once. There shouldn't be any redundant resolution done in the absence of snapshots. Ivy's not really safe for concurrent cache access (maybe if you tried hard you could make it so, but in practice it isn't), so yes, only one update runs at a time.

I assume a few hundred dependencies is the number of ultimately selected dependencies. That's useful to know and is correlated to the next number, but the direct variable is probably the total number of considered dependencies. (Probably not a major distinction, but figured I'd mention it.) Typically, I'd expect < 10 s for that size, so perhaps something is off. It sounds like it might be in the resolution algorithm, so it is probably hard to say anything without inspecting the actual project.

@brikis98

This comment has been minimized.

Show comment
Hide comment
@brikis98

brikis98 Nov 26, 2013

The impression I got was that when we used the run command on the root project, it ran update on each of the projects that root aggregates. This was extremely slow and stopped being a problem when we added aggregate in update := false.

The number of direct dependencies, judging by show libraryDependencies is 10-15; there are several hundred once you fan out to all the transitive dependencies.

We've noticed some other oddities:

  1. Even if everything is in the local ivy cache, running update is still slow: on the order of ~40 seconds. Most of the checks seem fast, but the occasional artifact takes 2-3 seconds. With several hundred dependencies in the entire graph, this adds up quickly. No idea why reading a tiny XML file on the hard drive would take that long.
  2. If we change the version setting on a project, it results in update running again. There is likely something else at play here, so we're still digging into it.

brikis98 commented Nov 26, 2013

The impression I got was that when we used the run command on the root project, it ran update on each of the projects that root aggregates. This was extremely slow and stopped being a problem when we added aggregate in update := false.

The number of direct dependencies, judging by show libraryDependencies is 10-15; there are several hundred once you fan out to all the transitive dependencies.

We've noticed some other oddities:

  1. Even if everything is in the local ivy cache, running update is still slow: on the order of ~40 seconds. Most of the checks seem fast, but the occasional artifact takes 2-3 seconds. With several hundred dependencies in the entire graph, this adds up quickly. No idea why reading a tiny XML file on the hard drive would take that long.
  2. If we change the version setting on a project, it results in update running again. There is likely something else at play here, so we're still digging into it.
@harrah

This comment has been minimized.

Show comment
Hide comment
@harrah

harrah Nov 26, 2013

Member

I believe you've seen http://www.scala-sbt.org/release/docs/Detailed-Topics/Dependency-Management-Flow.html#caching-and-configuration, so to apply some of the points from that section to what you are saying:

  1. "Normally, if no dependency management configuration has changed since the last successful resolution and the retrieved files are still present, sbt does not ask Ivy to perform resolution."

    When you run run, this is generally the situation. If you have not modified the configuration since the last successful update or run update directly from the command line, there should be zero (< 20 ms) time spent in update.

  2. "Changing the configuration, such as adding or removing dependencies or changing the version or other attributes of a dependency, will automatically cause resolution to be performed."

    Changing version changes the project's dependency configuration, so it is expected that update reruns in this case.

  3. "Directly running the update task (as opposed to a task that depends on it) will force resolution to run, whether or not configuration changed. This should be done in order to refresh remote SNAPSHOT dependencies."

    The SNAPSHOT part also applies to version ranges, since they are both dynamic revisions. Again, this situation should not apply to running run, however. This is only when explicitly running update from the command line. If you are seeing dependencies getting resolved when running run (after a previous run with no configuration changes, etc...), that's unexpected.

Just a comment on "No idea why reading a tiny XML file on the hard drive would take that long." I've previously profiled reading from the cache and it takes a negligible amount of time. If there is no network access, it is very likely the time is spent in pure computation. (In theory, dependency resolution is NP-hard. I'd be surprised if that is the root problem here, though.)

Also, what I meant by "number of ultimately selected dependencies" is that you might transitively visit multiple versions for each module and the dependency manager has to resolve those different versions. Depending on the graph, the total number of visited nodes might be much larger than the final number of nodes.

Member

harrah commented Nov 26, 2013

I believe you've seen http://www.scala-sbt.org/release/docs/Detailed-Topics/Dependency-Management-Flow.html#caching-and-configuration, so to apply some of the points from that section to what you are saying:

  1. "Normally, if no dependency management configuration has changed since the last successful resolution and the retrieved files are still present, sbt does not ask Ivy to perform resolution."

    When you run run, this is generally the situation. If you have not modified the configuration since the last successful update or run update directly from the command line, there should be zero (< 20 ms) time spent in update.

  2. "Changing the configuration, such as adding or removing dependencies or changing the version or other attributes of a dependency, will automatically cause resolution to be performed."

    Changing version changes the project's dependency configuration, so it is expected that update reruns in this case.

  3. "Directly running the update task (as opposed to a task that depends on it) will force resolution to run, whether or not configuration changed. This should be done in order to refresh remote SNAPSHOT dependencies."

    The SNAPSHOT part also applies to version ranges, since they are both dynamic revisions. Again, this situation should not apply to running run, however. This is only when explicitly running update from the command line. If you are seeing dependencies getting resolved when running run (after a previous run with no configuration changes, etc...), that's unexpected.

Just a comment on "No idea why reading a tiny XML file on the hard drive would take that long." I've previously profiled reading from the cache and it takes a negligible amount of time. If there is no network access, it is very likely the time is spent in pure computation. (In theory, dependency resolution is NP-hard. I'd be surprised if that is the root problem here, though.)

Also, what I meant by "number of ultimately selected dependencies" is that you might transitively visit multiple versions for each module and the dependency manager has to resolve those different versions. Depending on the graph, the total number of visited nodes might be much larger than the final number of nodes.

@brikis98

This comment has been minimized.

Show comment
Hide comment
@brikis98

brikis98 Nov 26, 2013

Ah, I think point number 2 about the version changes may explain the behavior we're seeing. Based on the strange way we publish artifacts internally, we automatically bump the version number for each checkin; therefore, it changes very frequently when you update your git/svn repo. Is there an easy way to skip that?

I can't imagine what computation it could possibly be doing that takes multiple 2-3 second pauses while resolving a few hundred items. We should probably bust out a profiler on our end and see where we're spending our time.

brikis98 commented Nov 26, 2013

Ah, I think point number 2 about the version changes may explain the behavior we're seeing. Based on the strange way we publish artifacts internally, we automatically bump the version number for each checkin; therefore, it changes very frequently when you update your git/svn repo. Is there an easy way to skip that?

I can't imagine what computation it could possibly be doing that takes multiple 2-3 second pauses while resolving a few hundred items. We should probably bust out a profiler on our end and see where we're spending our time.

@harrah

This comment has been minimized.

Show comment
Hide comment
@harrah

harrah Nov 26, 2013

Member

I apologize for quoting almost the whole linked section at this point, but I'd probably summarize it less clearly or miss something:

"5. Overriding all of the above, skip in update := true will tell sbt to never perform resolution. Note that this can cause dependent tasks to fail. For example, compilation may fail if jars have been deleted from the cache (and so needed classes are missing) or a dependency has been added (but will not be resolved because skip is true). Also, update itself will immediately fail if resolution has not been allowed to run since the last clean."

More control over caching, such as never updating based on version but updating on every other change, would require changes to sbt.

Member

harrah commented Nov 26, 2013

I apologize for quoting almost the whole linked section at this point, but I'd probably summarize it less clearly or miss something:

"5. Overriding all of the above, skip in update := true will tell sbt to never perform resolution. Note that this can cause dependent tasks to fail. For example, compilation may fail if jars have been deleted from the cache (and so needed classes are missing) or a dependency has been added (but will not be resolved because skip is true). Also, update itself will immediately fail if resolution has not been allowed to run since the last clean."

More control over caching, such as never updating based on version but updating on every other change, would require changes to sbt.

@dk14

This comment has been minimized.

Show comment
Hide comment
@dk14

dk14 Nov 28, 2013

There is a lot of time wasted on ivy's resolveConflicts() execution for my project. I used custom resolver to cache resolved dependencies - updating time decreased by only 10-20%. So, I believe, Ivy spent most of time analyzing dependency graph conflicts after resolution.

dk14 commented Nov 28, 2013

There is a lot of time wasted on ivy's resolveConflicts() execution for my project. I used custom resolver to cache resolved dependencies - updating time decreased by only 10-20%. So, I believe, Ivy spent most of time analyzing dependency graph conflicts after resolution.

@jrudolph

This comment has been minimized.

Show comment
Hide comment
@jrudolph

jrudolph Nov 28, 2013

Member

There was (at least) one notable appearance of version ranges. It was a common dependency which transitively depended on jackson with a version range:

"com.codahale" % "jerkson_2.9.1" % "0.5.0"

http://repo.codahale.com/com/codahale/jerkson_2.9.1/0.5.0/jerkson_2.9.1-0.5.0.pom

I've seen this dependency in many projects and it attributed a big share of the total time sbt spent during update. Another common one is commons-logging. This is especially bad when having a big multi-module project where every single subproject retries to find out what the concrete version for this version range is. Even worse: having to use a VPN to access a private repository mirror decreasing roundtrip times even more.

From my experience sbt works as @harrah says. However, there are those remaining problems:

  • caching helps less if you have a workflow that includes switching branches often which makes it more likely that details of the dependency configuration change (hard to solve?, possible solution: cache multiple update results by dpendency configuration?)
  • finding out the most recent version for a version range happens for each subproject anew (possible solution: cache the results of those requests, e.g. by registering a custom org.apache.ivy.util.url.URLHandler that intercepts and caches version listing (or any?) requests)
Member

jrudolph commented Nov 28, 2013

There was (at least) one notable appearance of version ranges. It was a common dependency which transitively depended on jackson with a version range:

"com.codahale" % "jerkson_2.9.1" % "0.5.0"

http://repo.codahale.com/com/codahale/jerkson_2.9.1/0.5.0/jerkson_2.9.1-0.5.0.pom

I've seen this dependency in many projects and it attributed a big share of the total time sbt spent during update. Another common one is commons-logging. This is especially bad when having a big multi-module project where every single subproject retries to find out what the concrete version for this version range is. Even worse: having to use a VPN to access a private repository mirror decreasing roundtrip times even more.

From my experience sbt works as @harrah says. However, there are those remaining problems:

  • caching helps less if you have a workflow that includes switching branches often which makes it more likely that details of the dependency configuration change (hard to solve?, possible solution: cache multiple update results by dpendency configuration?)
  • finding out the most recent version for a version range happens for each subproject anew (possible solution: cache the results of those requests, e.g. by registering a custom org.apache.ivy.util.url.URLHandler that intercepts and caches version listing (or any?) requests)
@manuelbernhardt

This comment has been minimized.

Show comment
Hide comment
@manuelbernhardt

manuelbernhardt Dec 2, 2013

+1 on @jrudolph 's comment about switching branches: indeed, even minor changes in the build configuration / dependency configuration trigger a completely new resolution process, having multiple dependency configurations being cached would be very much welcome, especially for larger multi-module builds.

One thing I wonder about is whether these caches survive a reload (or even, leaving SBT) - since switching a branch with dependency changes more often than not requires a reload.

manuelbernhardt commented Dec 2, 2013

+1 on @jrudolph 's comment about switching branches: indeed, even minor changes in the build configuration / dependency configuration trigger a completely new resolution process, having multiple dependency configurations being cached would be very much welcome, especially for larger multi-module builds.

One thing I wonder about is whether these caches survive a reload (or even, leaving SBT) - since switching a branch with dependency changes more often than not requires a reload.

@tutufool

This comment has been minimized.

Show comment
Hide comment
@tutufool

tutufool Mar 26, 2014

To harrah:

I'd suggest you to try to add a module with about 10 dependencies and copy this module for 99 times

When you try to run update from the root, those "exactly the same" 100 modules will take a long time.

From my understanding, running update at the root level should be something like open a session which is stateful. It should be able to remember/cache something when processing modules from 1 to 100.

But now, it seems that the root level update just trigger each module's update 1 by 1, stateless.

tutufool commented Mar 26, 2014

To harrah:

I'd suggest you to try to add a module with about 10 dependencies and copy this module for 99 times

When you try to run update from the root, those "exactly the same" 100 modules will take a long time.

From my understanding, running update at the root level should be something like open a session which is stateful. It should be able to remember/cache something when processing modules from 1 to 100.

But now, it seems that the root level update just trigger each module's update 1 by 1, stateless.

@jsuereth

This comment has been minimized.

Show comment
Hide comment
@jsuereth

jsuereth Mar 26, 2014

Member

Right now we aren't sharing the in-memory cache between projects. I have an experimental branch which does this, but haven't evaluated the improvement yet: https://github.com/sbt/sbt/tree/wip/shared-memory-cache-ivy

Feel free to try that branch out and see if you notice a speed up. Most likely, stabilizing that branch will loose some of the speed up, but I'm curious if you notice anything.

My $$$ right now is on some kind of horrid performance when Ivy reads values out of its own cache. Some combination of flags has caused us to ALWAYS re-resolve.

Member

jsuereth commented Mar 26, 2014

Right now we aren't sharing the in-memory cache between projects. I have an experimental branch which does this, but haven't evaluated the improvement yet: https://github.com/sbt/sbt/tree/wip/shared-memory-cache-ivy

Feel free to try that branch out and see if you notice a speed up. Most likely, stabilizing that branch will loose some of the speed up, but I'm curious if you notice anything.

My $$$ right now is on some kind of horrid performance when Ivy reads values out of its own cache. Some combination of flags has caused us to ALWAYS re-resolve.

@tutufool

This comment has been minimized.

Show comment
Hide comment
@tutufool

tutufool Mar 27, 2014

Hi, Josh

I'm very new to sbt, could you give some instructions/wikis on how to build a sbt deliverable for your branch wip/shared-memory-cache-ivy, so I can give it a try.

Thanks

tutufool commented Mar 27, 2014

Hi, Josh

I'm very new to sbt, could you give some instructions/wikis on how to build a sbt deliverable for your branch wip/shared-memory-cache-ivy, so I can give it a try.

Thanks

@tutufool

This comment has been minimized.

Show comment
Hide comment
@tutufool

tutufool Mar 27, 2014

Btw, from my observation, the bottleneck is org.apache.ivy.core.resolve.ResolveData.mediate

I saw debug messages like below: (exactly match)

dependency descriptor has been mediated: dependency: log4j#log4j;1.2.17 {compile=[compile(), master()], runtime=[runtime()]} => dependency: log4j#log4j;1.2.13 {compile=[compile(), master()], runtime=[runtime()]}

For over 11000 times in my update log.

And:

dependency descriptor has been mediated: dependency: commons-lang#commons-lang;2.6 {compile=[compile(), master()], runtime=[runtime()]} => dependency: commons-lang#commons-lang;2.5 {compile=[compile(), master()], runtime=[runtime()]

for around 3000 times.

The deeper node seems to have more chances to get re-mediate.

Is there any cache to keep the mediate result like from DependencyDescriptor DD1 to DD2 within some certain context?

tutufool commented Mar 27, 2014

Btw, from my observation, the bottleneck is org.apache.ivy.core.resolve.ResolveData.mediate

I saw debug messages like below: (exactly match)

dependency descriptor has been mediated: dependency: log4j#log4j;1.2.17 {compile=[compile(), master()], runtime=[runtime()]} => dependency: log4j#log4j;1.2.13 {compile=[compile(), master()], runtime=[runtime()]}

For over 11000 times in my update log.

And:

dependency descriptor has been mediated: dependency: commons-lang#commons-lang;2.6 {compile=[compile(), master()], runtime=[runtime()]} => dependency: commons-lang#commons-lang;2.5 {compile=[compile(), master()], runtime=[runtime()]

for around 3000 times.

The deeper node seems to have more chances to get re-mediate.

Is there any cache to keep the mediate result like from DependencyDescriptor DD1 to DD2 within some certain context?

@jsuereth

This comment has been minimized.

Show comment
Hide comment
@jsuereth

jsuereth Mar 27, 2014

Member

I'm not sure. The cache fix I have is for the descriptor, but I think
perhaps the wrong one. That's some really good information for us to dig
into though, thanks.

For using sbt locally:

Checkout that branch:

sbt> publishLocal

Then in your project/build.properties, set the version to 0.13.2-SNAPSHOT
(or whatever you just pushed). Note: You'll have to clean your
~/.sbt/boot directory every time you push a new -SNAPSHOT version. We
don't have -SNPSHOT working in the launcher.

On Wed, Mar 26, 2014 at 10:21 PM, tutufool notifications@github.com wrote:

Btw, from my observation, the bottleneck is
org.apache.ivy.core.resolve.ResolveData.mediate

I saw debug messages like below: (exactly match)

dependency descriptor has been mediated: dependency: log4j#log4j;1.2.17
{compile=[compile(), master()], runtime=[runtime()]} => dependency:
log4j#log4j;1.2.13 {compile=[compile(
), master()], runtime=[runtime()]}

For over 11000 times in my update log.

And:

dependency descriptor has been mediated: dependency:
commons-lang#commons-lang;2.6 {compile=[compile(), master()],
runtime=[runtime()]} => dependency: commons-lang#commons-lang;2.5
{compile=[compile(
), master()], runtime=[runtime()]

for around 3000 times.

The deeper node seems to have more chances to get re-mediate.

Is there any cache to keep the mediate result like from
DependencyDescriptor DD1 to DD2 within some certain context?

Reply to this email directly or view it on GitHubhttps://github.com/sbt/sbt/issues/413#issuecomment-38762853
.

Member

jsuereth commented Mar 27, 2014

I'm not sure. The cache fix I have is for the descriptor, but I think
perhaps the wrong one. That's some really good information for us to dig
into though, thanks.

For using sbt locally:

Checkout that branch:

sbt> publishLocal

Then in your project/build.properties, set the version to 0.13.2-SNAPSHOT
(or whatever you just pushed). Note: You'll have to clean your
~/.sbt/boot directory every time you push a new -SNAPSHOT version. We
don't have -SNPSHOT working in the launcher.

On Wed, Mar 26, 2014 at 10:21 PM, tutufool notifications@github.com wrote:

Btw, from my observation, the bottleneck is
org.apache.ivy.core.resolve.ResolveData.mediate

I saw debug messages like below: (exactly match)

dependency descriptor has been mediated: dependency: log4j#log4j;1.2.17
{compile=[compile(), master()], runtime=[runtime()]} => dependency:
log4j#log4j;1.2.13 {compile=[compile(
), master()], runtime=[runtime()]}

For over 11000 times in my update log.

And:

dependency descriptor has been mediated: dependency:
commons-lang#commons-lang;2.6 {compile=[compile(), master()],
runtime=[runtime()]} => dependency: commons-lang#commons-lang;2.5
{compile=[compile(
), master()], runtime=[runtime()]

for around 3000 times.

The deeper node seems to have more chances to get re-mediate.

Is there any cache to keep the mediate result like from
DependencyDescriptor DD1 to DD2 within some certain context?

Reply to this email directly or view it on GitHubhttps://github.com/sbt/sbt/issues/413#issuecomment-38762853
.

@eed3si9n eed3si9n added the Bug label Mar 28, 2014

@jsuereth jsuereth added the Bug label Mar 28, 2014

@jsuereth jsuereth added this to the 0.13.5 milestone Mar 28, 2014

@tutufool

This comment has been minimized.

Show comment
Hide comment
@tutufool

tutufool Apr 1, 2014

1 more finding:

DefaultRepositoryCacheManager will be created many times across modules, hence ModuleDescriptorMemoryCache can't be re-used across modules.

Does that mean if multiple modules depends on artifact A, A's ivy.xml will be parsed multiple times?

tutufool commented Apr 1, 2014

1 more finding:

DefaultRepositoryCacheManager will be created many times across modules, hence ModuleDescriptorMemoryCache can't be re-used across modules.

Does that mean if multiple modules depends on artifact A, A's ivy.xml will be parsed multiple times?

@charleso

This comment has been minimized.

Show comment
Hide comment
@charleso

charleso Aug 1, 2014

@eed3si9n Not sure if this helps but I've just tried updating to 0.13.6-M1 and our resolution times of ~60s looks roughly the same, if not worse:

https://github.com/ambiata/ivory

charleso commented Aug 1, 2014

@eed3si9n Not sure if this helps but I've just tried updating to 0.13.6-M1 and our resolution times of ~60s looks roughly the same, if not worse:

https://github.com/ambiata/ivory

@eed3si9n

This comment has been minimized.

Show comment
Hide comment
@eed3si9n

eed3si9n Aug 1, 2014

Member

@charleso Consolidated resolution is not enabled by default. See 0.13.6.md for details. Currently this is how you enable it (This is M1 so the following is subject to change):

updateOptions := updateOptions.value.withConsolidatedResolution(true)

Before it took 69s to update. After 40s ymmv.

Member

eed3si9n commented Aug 1, 2014

@charleso Consolidated resolution is not enabled by default. See 0.13.6.md for details. Currently this is how you enable it (This is M1 so the following is subject to change):

updateOptions := updateOptions.value.withConsolidatedResolution(true)

Before it took 69s to update. After 40s ymmv.

@charleso

This comment has been minimized.

Show comment
Hide comment
@charleso

charleso Aug 2, 2014

@eed3si9n My sincere apologies, that's my second dumb thing I've asked you about this week.

Works like a charm! Getting about a 35-45% speed up. Thanks so much.

charleso commented Aug 2, 2014

@eed3si9n My sincere apologies, that's my second dumb thing I've asked you about this week.

Works like a charm! Getting about a 35-45% speed up. Thanks so much.

@magro

This comment has been minimized.

Show comment
Hide comment
@magro

magro Aug 2, 2014

For us update time hasn't changed (~1'30), having 5 modules with different dependencies.

magro commented Aug 2, 2014

For us update time hasn't changed (~1'30), having 5 modules with different dependencies.

@fkoehler

This comment has been minimized.

Show comment
Hide comment
@fkoehler

fkoehler Aug 13, 2014

It is faster for us if you are in the same SBT session but not when started directly from the console.
Running sbt update from the console always does:

[info] Consolidating managed dependencies to org.scala-sbt.temp#temp-resolve-7d5c97c90f4b243ad4f7331f2ce8cd4f20cbd8c6;1.0 ..

Being in a running SBT session I get

[info] Found consolidated dependency org.scala-sbt.temp#temp-resolve-7d5c97c90f4b243ad4f7331f2ce8cd4f20cbd8c6;1.0 ...

afterwards which gives me a 6 second speed up after the first run. Pretty cool.

Is that expected behaviour or should we also get that speed with a fresh SBT boot?

fkoehler commented Aug 13, 2014

It is faster for us if you are in the same SBT session but not when started directly from the console.
Running sbt update from the console always does:

[info] Consolidating managed dependencies to org.scala-sbt.temp#temp-resolve-7d5c97c90f4b243ad4f7331f2ce8cd4f20cbd8c6;1.0 ..

Being in a running SBT session I get

[info] Found consolidated dependency org.scala-sbt.temp#temp-resolve-7d5c97c90f4b243ad4f7331f2ce8cd4f20cbd8c6;1.0 ...

afterwards which gives me a 6 second speed up after the first run. Pretty cool.

Is that expected behaviour or should we also get that speed with a fresh SBT boot?

@jsuereth

This comment has been minimized.

Show comment
Hide comment
@jsuereth

jsuereth Aug 13, 2014

Member

@fkoehler That's expected behavior. The second run we can completely skip resolution because the in-memory cache finds the previous resolved values.

Additonally, running update directly, ever, forces the FIRST cache to be ignored, which means you hit ivy. What you're seeing now is the inter-project resolution cache where we try to force ivy to share resolution graphs between things.

@eed3si9n This may violate sbt's notion of "update" should force hitting the raw ivy again, but I'm not sure.

Member

jsuereth commented Aug 13, 2014

@fkoehler That's expected behavior. The second run we can completely skip resolution because the in-memory cache finds the previous resolved values.

Additonally, running update directly, ever, forces the FIRST cache to be ignored, which means you hit ivy. What you're seeing now is the inter-project resolution cache where we try to force ivy to share resolution graphs between things.

@eed3si9n This may violate sbt's notion of "update" should force hitting the raw ivy again, but I'm not sure.

@godenji

This comment has been minimized.

Show comment
Hide comment
@godenji

godenji Oct 12, 2014

So, I've tried update on 0.13.6 withConsolidatedResolution and withLatestSnapshots settings, times are roughly the same, perhaps slightly faster.

I am building Play's stable 2.3 branch from source (not 2.4 master), which is tagged as a snapshot build. Are the resolution time improvements exclusive to non-snapshot projects?

The best fix in this thread has been aggregate in update := false, at least there the dependency resolution marathon only kicks in if dep(s) have in fact changed.

godenji commented Oct 12, 2014

So, I've tried update on 0.13.6 withConsolidatedResolution and withLatestSnapshots settings, times are roughly the same, perhaps slightly faster.

I am building Play's stable 2.3 branch from source (not 2.4 master), which is tagged as a snapshot build. Are the resolution time improvements exclusive to non-snapshot projects?

The best fix in this thread has been aggregate in update := false, at least there the dependency resolution marathon only kicks in if dep(s) have in fact changed.

@anquegi

This comment has been minimized.

Show comment
Hide comment
@anquegi

anquegi Aug 4, 2015

I also have the issue with common-logs dependency trying to use ensime with my scala, play projects, I always need to remove the ivy/cache/commons-log and then all goes wee

anquegi commented Aug 4, 2015

I also have the issue with common-logs dependency trying to use ensime with my scala, play projects, I always need to remove the ivy/cache/commons-log and then all goes wee

@jamesjieye

This comment has been minimized.

Show comment
Hide comment
@jamesjieye

jamesjieye Aug 13, 2015

It's like been waiting for days before sbt finishes resolving dependencies. This is really a wasting of time.

jamesjieye commented Aug 13, 2015

It's like been waiting for days before sbt finishes resolving dependencies. This is really a wasting of time.

@dk14

This comment has been minimized.

Show comment
Hide comment
@dk14

dk14 Aug 13, 2015

I've solved that by watching cats on youtube during resolution: https://www.youtube.com/watch?v=UIrEM_9qvZU.

dk14 commented Aug 13, 2015

I've solved that by watching cats on youtube during resolution: https://www.youtube.com/watch?v=UIrEM_9qvZU.

@johanandren

This comment has been minimized.

Show comment
Hide comment
@johanandren

johanandren Aug 13, 2015

Contributor

LOL. "Close: Solved"

Contributor

johanandren commented Aug 13, 2015

LOL. "Close: Solved"

@lucamolteni

This comment has been minimized.

Show comment
Hide comment
@lucamolteni

lucamolteni Sep 30, 2015

Is this really closed?

lucamolteni commented Sep 30, 2015

Is this really closed?

@eed3si9n

This comment has been minimized.

Show comment
Hide comment
@eed3si9n

eed3si9n Sep 30, 2015

Member

The original case states:

So I have a multi-module project - dozens of modules - in which many of the modules have very similar (transitive) dependencies.

This particular concern is now solved using cached resolution feature. If anyone has further issues with dependency resolution performance, please provide more context.

Member

eed3si9n commented Sep 30, 2015

The original case states:

So I have a multi-module project - dozens of modules - in which many of the modules have very similar (transitive) dependencies.

This particular concern is now solved using cached resolution feature. If anyone has further issues with dependency resolution performance, please provide more context.

@fommil

This comment has been minimized.

Show comment
Hide comment
@fommil

fommil Oct 1, 2015

I'm still seeing this being very slow in ensime-sbt and the intellij equivalent. Use ensime-server as an example (e.g. type gen-ensime). I'd expect the resolution of classifiers (sources/javadocs) to be a lot faster. This is a bottleneck for many projects enabling auto-update on their project.

fommil commented Oct 1, 2015

I'm still seeing this being very slow in ensime-sbt and the intellij equivalent. Use ensime-server as an example (e.g. type gen-ensime). I'd expect the resolution of classifiers (sources/javadocs) to be a lot faster. This is a bottleneck for many projects enabling auto-update on their project.

@pauldraper

This comment has been minimized.

Show comment
Hide comment
@pauldraper

pauldraper Oct 24, 2015

Contributor

Yeah, IntelliJ is still slow. But that's probably a bug on their side.

Contributor

pauldraper commented Oct 24, 2015

Yeah, IntelliJ is still slow. But that's probably a bug on their side.

@fommil

This comment has been minimized.

Show comment
Hide comment
@fommil

fommil Oct 24, 2015

@pauldraper nope, I think that is calling straight out to sbt. Same as our ensime generator.

fommil commented Oct 24, 2015

@pauldraper nope, I think that is calling straight out to sbt. Same as our ensime generator.

@valter-silva-git

This comment has been minimized.

Show comment
Hide comment
@valter-silva-git

valter-silva-git Nov 17, 2015

Still super slow a simple command like sbt update. Why is that ?

valter-silva-git commented Nov 17, 2015

Still super slow a simple command like sbt update. Why is that ?

@danyaljj

This comment has been minimized.

Show comment
Hide comment
@danyaljj

danyaljj Dec 23, 2015

Painfully slow ...

danyaljj commented Dec 23, 2015

Painfully slow ...

@fkirill

This comment has been minimized.

Show comment
Hide comment
@fkirill

fkirill Jan 21, 2016

Almost 4 years passed... Look guys, are you serious about SBT at all?

fkirill commented Jan 21, 2016

Almost 4 years passed... Look guys, are you serious about SBT at all?

@cmoro-deusto

This comment has been minimized.

Show comment
Hide comment
@cmoro-deusto

cmoro-deusto Feb 12, 2016

Does this CLOSED status mean that we should expect this to be this slow forever? Please don't get me wrong: I'm really grateful for SBT, Play Framework and the rest of the reactive platform... but guys, this is not reactive at all.

cmoro-deusto commented Feb 12, 2016

Does this CLOSED status mean that we should expect this to be this slow forever? Please don't get me wrong: I'm really grateful for SBT, Play Framework and the rest of the reactive platform... but guys, this is not reactive at all.

@eed3si9n

This comment has been minimized.

Show comment
Hide comment
@eed3si9n

eed3si9n Feb 12, 2016

Member

It's closed because I've added cached resolution feature - http://www.scala-sbt.org/0.13/docs/Cached-Resolution.html

Member

eed3si9n commented Feb 12, 2016

It's closed because I've added cached resolution feature - http://www.scala-sbt.org/0.13/docs/Cached-Resolution.html

@DBassel

This comment has been minimized.

Show comment
Hide comment
@DBassel

DBassel May 20, 2016

+1 on SBT 0.13.11 both multi-project builds and single project

DBassel commented May 20, 2016

+1 on SBT 0.13.11 both multi-project builds and single project

@pauldraper

This comment has been minimized.

Show comment
Hide comment
@pauldraper

pauldraper May 20, 2016

Contributor
updateOptions := updateOptions.value.withCachedResolution(false)

vs

updateOptions := updateOptions.value.withCachedResolution(true)

is a night-and-day difference in multi-module projects.

I agree that resolution is still slower than necessary (particularly the first time), but that is really a whole other issue. Single-threaded resolution in a "parallel by default" build system is beyond strange, but that really can only be fixed by overhauling/replacing Ivy. And that would be enormously difficult now that SBT now needs both Ivy and Maven compatibility.

In any case, any further performance improvements should be a new issue or mailing topic.

Contributor

pauldraper commented May 20, 2016

updateOptions := updateOptions.value.withCachedResolution(false)

vs

updateOptions := updateOptions.value.withCachedResolution(true)

is a night-and-day difference in multi-module projects.

I agree that resolution is still slower than necessary (particularly the first time), but that is really a whole other issue. Single-threaded resolution in a "parallel by default" build system is beyond strange, but that really can only be fixed by overhauling/replacing Ivy. And that would be enormously difficult now that SBT now needs both Ivy and Maven compatibility.

In any case, any further performance improvements should be a new issue or mailing topic.

@abrighton

This comment has been minimized.

Show comment
Hide comment
@abrighton

abrighton Jul 16, 2016

What if you need to use SNAPSHOT dependencies because you have multiple sbt projects that are not in a single multi-project build? The dependency resolution is very slow, caching won't work, and even setting offline := true does not work, since sbt (or ivy) goes online anyway...

abrighton commented Jul 16, 2016

What if you need to use SNAPSHOT dependencies because you have multiple sbt projects that are not in a single multi-project build? The dependency resolution is very slow, caching won't work, and even setting offline := true does not work, since sbt (or ivy) goes online anyway...

@achieverprince

This comment has been minimized.

Show comment
Hide comment
@achieverprince

achieverprince Sep 25, 2017

I am new to scala, wanted to see what is it.
Tried to import play-scala-forms-example, intellij is resolving dependency for last 30 min.
I think something wrong with it...

achieverprince commented Sep 25, 2017

I am new to scala, wanted to see what is it.
Tried to import play-scala-forms-example, intellij is resolving dependency for last 30 min.
I think something wrong with it...

@mohsenno1

This comment has been minimized.

Show comment
Hide comment
@mohsenno1

mohsenno1 Mar 27, 2018

It's really annoying.
I think the server firewall limits the download speed to 256 Kbps. I have 500Mbps connection but the download speed never exceed 256 kbps!!!!

Now I'm waiting about 1 hour for a new project to be initialized.

mohsenno1 commented Mar 27, 2018

It's really annoying.
I think the server firewall limits the download speed to 256 Kbps. I have 500Mbps connection but the download speed never exceed 256 kbps!!!!

Now I'm waiting about 1 hour for a new project to be initialized.

@achieverprince

This comment has been minimized.

Show comment
Hide comment
@achieverprince

achieverprince Mar 28, 2018

Looks like issue is open for 6 years

achieverprince commented Mar 28, 2018

Looks like issue is open for 6 years

@mohsenno1

This comment has been minimized.

Show comment
Hide comment
@mohsenno1

mohsenno1 Mar 28, 2018

I gave up learn ScalaJs. I found another alternative. Flutter from google. Let's give it a try.

mohsenno1 commented Mar 28, 2018

I gave up learn ScalaJs. I found another alternative. Flutter from google. Let's give it a try.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment