Cached resolution: OOM when compiling #1763

ornicar · 2014-12-04T16:18:13Z

With -Xmx1G. OOM does not happen with 0.13.7 without CachedResolution

Reproduce:

git clone -b cachedresolution https://github.com/ornicar/lila.git
cd lila
git submodule update --init modules/chess
./bin/build-deps.sh
sbt compile

[info] Loading global plugins from /home/thib/dotfiles/sbt/plugins
[info] Updating {file:/home/thib/.sbt/0.13/plugins/}global-plugins...
[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[info] Done updating.
[info] Loading project definition from /home/thib/tmp/lila/project
[info] Updating {file:/home/thib/tmp/lila/project/}lila-build...
[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[info] Done updating.
[info] Compiling 3 Scala sources to /home/thib/tmp/lila/project/target/scala-2.10/sbt-0.13/classes...
[warn] there were 1 deprecation warning(s); re-run with -deprecation for details
[warn] one warning found
[info] Set current project to lila (in build file:/home/thib/tmp/lila/)
[info] Updating {file:/home/thib/tmp/lila/}common...
[info] Updating {file:/home/thib/tmp/lila/}chess...
[info] Updating {file:/home/thib/tmp/lila/}swisssystem...
[info] Done updating.
[info] Done updating.
[info] Done updating.
[info] Updating {file:/home/thib/tmp/lila/}db...
[info] Updating {file:/home/thib/tmp/lila/}hub...
[info] Updating {file:/home/thib/tmp/lila/}memo...
[info] Done updating.
[info] Done updating.
[info] Done updating.
[warn] There may be incompatibilities among your library dependencies.
[info] Updating {file:/home/thib/tmp/lila/}rating...
[info] Updating {file:/home/thib/tmp/lila/}coordinate...
[warn] Here are some of the libraries that were evicted:
[warn]  * com.google.guava:guava:16.0.1 -> 18.0
[warn] Run 'evicted' to see detailed eviction warnings
[info] Updating {file:/home/thib/tmp/lila/}wiki...
[info] Compiling 22 Scala sources to /home/thib/tmp/lila/modules/common/target/scala-2.11/classes...
[info] Done updating.
[info] Updating {file:/home/thib/tmp/lila/}search...
[info] Done updating.
[info] Updating {file:/home/thib/tmp/lila/}socket...
[info] Done updating.
[info] Updating {file:/home/thib/tmp/lila/}worldMap...
[info] Done updating.
[info] Updating {file:/home/thib/tmp/lila/}user...
[info] Updating {file:/home/thib/tmp/lila/}monitor...
[info] Done updating.
[info] Updating {file:/home/thib/tmp/lila/}site...
[info] Done updating.
[info] Done updating.
[info] Done updating.
[info] Updating {file:/home/thib/tmp/lila/}donation...
[info] Updating {file:/home/thib/tmp/lila/}i18n...
[info] Updating {file:/home/thib/tmp/lila/}pref...
[info] Done updating.
[info] Updating {file:/home/thib/tmp/lila/}notification...
[info] Updating {file:/home/thib/tmp/lila/}puzzle...
[info] Updating {file:/home/thib/tmp/lila/}security...
[info] Done updating.
[info] Done updating.
[info] Updating {file:/home/thib/tmp/lila/}report...
[info] Done updating.
[info] Done updating.
[info] Done updating.
[info] Done updating.
[info] Updating {file:/home/thib/tmp/lila/}mod...
[info] Updating {file:/home/thib/tmp/lila/}qa...
[info] Done updating.
[info] Done updating.
[info] Done updating.
[info] Updating {file:/home/thib/tmp/lila/}chat...
[info] Done updating.
[info] Updating {file:/home/thib/tmp/lila/}forum...
[info] Updating {file:/home/thib/tmp/lila/}game...
[info] Done updating.
[info] Done updating.
[info] Updating {file:/home/thib/tmp/lila/}team...
[info] Updating {file:/home/thib/tmp/lila/}forumSearch...
[info] Updating {file:/home/thib/tmp/lila/}bookmark...
[info] Updating {file:/home/thib/tmp/lila/}relation...
[info] Updating {file:/home/thib/tmp/lila/}tv...
[info] Updating {file:/home/thib/tmp/lila/}history...
[info] Updating {file:/home/thib/tmp/lila/}gameSearch...
[info] Done updating.
[info] Done updating.
[info] Done updating.
[info] Updating {file:/home/thib/tmp/lila/}analyse...
[info] Updating {file:/home/thib/tmp/lila/}evaluation...
[info] Done updating.
[info] Updating {file:/home/thib/tmp/lila/}teamSearch...
[info] Done updating.
[info] Done updating.
[info] Done updating.
[info] Updating {file:/home/thib/tmp/lila/}timeline...
[info] Updating {file:/home/thib/tmp/lila/}message...
[info] Done updating.
[info] Done updating.
[info] Done updating.
[info] Done updating.
[info] Updating {file:/home/thib/tmp/lila/}ai...
[info] Done updating.
[info] Updating {file:/home/thib/tmp/lila/}blog...
[info] Done updating.
[info] Done updating.
[info] Updating {file:/home/thib/tmp/lila/}round...
[info] Done updating.
[info] Updating {file:/home/thib/tmp/lila/}importer...
[info] Updating {file:/home/thib/tmp/lila/}lobby...
[info] Updating {file:/home/thib/tmp/lila/}tournament...
[info] Done updating.
[info] Done updating.
[info] Done updating.
[info] Updating {file:/home/thib/tmp/lila/}setup...
[info] Done updating.
[info] Compiling 8 Scala sources to /home/thib/tmp/lila/modules/hub/target/scala-2.11/classes...
[info] Compiling 26 Scala sources to /home/thib/tmp/lila/modules/db/target/scala-2.11/classes...
[info] Compiling 5 Scala sources to /home/thib/tmp/lila/modules/memo/target/scala-2.11/classes...
java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space
        at java.util.concurrent.FutureTask.report(FutureTask.java:122)
        at java.util.concurrent.FutureTask.get(FutureTask.java:192)
        at sbt.ConcurrentRestrictions$$anon$4.take(ConcurrentRestrictions.scala:188)
        at sbt.Execute.next$1(Execute.scala:83)
        at sbt.Execute.processAll(Execute.scala:86)
        at sbt.Execute.runKeep(Execute.scala:66)
        at sbt.EvaluateTask$.liftedTree1$1(EvaluateTask.scala:342)
        at sbt.EvaluateTask$.run$1(EvaluateTask.scala:341)
        at sbt.EvaluateTask$.runTask(EvaluateTask.scala:361)
        at sbt.Aggregation$$anonfun$3.apply(Aggregation.scala:64)
        at sbt.Aggregation$$anonfun$3.apply(Aggregation.scala:62)
        at sbt.EvaluateTask$.withStreams(EvaluateTask.scala:293)
        at sbt.Aggregation$.timedRun(Aggregation.scala:62)
        at sbt.Aggregation$.runTasks(Aggregation.scala:71)
        at sbt.Aggregation$$anonfun$applyTasks$1.apply(Aggregation.scala:32)
        at sbt.Aggregation$$anonfun$applyTasks$1.apply(Aggregation.scala:31)
        at sbt.Command$$anonfun$applyEffect$2$$anonfun$apply$3.apply(Command.scala:60)
        at sbt.Command$$anonfun$applyEffect$2$$anonfun$apply$3.apply(Command.scala:60)
        at sbt.Aggregation$$anonfun$evaluatingParser$4$$anonfun$apply$5.apply(Aggregation.scala:153)
        at sbt.Aggregation$$anonfun$evaluatingParser$4$$anonfun$apply$5.apply(Aggregation.scala:152)
        at sbt.Act$$anonfun$sbt$Act$$actParser0$1$$anonfun$sbt$Act$$anonfun$$evaluate$1$1$$anonfun$apply$10.apply(Act.scala:244)
        at sbt.Act$$anonfun$sbt$Act$$actParser0$1$$anonfun$sbt$Act$$anonfun$$evaluate$1$1$$anonfun$apply$10.apply(Act.scala:241)
        at sbt.Command$.process(Command.scala:92)
        at sbt.MainLoop$$anonfun$1$$anonfun$apply$1.apply(MainLoop.scala:98)
        at sbt.MainLoop$$anonfun$1$$anonfun$apply$1.apply(MainLoop.scala:98)
        at sbt.State$$anon$1.process(State.scala:184)
        at sbt.MainLoop$$anonfun$1.apply(MainLoop.scala:98)
        at sbt.MainLoop$$anonfun$1.apply(MainLoop.scala:98)
        at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:17)
        at sbt.MainLoop$.next(MainLoop.scala:98)
        at sbt.MainLoop$.run(MainLoop.scala:91)
        at sbt.MainLoop$$anonfun$runWithNewLog$1.apply(MainLoop.scala:70)
        at sbt.MainLoop$$anonfun$runWithNewLog$1.apply(MainLoop.scala:65)
        at sbt.Using.apply(Using.scala:24)
        at sbt.MainLoop$.runWithNewLog(MainLoop.scala:65)
        at sbt.MainLoop$.runAndClearLast(MainLoop.scala:48)
        at sbt.MainLoop$.runLoggedLoop(MainLoop.scala:32)
        at sbt.MainLoop$.runLogged(MainLoop.scala:24)
        at sbt.StandardMain$.runManaged(Main.scala:53)
        at sbt.xMain.run(Main.scala:28)
        at xsbt.boot.Launch$$anonfun$run$1.apply(Launch.scala:109)
        at xsbt.boot.Launch$.withContextLoader(Launch.scala:128)
        at xsbt.boot.Launch$.run(Launch.scala:109)
        at xsbt.boot.Launch$$anonfun$apply$1.apply(Launch.scala:35)
        at xsbt.boot.Launch$.launch(Launch.scala:117)
        at xsbt.boot.Launch$.apply(Launch.scala:18)
        at xsbt.boot.Boot$.runImpl(Boot.scala:41)
        at xsbt.boot.Boot$.main(Boot.scala:17)
        at xsbt.boot.Boot.main(Boot.scala)
Caused by: java.lang.OutOfMemoryError: Java heap space
        at java.util.Arrays.copyOf(Arrays.java:3236)
        at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:113)
        at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
        at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:140)
        at sbinary.JavaOutput.writeAll(io.scala:83)
        at sbinary.JavaUTF$StringFormat$.writes(javaprotocol.scala:199)
        at sbinary.JavaUTF$StringFormat$.writes(javaprotocol.scala:100)
        at sbinary.Operations$.write(operations.scala:10)
        at sbinary.BasicTypes$$anon$6.writes(standardtypes.scala:79)
        at sbinary.BasicTypes$$anon$6.writes(standardtypes.scala:67)
        at sbinary.Operations$.write(operations.scala:10)
        at sbinary.BasicTypes$$anon$4.writes(standardtypes.scala:34)
        at sbinary.BasicTypes$$anon$4.writes(standardtypes.scala:24)
        at sbinary.Operations$.write(operations.scala:10)
        at sbinary.Generic$$anon$5.writes(generic.scala:72)
        at sbinary.Operations$.write(operations.scala:10)
        at sbinary.BasicTypes$$anon$9.writes(standardtypes.scala:169)
        at sbinary.BasicTypes$$anon$9.writes(standardtypes.scala:154)
        at sbinary.Operations$.write(operations.scala:10)
        at sbinary.Generic$$anon$5.writes(generic.scala:72)
        at sbinary.Operations$.write(operations.scala:10)
        at sbinary.Generic$CollectionFormat$$anonfun$writes$1.apply(generic.scala:24)
        at sbinary.Generic$CollectionFormat$$anonfun$writes$1.apply(generic.scala:24)
        at scala.collection.immutable.List.foreach(List.scala:318)
        at sbinary.Generic$LengthEncoded.foreach(generic.scala:32)
        at sbinary.Generic$LengthEncoded.foreach(generic.scala:30)
        at sbinary.Generic$CollectionFormat.writes(generic.scala:24)
        at sbinary.Operations$.write(operations.scala:10)
        at sbinary.Generic$$anon$5.writes(generic.scala:72)
        at sbinary.Operations$.write(operations.scala:10)
        at sbinary.BasicTypes$$anon$20.writes(standardtypes.scala:747)
        at sbinary.BasicTypes$$anon$20.writes(standardtypes.scala:704)
[error] java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space
[error] Use 'last' for the full log.

The text was updated successfully, but these errors were encountered:

ornicar · 2014-12-05T09:17:38Z

BTW I think lila is an interesting project to test sbt with, for it has 46 more or less lousely coupled subprojects.

https://github.com/ornicar/lila/blob/master/project/Build.scala#L48-L262

eed3si9n · 2015-02-27T18:45:43Z

One possible explanation for more memory usage only when cached resolution is enabled is #1760.

easel · 2015-02-27T20:42:34Z

@ornicar I tried to pull down lila and reproduce the OOM but I'm missing a resolver for com.github.ornicar:scalalib_2.11:5.2. Could you perhaps add the necessary resolver or instructions on how to build it into the README so I can try to reproduce?

eed3si9n · 2015-03-03T21:01:44Z

I had to locally publish scalalib 5.1

$ git clone git@github.com:ornicar/scalalib.git
$ cd scalalib
$ git co 8fd6c3791623257962fa37eb50f4f5e9df4d15e7 -b v5.1
$ sbt publishLocal

eed3si9n · 2015-03-03T21:16:36Z

Oh and I had to do this:

$ git clone git@github.com:hrlqn/maxmind-geoip2-scala.git
$ cd maxmind-geoip2-scala
$ git co origin/feature/scala_2.11 -b feature/scala_2.11
$ sbt publishLocal

and change project/Dependencies.scala to:

  val maxmind = "com.sanoma.cda" %% "maxmind-geoip2-scala" % "1.3.3"

ornicar · 2015-03-03T23:02:02Z

Or you can run ./bin/build-deps.sh

https://github.com/ornicar/lila/blob/master/bin/build-deps.sh

easel · 2015-03-04T14:15:31Z

Thanks @ornicar, build-deps worked great. This is the same OOM I've been seeing in my project, I was able to reproduce it in 13.7 immediately. Let me know if there's anything I can do to help track down the issue.

Some heap optimization during merging too. YourKit showed that mergeOrganizationArtifactReports takes up huge amount of heap.

ornicar · 2015-03-10T20:32:47Z

Hurray! I can't wait to try it out. Did you manage to compile lila with these changes?

eed3si9n · 2015-03-10T21:39:39Z

This case got closed when I merged #1904, but OOM persists for lila using sbt 0.13.8-RC1.

On a build that used to OOM on update, one graph.json file for a project was growing to 250MB. 97% of the content seemed to be caller info. Parsing the json file from the REPL alone would take many hours. After removing the caller, update for the entire update finishes within minutes.

- On some of the builds graph.json is reaching 250MB+ - JSON parsing alone takes hours - 97% of the content are caller info - This change summarizes all callers into one (zero caller would have correctness issues)

Fixes #1721/#1763. Cached resolution: summarize callers in graph.json

eed3si9n · 2015-06-09T16:29:38Z

@ornicar sbt 0.13.9-M1 is out. Could you test if cached resolution works for you?
Locally the update completes in 19s.

ornicar · 2015-06-10T09:30:11Z

lichess-org/lila@77c9234

Not only does it work, but http://lichess.org is already running code built by 0.13.9-M1!

update runs in 57 seconds here.

Thank you very much!

ornicar · 2015-06-10T09:35:32Z

4 seconds after enabling cached resolution.

…g merge

Fixes #2105/#1763. Cached resolution: removes duplicate callers during merge

eed3si9n changed the title ~~OOM when compiling with withCachedResolution(true)~~ Cached resolution: OOM when compiling Jan 18, 2015

eed3si9n added area/library_management library management Bug labels Jan 18, 2015

eed3si9n added this to the 0.13.8 milestone Jan 18, 2015

eed3si9n mentioned this issue Jan 18, 2015

Cached resolution issues #1811

Closed

eed3si9n mentioned this issue Feb 27, 2015

Cached resolution: OutOfMemoryException #1721

Closed

eed3si9n added the in progress label Feb 27, 2015

eed3si9n added a commit that referenced this issue Mar 4, 2015

Implement pickler for UpdateReport. #1763

753414c

eed3si9n added a commit that referenced this issue Mar 4, 2015

Use pickler to cache UpdateReport for update task. #1763

8b9b8e0

eed3si9n added a commit that referenced this issue Mar 4, 2015

OOM handling. #1763

a475556

eed3si9n added a commit that referenced this issue Mar 7, 2015

Improve caching internal deps. #1763

863d66f

Some heap optimization during merging too. YourKit showed that mergeOrganizationArtifactReports takes up huge amount of heap.

eed3si9n added a commit that referenced this issue Mar 7, 2015

Write JSON to file without String. #1763

1851551

eed3si9n added a commit that referenced this issue Mar 7, 2015

Implement pickler for UpdateReport. #1763

ee87011

eed3si9n added a commit that referenced this issue Mar 7, 2015

Use pickler to cache UpdateReport for update task. #1763

a2e27bb

eed3si9n added a commit that referenced this issue Mar 7, 2015

OOM handling. #1763

485e508

eed3si9n added a commit that referenced this issue Mar 7, 2015

Improve caching internal deps. #1763

2f9657b

Some heap optimization during merging too. YourKit showed that mergeOrganizationArtifactReports takes up huge amount of heap.

eed3si9n added a commit that referenced this issue Mar 7, 2015

Write JSON to file without String. #1763

5e123ff

eed3si9n mentioned this issue Mar 7, 2015

Use pickler for cached resolution #1904

Merged

eed3si9n self-assigned this Mar 7, 2015

eed3si9n added a commit that referenced this issue Mar 10, 2015

Implement pickler for UpdateReport. #1763

ef1ec99

eed3si9n added a commit that referenced this issue Mar 10, 2015

Use pickler to cache UpdateReport for update task. #1763

b70fa6e

eed3si9n added a commit that referenced this issue Mar 10, 2015

OOM handling. #1763

ca9257c

eed3si9n added a commit that referenced this issue Mar 10, 2015

Improve caching internal deps. #1763

0cad84b

Some heap optimization during merging too. YourKit showed that mergeOrganizationArtifactReports takes up huge amount of heap.

eed3si9n added a commit that referenced this issue Mar 10, 2015

Write JSON to file without String. #1763

40b65c9

eed3si9n closed this as completed in #1904 Mar 10, 2015

eed3si9n removed the in progress label Mar 10, 2015

eed3si9n reopened this Mar 10, 2015

eed3si9n removed this from the 0.13.8 milestone Mar 26, 2015

eed3si9n mentioned this issue May 21, 2015

Fixes #1721/#1763. Cached resolution: summarize callers in graph.json #2030

Merged

jsuereth added a commit that referenced this issue May 22, 2015

Merge pull request #2030 from sbt/wip/no-caller

088f15f

Fixes #1721/#1763. Cached resolution: summarize callers in graph.json

eed3si9n added a commit that referenced this issue Jul 10, 2015

#1763 Remove artificial callers

5412376

eed3si9n added a commit that referenced this issue Jul 10, 2015

#1763 fix by removing one but all artificial callers

22c743c

This was referenced Jul 10, 2015

Fixes #2046. Cached resolution: Fixes double eviction #2097

Merged

withConsolidatedResolution(true) - update gets slower and slower as more time is spent parsing and saving graph.json #2100

Closed

eed3si9n closed this as completed Jul 13, 2015

This was referenced Jul 13, 2015

cached resolution falsely evicts netty (spark + cassandra) #2046

Closed

cached resolution: remove transitive dependencies from callers array when using consolidated resolution #2105

Closed

eed3si9n reopened this Jul 18, 2015

eed3si9n added a commit that referenced this issue Jul 18, 2015

Fixes #2105/#1763. Cached resolution: removes duplicate callers durin…

7eb7df0

…g merge

eed3si9n mentioned this issue Jul 18, 2015

Fixes #2105/#1763. Cached resolution: removes duplicate callers during merge #2115

Merged

jsuereth added a commit that referenced this issue Jul 18, 2015

Merge pull request #2115 from sbt/wip/caller

e6f8d66

Fixes #2105/#1763. Cached resolution: removes duplicate callers during merge

eed3si9n closed this as completed Oct 25, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cached resolution: OOM when compiling #1763

Cached resolution: OOM when compiling #1763

ornicar commented Dec 4, 2014

ornicar commented Dec 5, 2014

eed3si9n commented Feb 27, 2015

easel commented Feb 27, 2015

eed3si9n commented Mar 3, 2015

eed3si9n commented Mar 3, 2015

ornicar commented Mar 3, 2015

easel commented Mar 4, 2015

ornicar commented Mar 10, 2015

eed3si9n commented Mar 10, 2015

eed3si9n commented Jun 9, 2015

ornicar commented Jun 10, 2015

ornicar commented Jun 10, 2015

Cached resolution: OOM when compiling #1763

Cached resolution: OOM when compiling #1763

Comments

ornicar commented Dec 4, 2014

ornicar commented Dec 5, 2014

eed3si9n commented Feb 27, 2015

easel commented Feb 27, 2015

eed3si9n commented Mar 3, 2015

eed3si9n commented Mar 3, 2015

ornicar commented Mar 3, 2015

easel commented Mar 4, 2015

ornicar commented Mar 10, 2015

eed3si9n commented Mar 10, 2015

eed3si9n commented Jun 9, 2015

ornicar commented Jun 10, 2015

ornicar commented Jun 10, 2015