Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[1.3.0-RC3] Heap exhaustion when running multiple commands with many whitespaces #4890

Closed
bjaglin opened this issue Jul 22, 2019 · 3 comments · Fixed by #4904
Closed

[1.3.0-RC3] Heap exhaustion when running multiple commands with many whitespaces #4890

bjaglin opened this issue Jul 22, 2019 · 3 comments · Fixed by #4904

Comments

@bjaglin
Copy link
Contributor

bjaglin commented Jul 22, 2019

sbt version:

  • master (8793034) ❎
  • 1.3.0-RC3 ❎
  • 1.3.0-RC2 ✔️
  • 1.2.8 ✔️
$ cat .jvmopts 
-Xms1536m
-Xmx1536m
-Xss8m

steps

$ sbt ";clean                                                    ;clean                                                    ;clean                                                    ;clean"

problem

Exception in thread "classloader-cache-cleanup-0" java.lang.OutOfMemoryError: Java heap space
        at scala.collection.convert.DecorateAsScala.mapAsScalaConcurrentMapConverter(DecorateAsScala.scala:77)
        at scala.collection.convert.DecorateAsScala.mapAsScalaConcurrentMapConverter$(DecorateAsScala.scala:76)
        at scala.collection.JavaConverters$.mapAsScalaConcurrentMapConverter(JavaConverters.scala:77)
        at sbt.internal.classpath.ClassLoaderCache.sbt$internal$classpath$ClassLoaderCache$$closeExpiredLoaders(ClassLoaderCache.scala:75)
	at sbt.internal.classpath.ClassLoaderCache$CleanupThread.run(ClassLoaderCache.scala:100)
java.lang.OutOfMemoryError: Java heap space
	at scala.collection.immutable.HashSet$HashTrieSet.updated0(HashSet.scala:551)
	at scala.collection.immutable.HashSet.$plus(HashSet.scala:84)
	at scala.collection.immutable.HashSet.$plus(HashSet.scala:35)
	at scala.collection.mutable.SetBuilder.$plus$eq(SetBuilder.scala:28)
	at scala.collection.mutable.SetBuilder.$plus$eq(SetBuilder.scala:24)
	at scala.collection.generic.Growable.loop$1(Growable.scala:56)
	at scala.collection.generic.Growable.$plus$plus$eq(Growable.scala:61)
	at scala.collection.generic.Growable.$plus$plus$eq$(Growable.scala:53)
	at scala.collection.mutable.SetBuilder.$plus$plus$eq(SetBuilder.scala:24)
	at scala.collection.generic.GenericTraversableTemplate.$anonfun$flatten$1(GenericTraversableTemplate.scala:174)
	at scala.collection.generic.GenericTraversableTemplate$$Lambda$306/0x0000000840471840.apply(Unknown Source)
	at scala.collection.Iterator.foreach(Iterator.scala:941)
	at scala.collection.Iterator.foreach$(Iterator.scala:941)
	at scala.collection.AbstractIterator.foreach(Iterator.scala:1429)
	at scala.collection.MapLike$DefaultKeySet.foreach(MapLike.scala:181)
	at scala.collection.generic.GenericTraversableTemplate.flatten(GenericTraversableTemplate.scala:173)
	at scala.collection.generic.GenericTraversableTemplate.flatten$(GenericTraversableTemplate.scala:171)
	at scala.collection.AbstractTraversable.flatten(Traversable.scala:108)
	at sbt.internal.AKeyIndex.tasks(KeyIndex.scala:122)
	at sbt.internal.KeyIndex0.tasks(KeyIndex.scala:221)
	at sbt.internal.Act$.$anonfun$taskKeyExtra$1(Act.scala:115)
	at sbt.internal.Act$$$Lambda$4490/0x00000008413a4840.apply(Unknown Source)
	at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:237)
	at scala.collection.TraversableLike$$Lambda$17/0x00000008400da840.apply(Unknown Source)
	at scala.collection.immutable.List.foreach(List.scala:392)
	at scala.collection.TraversableLike.map(TraversableLike.scala:237)
	at scala.collection.TraversableLike.map$(TraversableLike.scala:230)
	at scala.collection.immutable.List.map(List.scala:298)
	at sbt.internal.Act$.taskKeyExtra(Act.scala:113)
	at sbt.internal.Act$.$anonfun$scopedKeyFull$5(Act.scala:86)
	at sbt.internal.Act$$$Lambda$4481/0x000000084139f840.apply(Unknown Source)
	at scala.Function1.$anonfun$andThen$1(Function1.scala:57)
Error during sbt execution: java.lang.OutOfMemoryError: Java heap space

expectation

Clean happens and sbt returns

notes

Relevant stack trace duing the freeze via jstack:

"main" #1 prio=5 os_prio=0 cpu=42036.01ms elapsed=70.11s tid=0x00007f4d78012800 nid=0x6568 runnable  [0x00007f4d7d099000]
   java.lang.Thread.State: RUNNABLE
	at scala.collection.generic.GenericTraversableTemplate.genericBuilder(GenericTraversableTemplate.scala:72)
	at scala.collection.generic.GenericTraversableTemplate.genericBuilder$(GenericTraversableTemplate.scala:72)
	at scala.collection.AbstractTraversable.genericBuilder(Traversable.scala:108)
	at scala.collection.generic.GenTraversableFactory$GenericCanBuildFrom.apply(GenTraversableFactory.scala:59)
	at scala.collection.generic.GenTraversableFactory$GenericCanBuildFrom.apply(GenTraversableFactory.scala:54)
	at scala.collection.TraversableLike.collect(TraversableLike.scala:273)
	at scala.collection.TraversableLike.collect$(TraversableLike.scala:272)
	at scala.collection.AbstractTraversable.collect(Traversable.scala:108)
	at sbt.internal.util.complete.FixedSetExamples.examplesWithRemovedPrefix(ExampleSource.scala:46)
	at sbt.internal.util.complete.FixedSetExamples.withAddedPrefix(ExampleSource.scala:42)
	at sbt.internal.util.complete.ParserWithExamples.derive(Parser.scala:866)
	at sbt.internal.util.complete.SeqParser.derive(Parser.scala:690)
	at sbt.internal.util.complete.MapParser.derive(Parser.scala:767)
	at sbt.internal.util.complete.SeqParser.derive(Parser.scala:690)
	at sbt.internal.util.complete.MapParser.derive(Parser.scala:767)
	at sbt.internal.util.complete.TokenStart.derive(Parser.scala:806)
	at sbt.internal.util.complete.ParserWithExamples.derive(Parser.scala:865)
	at sbt.internal.util.complete.MapParser.derive(Parser.scala:767)
	at sbt.internal.util.complete.BindParser.derive(Parser.scala:752)
	at sbt.internal.util.complete.ParserSeq.$anonfun$derive$3(Parser.scala:734)
	at sbt.internal.util.complete.ParserSeq$$Lambda$4460/0x00000008413ab840.apply(Unknown Source)
	at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:237)
	at scala.collection.TraversableLike$$Lambda$17/0x00000008400db040.apply(Unknown Source)
	at scala.collection.immutable.List.foreach(List.scala:392)
	at scala.collection.TraversableLike.map(TraversableLike.scala:237)
	at scala.collection.TraversableLike.map$(TraversableLike.scala:230)
	at scala.collection.immutable.List.map(List.scala:298)
	at sbt.internal.util.complete.ParserSeq.derive(Parser.scala:734)
	at sbt.internal.util.complete.BindParser.derive(Parser.scala:752)
	at sbt.internal.util.complete.MapParser.derive(Parser.scala:767)
	at sbt.internal.util.complete.BindParser.derive(Parser.scala:752)
	at sbt.internal.util.complete.MatchedString.derive(Parser.scala:792)
	at sbt.internal.util.complete.MapParser.derive(Parser.scala:767)
	at sbt.internal.util.complete.SeqParser.derive(Parser.scala:690)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.MapParser.derive(Parser.scala:767)
	at sbt.internal.util.complete.And.derive(Parser.scala:822)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.MapParser.derive(Parser.scala:767)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.MapParser.derive(Parser.scala:767)
	at sbt.internal.util.complete.Repeat.derive(Parser.scala:941)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.MapParser.derive(Parser.scala:767)
	at sbt.internal.util.complete.SeqParser.derive(Parser.scala:690)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.HomParser.derive(Parser.scala:703)
	at sbt.internal.util.complete.MapParser.derive(Parser.scala:767)
	at sbt.internal.util.complete.BindParser.derive(Parser.scala:752)
	at sbt.internal.util.complete.MapParser.derive(Parser.scala:767)
	at sbt.internal.util.complete.ParserMain.loop$1(Parser.scala:483)
	at sbt.internal.util.complete.ParserMain.result(Parser.scala:485)
	at sbt.internal.util.complete.ParserMain.result$(Parser.scala:471)
	at sbt.internal.util.complete.Parser$.result(Parser.scala:138)
	at sbt.internal.util.complete.ParserMain.parse(Parser.scala:434)
	at sbt.internal.util.complete.ParserMain.parse$(Parser.scala:433)
	at sbt.internal.util.complete.DefaultParsers$.parse(Parsers.scala:344)
	at sbt.Command$.process(Command.scala:186)
	at sbt.MainLoop$.process$1(MainLoop.scala:188)
	at sbt.MainLoop$.processCommand(MainLoop.scala:223)
	at sbt.MainLoop$.$anonfun$next$2(MainLoop.scala:142)
	at sbt.MainLoop$$$Lambda$622/0x0000000840662040.apply(Unknown Source)
	at sbt.State$StateOpsImpl$.runCmd$1(State.scala:273)
	at sbt.State$StateOpsImpl$.process$extension(State.scala:277)
	at sbt.MainLoop$.$anonfun$next$1(MainLoop.scala:142)
	at sbt.MainLoop$$$Lambda$621/0x0000000840660c40.apply(Unknown Source)
	at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:19)
	at sbt.MainLoop$.next(MainLoop.scala:142)
	at sbt.MainLoop$.run(MainLoop.scala:133)
	at sbt.MainLoop$.$anonfun$runWithNewLog$1(MainLoop.scala:111)
	at sbt.MainLoop$$$Lambda$614/0x000000084065d040.apply(Unknown Source)
	at sbt.io.Using.apply(Using.scala:27)
	at sbt.MainLoop$.runWithNewLog(MainLoop.scala:105)
	at sbt.MainLoop$.runAndClearLast(MainLoop.scala:60)
	at sbt.MainLoop$.runLoggedLoop(MainLoop.scala:45)
	at sbt.MainLoop$.runLogged(MainLoop.scala:36)
	at sbt.StandardMain$.runManaged(Main.scala:124)
	at sbt.xMainImpl$.run(Main.scala:67)
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(java.base@11.0.3/Native Method)
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(java.base@11.0.3/NativeMethodAccessorImpl.java:62)
	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@11.0.3/DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(java.base@11.0.3/Method.java:566)
	at sbt.internal.XMainConfiguration.runXMain(XMainConfiguration.scala:46)
	at sbt.xMain.run(Main.scala:39)
	at xsbt.boot.Launch$$anonfun$run$1.apply(Launch.scala:111)
	at xsbt.boot.Launch$.withContextLoader(Launch.scala:130)
	at xsbt.boot.Launch$.run(Launch.scala:111)
	at xsbt.boot.Launch$$anonfun$apply$1.apply(Launch.scala:37)
	at xsbt.boot.Launch$.launch(Launch.scala:119)
	at xsbt.boot.Launch$.apply(Launch.scala:20)
	at xsbt.boot.Boot$.runImpl(Boot.scala:56)
	at xsbt.boot.Boot$.main(Boot.scala:18)
	at xsbt.boot.Boot.main(Boot.scala)

The step above sounds like a stupid use-case, but it's a simplification of a problem I ran into with the following script invocation:

sbt "
    ; clean
    ; protofmtCheck
    ; protostyleCheck
    ; scalafmtCheckAll
    ; scalafmtSbtCheck
    ; scalafix --check
    ; protomerge
    ; compile
    ; test
    "

The workaround in that case was obviously to remove the cosmetic heading whitespaces, but it took a while to figure it out.

@bjaglin bjaglin added the Bug label Jul 22, 2019
@eed3si9n
Copy link
Member

@bjaglin Thanks for the report!

@eed3si9n
Copy link
Member

eed3si9n commented Jul 22, 2019

Potentially related #4871

@eed3si9n
Copy link
Member

@eatkins It can't be #4871 because that's after RC-3. So maybe it goes back to #4811 or some fixes after that.

@eatkins eatkins mentioned this issue Jul 28, 2019
@github-actions github-actions bot locked as resolved and limited conversation to collaborators May 4, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants