GWT compiler issues - OutOfMemoryError on 64bit machine with massive gwt app

[dankurka]

Originally reported on Google Code with ID 7191

Hi.

My team has massive gwt app. We have problem compiling it to production.
Our (one of tree) biggest, main module generates 70 permutations. In each permutation
we have 130 split points.
Gwt compiler when splitting permutation into separate javascript files requires huge
amount of memory. 

We first were compiling it on machine:
- windows server 2003 32 bit
- 4 GB of ram
- 4 cores 2.4 GHz

We run compile as part of ant file with this macrodef
<macrodef name="gwtCompile">
        <attribute name="module" />
        <attribute name="classpathref" />
        <sequential>
            <java classpathref="@{classpathref}" classname="com.google.gwt.dev.Compiler" fork="true"
maxmemory="896m">
                <arg value="@{module}" />
                <arg value="-war" />
                <arg value="${dir.result.name}" />
                <arg value="-style" />
                <arg value="OBF" />
                <arg value="-localWorkers" />
                <arg value="1" />
            </java>
        </sequential>
    </macrodef>


It took about 140 minutes, but compiled ok. We wanted to switch to 64 bit since jvm
heap size limitations in Win 32bit.

So we set up machine:
- windows server 2008 64 bit R2
- 6 GB of ram
- 4 cores 2.4 GHz

We changed a little bit compile macrodef:
<macrodef name="gwtCompile">
        <attribute name="module" />
        <attribute name="classpathref" />
        <sequential>
            <java classpathref="@{classpathref}" classname="com.google.gwt.dev.Compiler" fork="true">
                <arg value="@{module}" />
                <arg value="-war" />
                <arg value="build/war" />
                <arg value="-style" />
                <arg value="OBF" />
                <arg value="-localWorkers" />
                <arg value="1" />

                <jvmarg value="-Xms256m"/>
                <jvmarg value="-Xmx1024m"/>
                <jvmarg value="-XX:MaxPermSize=64m"/>
                <jvmarg value="-XX:+UseConcMarkSweepGC"/>
            </java>
        </sequential>
    </macrodef>

But on new machine compilation fails. It compiles only when we set Xmx above 1400m.
The error is:
 [java]             Compiling permutation 2...
     [java]       [ERROR] OutOfMemoryError: Increase heap size or lower gwt.jjs.maxThreads
     [java] java.lang.OutOfMemoryError: Java heap space
     [java]     at java.util.HashMap.resize(HashMap.java:462)
     [java]     at java.util.HashMap.addEntry(HashMap.java:755)
     [java]     at java.util.HashMap.put(HashMap.java:385)
     [java]     at java.util.HashSet.add(HashSet.java:200)
     [java]     at com.google.gwt.dev.jjs.impl.ControlFlowAnalyzer$RescueVisitor.visit(ControlFlowAnalyzer.java:495)
     [java]     at com.google.gwt.dev.jjs.ast.JStringLiteral.traverse(JStringLiteral.java:62)
     [java]     at com.google.gwt.dev.jjs.ast.JVisitor.accept(JVisitor.java:144)
     [java]     at com.google.gwt.dev.jjs.ast.JVisitor.acceptImmutable(JVisitor.java:152)
     [java]     at com.google.gwt.dev.jjs.ast.JMethodCall.visitChildren(JMethodCall.java:250)
     [java]     at com.google.gwt.dev.jjs.ast.JMethodCall.traverse(JMethodCall.java:241)
     [java]     at com.google.gwt.dev.jjs.ast.JVisitor.accept(JVisitor.java:125)
     [java]     at com.google.gwt.dev.jjs.ast.JVisitor.accept(JVisitor.java:120)
     [java]     at com.google.gwt.dev.jjs.ast.JVisitor.accept(JVisitor.java:116)
     [java]     at com.google.gwt.dev.jjs.ast.JBinaryOperation.traverse(JBinaryOperation.java:79)
     [java]     at com.google.gwt.dev.jjs.ast.JVisitor.accept(JVisitor.java:125)
     [java]     at com.google.gwt.dev.jjs.ast.JVisitor.accept(JVisitor.java:120)
     [java]     at com.google.gwt.dev.jjs.ast.JVisitor.accept(JVisitor.java:116)
     [java]     at com.google.gwt.dev.jjs.ast.JBinaryOperation.traverse(JBinaryOperation.java:81)
     [java]     at com.google.gwt.dev.jjs.ast.JVisitor.accept(JVisitor.java:125)
     [java]     at com.google.gwt.dev.jjs.ast.JVisitor.accept(JVisitor.java:120)
     [java]     at com.google.gwt.dev.jjs.ast.JVisitor.accept(JVisitor.java:116)
     [java]     at com.google.gwt.dev.jjs.ast.JExpressionStatement.traverse(JExpressionStatement.java:41)
     [java]     at com.google.gwt.dev.jjs.ast.JVisitor.accept(JVisitor.java:144)
     [java]     at com.google.gwt.dev.jjs.ast.JVisitor.acceptWithInsertRemoveImmutable(JVisitor.java:170)
     [java]     at com.google.gwt.dev.jjs.ast.JBlock.traverse(JBlock.java:83)
     [java]     at com.google.gwt.dev.jjs.ast.JVisitor.accept(JVisitor.java:125)
     [java]     at com.google.gwt.dev.jjs.ast.JVisitor.accept(JVisitor.java:137)
     [java]     at com.google.gwt.dev.jjs.ast.JVisitor.accept(JVisitor.java:133)
     [java]     at com.google.gwt.dev.jjs.ast.JMethodBody.traverse(JMethodBody.java:82)
     [java]     at com.google.gwt.dev.jjs.ast.JVisitor.accept(JVisitor.java:125)
     [java]     at com.google.gwt.dev.jjs.ast.JVisitor.accept(JVisitor.java:120)
     [java]     at com.google.gwt.dev.jjs.ast.JMethod.visitChildren(JMethod.java:413)
     [java]          [ERROR] Out of memory; to increase the amount of memory, use the
-Xmx flag at startup (java -Xmx128M ...)
     [java]       [ERROR] Unrecoverable exception, shutting down
     [java] com.google.gwt.core.ext.UnableToCompleteException: (see previous log entries)
     [java]     at com.google.gwt.dev.ThreadedPermutationWorkerFactory$ThreadedPermutationWorker.compile(ThreadedPermutationWorkerFactory.java:56)
     [java]     at com.google.gwt.dev.PermutationWorkerFactory$Manager$WorkerThread.run(PermutationWorkerFactory.java:73)
     [java]     at java.lang.Thread.run(Thread.java:619)
     [java]       [ERROR] Not all permutation were compiled , completed (1/4)
     [java]       [WARN] Lost communication with remote process
     [java] java.net.SocketException: socket closed
     [java]     at java.net.SocketInputStream.socketRead0(Native Method)
     [java]     at java.net.SocketInputStream.read(SocketInputStream.java:129)
     [java]     at java.net.SocketInputStream.read(SocketInputStream.java:182)
     [java]     at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2249)
     [java]     at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2542)
     [java]     at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2552)
     [java]     at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1297)
     [java]     at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
     [java]     at com.google.gwt.dev.ExternalPermutationWorkerFactory$ExternalPermutationWorker.compile(ExternalPermutationWorkerFactory.java:152)
     [java]     at com.google.gwt.dev.PermutationWorkerFactory$Manager$WorkerThread.run(PermutationWorkerFactory.java:73)
     [java]     at java.lang.Thread.run(Thread.java:619)
     [java] Java Result: 1


Are we configuring our build wrong? 
Or this is gwt compiler 'issue' to consume that big amount of memory while compiling
big apps?
How it compiles with 896m of heap on 32bit and breaks with 1024m on 64 bits?
Is this related with variable sizes on 64bit?
Is there different solution, beside extending heap size?

Im curious how much memory it takes to compile for instance AdSense gwt part.


Found in GWT Release (e.g. 2.4.0, 2.5.0 RC):
2.4.0

Encountered on OS / Browser (e.g. WinXP, IE8-9, FF7):
Not browser related.

Workaround if you have one:
Increase Compiler heap size to 1400 MB

Reported by ljaderdev on 2012-02-15 09:49:56

[dankurka]

Ray, John, Toby - can you guys comment on this?

Reported by rdayal@google.com on 2012-02-17 21:56:41

[dankurka]

I'm not a platform export here, but it wouldn't surprise me that there would be an increase
in memory usage on a 64-bit platform.

Reported by rdayal@google.com on 2012-02-17 21:59:24

[dankurka]

Reported by rdayal@google.com on 2012-02-17 21:59:36

• Status changed: AsDesigned

[dankurka]

Pointers in the JVM take 8 bytes instead of 4 (there are possibilities for a mixed model
where you have up to 4G of heap to use 32-bit pointers, and keep other things outside
that 4G area) so simply moving to a 64-bit JVM is likely to increase memory usage from
1.5x to 2x.

These days with memory so cheap, why would you have only 6G in a developer workstation?
 I put 8G in my 5-year-old laptop for $35 not to long ago.

Regarding memory usage internally, for a long time they were all built with 32-bit
JVMs and memory has been tight from time to time with a max of about 2.3G heap.  However,
I am not aware of any with that many split points, as there are diminishing returns
after a while (and your leftover fragment gets large as more code is reachable from
more than one split point).  You should look carefully at your SOYC report and make
sure you are getting benefits from that many split points.

Reported by jat@google.com on 2012-02-17 22:11:44

• Labels added: Category-Compiler

[dankurka]

You might try "-XX:+UseCompressedOops", see:

http://blog.juma.me.uk/2008/10/14/32-bit-or-64-bit-jvm-how-about-a-hybrid/

Reported by stephen.haberman on 2012-02-27 20:45:49

[dankurka]

I am also getting the same problem in the process of GWT compile after upgrading to
2.5-RC1.  (following exception). Also it seems that these two issues need to be linked:
http://code.google.com/p/google-web-toolkit/issues/detail?id=7166

GWT 2.5-RC1 
Ubuntu 12.04 64bits

Work around: in your eclipse right click on the project:
Google --> GWT compile --> advanced --> vm args: -Xmx1024m

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
    at java.util.Arrays.copyOf(Arrays.java:2798)
    at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:111)
    at java.io.ObjectOutputStream$BlockDataOutputStream.writeBlockHeader(ObjectOutputStream.java:1870)
    at java.io.ObjectOutputStream$BlockDataOutputStream.drain(ObjectOutputStream.java:1855)
    at java.io.ObjectOutputStream$BlockDataOutputStream.setBlockDataMode(ObjectOutputStream.java:1766)
    at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1185)
    at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1528)
    at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1493)
    at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1416)
    at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1174)
    at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:346)
    at com.google.gwt.dev.util.Util.writeObjectToStream(Util.java:1193)
    at com.google.gwt.dev.util.DiskCache.writeObject(DiskCache.java:231)
    at com.google.gwt.dev.jjs.UnifiedAst.<init>(UnifiedAst.java:109)
    at com.google.gwt.dev.jjs.JavaToJavaScriptCompiler.precompile(JavaToJavaScriptCompiler.java:700)
    at com.google.gwt.dev.jjs.JavaScriptCompiler.precompile(JavaScriptCompiler.java:33)
    at com.google.gwt.dev.Precompile.precompile(Precompile.java:278)
    at com.google.gwt.dev.Precompile.precompile(Precompile.java:229)
    at com.google.gwt.dev.Precompile.precompile(Precompile.java:141)
    at com.google.gwt.dev.Compiler.run(Compiler.java:232)
    at com.google.gwt.dev.Compiler.run(Compiler.java:198)
    at com.google.gwt.dev.Compiler$1.run(Compiler.java:170)
    at com.google.gwt.dev.CompileTaskRunner.doRun(CompileTaskRunner.java:88)
    at com.google.gwt.dev.CompileTaskRunner.runWithAppropriateLogger(CompileTaskRunner.java:82)
    at com.google.gwt.dev.Compiler.main(Compiler.java:177)

Reported by rakhshan on 2012-07-07 23:49:26

[dankurka]

I get the same on Windows 7 64Bit.

Options: -Xmx2048m -XX:MaxPermSize=1024m -XX:+UseCompressedOops

[INFO]    Computing all possible rebind results for 'com.google.gwt.user.client.ui.impl.ClippedImageImpl.DraggableTemplate'
[INFO]       Rebinding com.google.gwt.user.client.ui.impl.ClippedImageImpl.DraggableTemplate
[INFO]          Invoking generator com.google.gwt.safehtml.rebind.SafeHtmlTemplatesGenerator
[INFO]             Constructing interface com.google.gwt.user.client.ui.impl.ClippedImageImpl.DraggableTemplate
[INFO]                Generating method body for image()
[INFO] Wrote 1 units to persistent cache.
[INFO]    Adding '1' new generated units
[INFO]       Compiling...
[INFO]          Compilation completed in 0,00 seconds
[INFO]       Added 1 units to cache since last cleanup.
[INFO]       Removing invalidated units
[INFO] Wrote 1 units to persistent cache.
[INFO]    Replacing GWT.runAsync with island loader calls
[INFO]    Looking up initial load sequence for split points
[ERROR] Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
[ERROR]     at java.util.Arrays.copyOf(Arrays.java:2271)
[ERROR]     at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:113)
[ERROR]     at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
[ERROR]     at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:140)
[ERROR]     at java.io.ObjectOutputStream$BlockDataOutputStream.drain(ObjectOutputStream.java:1857)
[ERROR]     at java.io.ObjectOutputStream$BlockDataOutputStream.setBlockDataMode(ObjectOutputStream.java:1766)
[ERROR]     at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1185)
[ERROR]     at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:346)
[ERROR]     at com.google.gwt.dev.util.Util.writeObjectToStream(Util.java:1193)
[ERROR]     at com.google.gwt.dev.util.DiskCache.writeObject(DiskCache.java:231)
[ERROR]     at com.google.gwt.dev.jjs.UnifiedAst.<init>(UnifiedAst.java:109)
[ERROR]     at com.google.gwt.dev.jjs.JavaToJavaScriptCompiler.precompile(JavaToJavaScriptCompiler.java:720)
[ERROR]     at com.google.gwt.dev.jjs.JavaScriptCompiler.precompile(JavaScriptCompiler.java:33)
[ERROR]     at com.google.gwt.dev.Precompile.precompile(Precompile.java:278)
[ERROR]     at com.google.gwt.dev.Precompile.precompile(Precompile.java:229)
[ERROR]     at com.google.gwt.dev.Precompile.precompile(Precompile.java:141)
[ERROR]     at com.google.gwt.dev.Compiler.run(Compiler.java:232)
[ERROR]     at com.google.gwt.dev.Compiler.run(Compiler.java:198)
[ERROR]     at com.google.gwt.dev.Compiler$1.run(Compiler.java:170)
[ERROR]     at com.google.gwt.dev.CompileTaskRunner.doRun(CompileTaskRunner.java:88)
[ERROR]     at com.google.gwt.dev.CompileTaskRunner.runWithAppropriateLogger(CompileTaskRunner.java:82)
[ERROR]     at com.google.gwt.dev.Compiler.main(Compiler.java:177)

Reported by udo.walker@abas.de on 2013-04-26 11:47:29

[dankurka]

If I remove the options then it works fine. The virtual machine has 7GB ram available.

Reported by udo.walker@abas.de on 2013-04-26 11:53:15

[chamlyidunil]

use <jvmarg value="-XX:+UseSerialGC" />
It fix for me