Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Compiling with GWT 2.7.0 much much slower (3x) than with GWT 2.6.1 #8960

Closed
dankurka opened this issue Jun 10, 2015 · 37 comments
Closed

Compiling with GWT 2.7.0 much much slower (3x) than with GWT 2.6.1 #8960

dankurka opened this issue Jun 10, 2015 · 37 comments
Assignees
@dankurka
Labels
Category-Compiler Priority-High
Milestone
2.8

Comments

@dankurka
Copy link
Member

Originally reported on Google Code with ID 9006

Issue: Compiling with GWT 2.7.0 much much slower (3x) than with GWT 2.6.1.  


Found in GWT Release (e.g. 2.4.0, 2.5.1, trunk):
2.7.0 RC1 and 2.7.0


Encountered on OS / Browser (e.g. WinXP, IE9, FF10):
I am compiling on a Dell Precision 6600 (quad core i7, SSD, 16 GB RAM) with Windows
7.  


Detailed description:
Compiling with GWT 2.6.1 takes about 5 minutes (285 seconds).

Compiling with GWT 2.7.0 takes about 15 minutes (903 seconds).  It takes over 13 minutes
to get to the point where the permutations are being compiled, so most of the time
is spent computing all possible rebind results.


Shortest code snippet which demonstrates issue (please indicate where
actual result differs from expected result):
n/a


Workaround if you have one:
none


Links to relevant GWT Developer Forum posts:
n/a


Link to patch posted at http://gwt-review.googlesource.com
n/a

Reported by marc.durand on 2014-11-20 15:24:11
@dankurka dankurka self-assigned this Jun 10, 2015
@dankurka
Copy link
Member Author 

The same problem! I think an issue with Remote Services and GPE. 
See
https://youtrack.jetbrains.com/issue/IDEA-132489
https://code.google.com/p/google-plugin-for-eclipse/issues/detail?id=316

Reported by atarasov.mail on 2014-11-24 10:23:58

@dankurka
Copy link
Member Author 

In my case, I am building using ant from the command line.  No IDE involved.

Reported by marc.durand on 2014-11-24 13:41:26

@dankurka
Copy link
Member Author 

My services are not optimized and methods return List and Map interfaces (not specific
implementation) almost everywhere. And compiler spends time computing all possible
rebind results as well. But it happens only in IDE for me. It's weird.

Reported by atarasov.mail on 2014-11-24 14:12:13

@dankurka
Copy link
Member Author 

@stalcup Any ideas (I just thought about this bug that made us drop units from the unit
cache, could this be the case?)

Reported by dankurka@google.com on 2014-11-27 11:35:46

@dankurka
Copy link
Member Author 

Have you tried increasing the RAM? 2.7 attempts to trade off increasing memory to also
increase performance, but if your compile is on the edge of running out of memory it
might actually slow you down.

Reported by stalcup@google.com on 2014-12-01 19:01:20

@dankurka
Copy link
Member Author 

Increasing RAM didn't have any impact...

Reported by marc.durand on 2014-12-05 14:19:24

@dankurka
Copy link
Member Author 

I am encountering the same issue using gwt-maven-plugin. The initial compile process
takes a good 10 minutes to complete before it begins compiling permutations. Watching
the Java thread it uses about 13% of my quad cores with hyper threading and maxes out
in memory usage somewhere close 1,700,000 KB. Once that completes the actual permutations
compile quickly. I was expecting a performance gain in compilation by moving to 2.7
not a decrease.

Reported by joshua.pappalardo on 2014-12-10 20:28:01

@dankurka
Copy link
Member Author 

@joshua how much memory are you giving your compile. If it already uses 1,7 GB you should
at least give it 4GB probably even 6GB.

Reported by dankurka@google.com on 2015-01-10 14:56:28

  • Status changed: NeedsInfo

@dankurka
Copy link
Member Author 

I have just migrated my application (~20MB obfuscated JavaScript) from GWT 2.5.1 to
GWT 2.7.0 and observe the same initially described problem. 
Using ProcessMonitor I see lots of failing file system accesses during GWT compilation
(before the permutations are built) like:

QueryOpen: D:\Projekte\wms_trunk\dev\demo\target\classes\java\lang\com\dematic\wms\app\base\ui\client\ValidationGwtClientException.class
: PATH NOT FOUND
QueryOpen: D:\Projekte\wms_trunk\dev\demo\src\main\java\java\lang\com\dematic\wms\app\base\ui\client\ValidationGwtClientException.class
: PATH NOT FOUND
QueryOpen : D:\Projekte\wms_trunk\dev\demo\src\main\generated\java\lang\com\dematic\wms\app\base\ui\client\ValidationGwtClientException.class
: PATH NOT FOUND

The 3 accesses for the single class file occurs 300 times with the same paths

Additionally I see failing file system accesses like:
QueryOpen   D:\Projekte\wms_trunk\dev\demo\target\classes\java\lang\com\dematic\wms\pac\palletizer\ui\client\dialog\viz\resources\PalletizingResources.class    PATH
NOT FOUND   
QueryOpen   D:\Projekte\wms_trunk\dev\demo\src\main\java\java\lang\com\dematic\wms\pac\palletizer\ui\client\dialog\viz\resources\PalletizingResources.class PATH
NOT FOUND   
QueryOpen   D:\Projekte\wms_trunk\dev\demo\src\main\generated\java\lang\com\dematic\wms\pac\palletizer\ui\client\dialog\viz\resources\PalletizingResources.class    PATH
NOT FOUND   
QueryOpen   D:\Projekte\wms_trunk\dev\demo\target\classes\java\lang\com\dematic\wms\pac\palletizer\ui\client\dialog\viz\resources.class PATH
NOT FOUND   
QueryOpen   D:\Projekte\wms_trunk\dev\demo\src\main\java\java\lang\com\dematic\wms\pac\palletizer\ui\client\dialog\viz\resources.class  PATH
NOT FOUND   
QueryOpen   D:\Projekte\wms_trunk\dev\demo\src\main\generated\java\lang\com\dematic\wms\pac\palletizer\ui\client\dialog\viz\resources.class PATH
NOT FOUND   
QueryOpen   D:\Projekte\wms_trunk\dev\demo\target\classes\java\lang\com\dematic\wms\pac\palletizer\ui\client\dialog\viz.class   PATH
NOT FOUND   
QueryOpen   D:\Projekte\wms_trunk\dev\demo\src\main\java\java\lang\com\dematic\wms\pac\palletizer\ui\client\dialog\viz.class    PATH
NOT FOUND   
QueryOpen   D:\Projekte\wms_trunk\dev\demo\src\main\generated\java\lang\com\dematic\wms\pac\palletizer\ui\client\dialog\viz.class   PATH
NOT FOUND   
QueryOpen   D:\Projekte\wms_trunk\dev\demo\target\classes\java\lang\com\dematic\wms\pac\palletizer\ui\client\dialog.class   PATH
NOT FOUND   
QueryOpen   D:\Projekte\wms_trunk\dev\demo\src\main\java\java\lang\com\dematic\wms\pac\palletizer\ui\client\dialog.class    PATH
NOT FOUND   
QueryOpen   D:\Projekte\wms_trunk\dev\demo\src\main\generated\java\lang\com\dematic\wms\pac\palletizer\ui\client\dialog.class   PATH
NOT FOUND   
QueryOpen   D:\Projekte\wms_trunk\dev\demo\target\classes\java\lang\com\dematic\wms\pac\palletizer\ui\client.class  PATH
NOT FOUND   
QueryOpen   D:\Projekte\wms_trunk\dev\demo\src\main\java\java\lang\com\dematic\wms\pac\palletizer\ui\client.class   PATH
NOT FOUND   
QueryOpen   D:\Projekte\wms_trunk\dev\demo\src\main\generated\java\lang\com\dematic\wms\pac\palletizer\ui\client.class  PATH
NOT FOUND   
QueryOpen   D:\Projekte\wms_trunk\dev\demo\target\classes\java\lang\com\dematic\wms\pac\palletizer\ui.class PATH
NOT FOUND   
QueryOpen   D:\Projekte\wms_trunk\dev\demo\src\main\java\java\lang\com\dematic\wms\pac\palletizer\ui.class  PATH
NOT FOUND   
QueryOpen   D:\Projekte\wms_trunk\dev\demo\src\main\generated\java\lang\com\dematic\wms\pac\palletizer\ui.class PATH
NOT FOUND   
QueryOpen   D:\Projekte\wms_trunk\dev\demo\target\classes\java\lang\com\dematic\wms\pac\palletizer.class    PATH
NOT FOUND   
QueryOpen   D:\Projekte\wms_trunk\dev\demo\src\main\java\java\lang\com\dematic\wms\pac\palletizer.class PATH
NOT FOUND   
QueryOpen   D:\Projekte\wms_trunk\dev\demo\src\main\generated\java\lang\com\dematic\wms\pac\palletizer.class    PATH
NOT FOUND   
QueryOpen   D:\Projekte\wms_trunk\dev\demo\target\classes\java\lang\com\dematic\wms\pac.class   PATH
NOT FOUND   
QueryOpen   D:\Projekte\wms_trunk\dev\demo\src\main\java\java\lang\com\dematic\wms\pac.class    PATH
NOT FOUND   
QueryOpen   D:\Projekte\wms_trunk\dev\demo\src\main\generated\java\lang\com\dematic\wms\pac.class   PATH
NOT FOUND   
QueryOpen   D:\Projekte\wms_trunk\dev\demo\target\classes\java\lang\com\dematic\wms.class   PATH
NOT FOUND   
QueryOpen   D:\Projekte\wms_trunk\dev\demo\src\main\java\java\lang\com\dematic\wms.class    PATH
NOT FOUND   
QueryOpen   D:\Projekte\wms_trunk\dev\demo\src\main\generated\java\lang\com\dematic\wms.class   PATH
NOT FOUND   
QueryOpen   D:\Projekte\wms_trunk\dev\demo\target\classes\java\lang\com\dematic.class   PATH
NOT FOUND   
QueryOpen   D:\Projekte\wms_trunk\dev\demo\src\main\java\java\lang\com\dematic.class    PATH
NOT FOUND   
QueryOpen   D:\Projekte\wms_trunk\dev\demo\src\main\generated\java\lang\com\dematic.class   PATH
NOT FOUND   

All this failing file system accesses are not visible in GWT 2.5.1

In the GWT release notes I found a note:
"JSNI references no longer require fully qualified class names when this wouldn’t be
necessary in Java. (For example, imports work.)"

If I observe thread dumps on the main thread I get very often a stack trace like:

java.io.WinNTFileSystem.getBooleanAttributes(Native Method)
java.io.File.exists(File.java:772)
sun.misc.URLClassPath$FileLoader.getResource(URLClassPath.java:1057)
sun.misc.URLClassPath$FileLoader.findResource(URLClassPath.java:1024)
sun.misc.URLClassPath.findResource(URLClassPath.java:172)
java.net.URLClassLoader$2.run(URLClassLoader.java:551)
java.net.URLClassLoader$2.run(URLClassLoader.java:549)
java.security.AccessController.doPrivileged(Native Method)
java.net.URLClassLoader.findResource(URLClassLoader.java:548)
java.lang.ClassLoader.getResource(ClassLoader.java:1138)
com.google.gwt.dev.javac.JdtCompiler$INameEnvironmentImpl.findTypeInClassPath(JdtCompiler.java:515)
com.google.gwt.dev.javac.JdtCompiler$INameEnvironmentImpl.findType(JdtCompiler.java:461)
org.eclipse.jdt.internal.compiler.lookup.LookupEnvironment.askForType(LookupEnvironment.java:134)
org.eclipse.jdt.internal.compiler.lookup.LookupEnvironment.getType(LookupEnvironment.java:1148)
com.google.gwt.dev.javac.JdtCompiler.resolveType(JdtCompiler.java:652)
com.google.gwt.dev.javac.JdtCompiler.resolveType(JdtCompiler.java:688)
com.google.gwt.dev.javac.JdtCompiler.resolveType(JdtCompiler.java:1006)
com.google.gwt.dev.javac.CompilationStateBuilder$CompileMoreLater$UnitProcessorImpl$1.resolveType(CompilationStateBuilder.java:107)
com.google.gwt.dev.javac.JsniReferenceResolver$JsniReferenceResolverVisitor.resolveClassReference(JsniReferenceResolver.java:344)
com.google.gwt.dev.javac.JsniReferenceResolver$JsniReferenceResolverVisitor.resolveReference(JsniReferenceResolver.java:448)
com.google.gwt.dev.javac.JsniReferenceResolver$JsniReferenceResolverVisitor.endVisit(JsniReferenceResolver.java:259)
com.google.gwt.dev.js.ast.JsNameRef.traverse(JsNameRef.java:126)
com.google.gwt.dev.js.ast.JsVisitor.doTraverse(JsVisitor.java:467)
com.google.gwt.dev.js.ast.JsModVisitor$NodeContext.traverse(JsModVisitor.java:153)
com.google.gwt.dev.js.ast.JsModVisitor.doAccept(JsModVisitor.java:176)
com.google.gwt.dev.js.ast.JsVisitor.accept(JsVisitor.java:109)
com.google.gwt.dev.js.ast.JsInvocation.traverse(JsInvocation.java:92)
com.google.gwt.dev.js.ast.JsVisitor.doTraverse(JsVisitor.java:467)
com.google.gwt.dev.js.ast.JsModVisitor$NodeContext.traverse(JsModVisitor.java:153)
com.google.gwt.dev.js.ast.JsModVisitor.doAccept(JsModVisitor.java:176)
com.google.gwt.dev.js.ast.JsVisitor.accept(JsVisitor.java:109)
com.google.gwt.dev.js.ast.JsArrayAccess.traverse(JsArrayAccess.java:77)
com.google.gwt.dev.js.ast.JsVisitor.doTraverse(JsVisitor.java:467)
com.google.gwt.dev.js.ast.JsModVisitor$NodeContext.traverse(JsModVisitor.java:153)
com.google.gwt.dev.js.ast.JsModVisitor.doAcceptLvalue(JsModVisitor.java:192)
com.google.gwt.dev.js.ast.JsVisitor.acceptLvalue(JsVisitor.java:117)
com.google.gwt.dev.js.ast.JsBinaryOperation.traverse(JsBinaryOperation.java:114)
com.google.gwt.dev.js.ast.JsVisitor.doTraverse(JsVisitor.java:467)
com.google.gwt.dev.js.ast.JsModVisitor$NodeContext.traverse(JsModVisitor.java:153)
com.google.gwt.dev.js.ast.JsModVisitor.doAccept(JsModVisitor.java:176)
com.google.gwt.dev.js.ast.JsVisitor.accept(JsVisitor.java:109)
com.google.gwt.dev.js.ast.JsExprStmt.traverse(JsExprStmt.java:42)
com.google.gwt.dev.js.ast.JsVisitor.doTraverse(JsVisitor.java:467)
com.google.gwt.dev.js.ast.JsModVisitor$ListContext.traverse(JsModVisitor.java:80)
com.google.gwt.dev.js.ast.JsModVisitor.doAcceptWithInsertRemove(JsModVisitor.java:197)
com.google.gwt.dev.js.ast.JsVisitor.acceptWithInsertRemove(JsVisitor.java:121)
com.google.gwt.dev.js.ast.JsBlock.traverse(JsBlock.java:53)
com.google.gwt.dev.js.ast.JsVisitor.doTraverse(JsVisitor.java:467)
com.google.gwt.dev.js.ast.JsModVisitor$NodeContext.traverse(JsModVisitor.java:153)
com.google.gwt.dev.js.ast.JsModVisitor.doAccept(JsModVisitor.java:176)
com.google.gwt.dev.js.ast.JsVisitor.accept(JsVisitor.java:109)
com.google.gwt.dev.js.ast.JsFunction.traverse(JsFunction.java:202)
com.google.gwt.dev.js.ast.JsVisitor.doTraverse(JsVisitor.java:467)
com.google.gwt.dev.js.ast.JsModVisitor$NodeContext.traverse(JsModVisitor.java:153)
com.google.gwt.dev.js.ast.JsModVisitor.doAccept(JsModVisitor.java:176)
com.google.gwt.dev.js.ast.JsVisitor.accept(JsVisitor.java:109)
com.google.gwt.dev.javac.JsniReferenceResolver$JsniReferenceResolverVisitor.resolve(JsniReferenceResolver.java:208)
com.google.gwt.dev.javac.JsniReferenceResolver$JsniDeclChecker.endVisit(JsniReferenceResolver.java:118)
org.eclipse.jdt.internal.compiler.ast.MethodDeclaration.traverse(MethodDeclaration.java:350)
org.eclipse.jdt.internal.compiler.ast.TypeDeclaration.traverse(TypeDeclaration.java:1379)
org.eclipse.jdt.internal.compiler.ast.CompilationUnitDeclaration.traverse(CompilationUnitDeclaration.java:745)
org.eclipse.jdt.internal.compiler.ast.CompilationUnitDeclaration.traverse(CompilationUnitDeclaration.java:706)
com.google.gwt.dev.javac.JsniReferenceResolver.resolve(JsniReferenceResolver.java:821)
com.google.gwt.dev.javac.JsniReferenceResolver.resolve(JsniReferenceResolver.java:755)
com.google.gwt.dev.javac.CompilationStateBuilder$CompileMoreLater$UnitProcessorImpl.process(CompilationStateBuilder.java:102)
com.google.gwt.dev.javac.JdtCompiler$CompilerImpl.process(JdtCompiler.java:384)
org.eclipse.jdt.internal.compiler.Compiler.compile(Compiler.java:470)
com.google.gwt.dev.javac.JdtCompiler.doCompile(JdtCompiler.java:985)
com.google.gwt.dev.javac.CompilationStateBuilder$CompileMoreLater.compile(CompilationStateBuilder.java:339)
com.google.gwt.dev.javac.CompilationStateBuilder.doBuildGeneratedTypes(CompilationStateBuilder.java:649)
   - locked com.google.gwt.dev.javac.CompilationStateBuilder@f3f7b28
com.google.gwt.dev.javac.CompilationStateBuilder$CompileMoreLater.addGeneratedTypes(CompilationStateBuilder.java:242)
com.google.gwt.dev.javac.CompilationState.addGeneratedCompilationUnits(CompilationState.java:125)
com.google.gwt.dev.javac.StandardGeneratorContext.finish(StandardGeneratorContext.java:557)
com.google.gwt.dev.jjs.impl.UnifyAst$UnifyVisitor.createStaticRebindExpression(UnifyAst.java:491)
com.google.gwt.dev.jjs.impl.UnifyAst$UnifyVisitor.createRebindExpression(UnifyAst.java:443)
com.google.gwt.dev.jjs.impl.UnifyAst$UnifyVisitor.handleMagicMethodCall(UnifyAst.java:576)
com.google.gwt.dev.jjs.impl.UnifyAst$UnifyVisitor.endVisit(UnifyAst.java:306)
com.google.gwt.dev.jjs.ast.JMethodCall.traverse(JMethodCall.java:248)
com.google.gwt.dev.jjs.ast.JModVisitor.traverse(JModVisitor.java:381)
com.google.gwt.dev.jjs.ast.JModVisitor.accept(JModVisitor.java:293)
com.google.gwt.dev.jjs.ast.JModVisitor.accept(JModVisitor.java:285)
com.google.gwt.dev.jjs.ast.JVisitor.accept(JVisitor.java:128)
com.google.gwt.dev.jjs.ast.JCastOperation.traverse(JCastOperation.java:67)
com.google.gwt.dev.jjs.ast.JModVisitor.traverse(JModVisitor.java:381)
com.google.gwt.dev.jjs.ast.JModVisitor.accept(JModVisitor.java:293)
com.google.gwt.dev.jjs.ast.JModVisitor.accept(JModVisitor.java:285)
com.google.gwt.dev.jjs.ast.JVisitor.accept(JVisitor.java:128)
com.google.gwt.dev.jjs.ast.JBinaryOperation.traverse(JBinaryOperation.java:83)
com.google.gwt.dev.jjs.ast.JModVisitor.traverse(JModVisitor.java:381)
com.google.gwt.dev.jjs.ast.JModVisitor.accept(JModVisitor.java:293)
com.google.gwt.dev.jjs.ast.JModVisitor.accept(JModVisitor.java:285)
com.google.gwt.dev.jjs.ast.JVisitor.accept(JVisitor.java:128)
com.google.gwt.dev.jjs.ast.JExpressionStatement.traverse(JExpressionStatement.java:42)
com.google.gwt.dev.jjs.ast.JModVisitor.traverse(JModVisitor.java:381)
com.google.gwt.dev.jjs.ast.JModVisitor.accept(JModVisitor.java:293)
com.google.gwt.dev.jjs.ast.JVisitor.accept(JVisitor.java:149)
com.google.gwt.dev.jjs.ast.JIfStatement.traverse(JIfStatement.java:53)
com.google.gwt.dev.jjs.ast.JModVisitor$ListContext.traverse(JModVisitor.java:95)
com.google.gwt.dev.jjs.ast.JModVisitor.acceptWithInsertRemove(JModVisitor.java:351)
com.google.gwt.dev.jjs.ast.JBlock.traverse(JBlock.java:92)
com.google.gwt.dev.jjs.ast.JModVisitor.traverse(JModVisitor.java:381)
com.google.gwt.dev.jjs.ast.JModVisitor.accept(JModVisitor.java:293)
com.google.gwt.dev.jjs.ast.JVisitor.accept(JVisitor.java:149)
com.google.gwt.dev.jjs.ast.JVisitor.accept(JVisitor.java:145)
com.google.gwt.dev.jjs.ast.JMethodBody.traverse(JMethodBody.java:83)
com.google.gwt.dev.jjs.ast.JModVisitor.traverse(JModVisitor.java:381)
com.google.gwt.dev.jjs.ast.JModVisitor.accept(JModVisitor.java:293)
com.google.gwt.dev.jjs.ast.JModVisitor.accept(JModVisitor.java:285)
com.google.gwt.dev.jjs.ast.JMethod.visitChildren(JMethod.java:600)
com.google.gwt.dev.jjs.ast.JMethod.traverse(JMethod.java:569)
com.google.gwt.dev.jjs.ast.JModVisitor.traverse(JModVisitor.java:381)
com.google.gwt.dev.jjs.ast.JModVisitor.accept(JModVisitor.java:293)
com.google.gwt.dev.jjs.ast.JModVisitor.accept(JModVisitor.java:285)
com.google.gwt.dev.jjs.impl.UnifyAst.mainLoop(UnifyAst.java:1505)
com.google.gwt.dev.jjs.impl.UnifyAst.exec(UnifyAst.java:870)
com.google.gwt.dev.jjs.JavaToJavaScriptCompiler$Precompiler.unifyJavaAst(JavaToJavaScriptCompiler.java:1305)
com.google.gwt.dev.jjs.JavaToJavaScriptCompiler$Precompiler.constructJavaAst(JavaToJavaScriptCompiler.java:1038)
com.google.gwt.dev.jjs.JavaToJavaScriptCompiler$Precompiler.precompile(JavaToJavaScriptCompiler.java:954)
com.google.gwt.dev.jjs.MonolithicJavaToJavaScriptCompiler.precompile(MonolithicJavaToJavaScriptCompiler.java:303)
com.google.gwt.dev.jjs.JavaScriptCompiler.precompile(JavaScriptCompiler.java:38)
com.google.gwt.dev.Precompile.precompile(Precompile.java:286)
com.google.gwt.dev.Precompile.precompile(Precompile.java:229)
com.google.gwt.dev.Precompile.precompile(Precompile.java:145)
com.google.gwt.dev.Compiler.run(Compiler.java:206)
com.google.gwt.dev.Compiler.run(Compiler.java:158)
com.google.gwt.dev.Compiler$1.run(Compiler.java:120)
com.google.gwt.dev.CompileTaskRunner.doRun(CompileTaskRunner.java:55)
com.google.gwt.dev.CompileTaskRunner.runWithAppropriateLogger(CompileTaskRunner.java:50)
com.google.gwt.dev.Compiler.main(Compiler.java:127)

In the stack trace is also the class JsniReferenceResolver involved, which seems to
be related to the release note mentioned above

Reported by helmut.reichhold@dematic.com on 2015-01-22 14:35:04

@dankurka
Copy link
Member Author 

Patch is in review at https://gwt-review.googlesource.com/#/c/11300/

Reported by rluble@google.com on 2015-01-23 22:49:27

  • Status changed: ReviewPending
  • Labels added: Category-Compiler

@dankurka
Copy link
Member Author 

The patch fixed the problem for me. The compile times are O.K. again
Tanks for the quick fix

Reported by helmut.reichhold@dematic.com on 2015-01-26 08:15:10

@dankurka
Copy link
Member Author 

It's me again ;-)
Is it possible, that similar code is used for SuperDevMode? Or is the new cache provided
by the fix not used in SDM?
I've hoped that after having the fix in place, my SDM performance issue would also
be solved.
My normal GWT compile for one permutation (with the fix) takes now ~5min.
Starting SDM in IntelliJ or Eclipse takes more than 1 hour (I do not have an exact
figure, since I haven't been patient enough)
But I see even more unsuccessful file system accesses (~100 million!) as during the
GWT compile without the fix.

Example stack trace from SDM compile:
Name: pool-3-thread-1
State: RUNNABLE
Total blocked: 3.237  Total waited: 20.072

Stack trace: 
java.io.WinNTFileSystem.getBooleanAttributes(Native Method)
java.io.File.exists(File.java:772)
sun.misc.URLClassPath$FileLoader.getResource(URLClassPath.java:1057)
sun.misc.URLClassPath$FileLoader.findResource(URLClassPath.java:1024)
sun.misc.URLClassPath.findResource(URLClassPath.java:172)
java.net.URLClassLoader$2.run(URLClassLoader.java:551)
java.net.URLClassLoader$2.run(URLClassLoader.java:549)
java.security.AccessController.doPrivileged(Native Method)
java.net.URLClassLoader.findResource(URLClassLoader.java:548)
java.lang.ClassLoader.getResource(ClassLoader.java:1138)
com.google.gwt.dev.javac.JdtCompiler$INameEnvironmentImpl.findTypeInClassPath(JdtCompiler.java:515)
com.google.gwt.dev.javac.JdtCompiler$INameEnvironmentImpl.findType(JdtCompiler.java:461)
org.eclipse.jdt.internal.compiler.lookup.LookupEnvironment.askForType(LookupEnvironment.java:134)
org.eclipse.jdt.internal.compiler.lookup.LookupEnvironment.getType(LookupEnvironment.java:1148)
com.google.gwt.dev.javac.JdtCompiler.resolveType(JdtCompiler.java:652)
com.google.gwt.dev.javac.JdtCompiler.resolveType(JdtCompiler.java:688)
com.google.gwt.dev.javac.JdtCompiler.resolveType(JdtCompiler.java:1006)
com.google.gwt.dev.javac.CompilationStateBuilder$CompileMoreLater$UnitProcessorImpl$1.resolveType(CompilationStateBuilder.java:107)
com.google.gwt.dev.javac.JsniReferenceResolver$JsniReferenceResolverVisitor.resolveClassReference(JsniReferenceResolver.java:344)
com.google.gwt.dev.javac.JsniReferenceResolver$JsniReferenceResolverVisitor.resolveReference(JsniReferenceResolver.java:448)
com.google.gwt.dev.javac.JsniReferenceResolver$JsniReferenceResolverVisitor.endVisit(JsniReferenceResolver.java:259)
com.google.gwt.dev.js.ast.JsNameRef.traverse(JsNameRef.java:126)
com.google.gwt.dev.js.ast.JsVisitor.doTraverse(JsVisitor.java:467)
com.google.gwt.dev.js.ast.JsModVisitor$ListContext.traverse(JsModVisitor.java:80)
com.google.gwt.dev.js.ast.JsModVisitor.doAcceptWithInsertRemove(JsModVisitor.java:197)
com.google.gwt.dev.js.ast.JsVisitor.acceptWithInsertRemove(JsVisitor.java:121)
com.google.gwt.dev.js.ast.JsArrayLiteral.traverse(JsArrayLiteral.java:92)
com.google.gwt.dev.js.ast.JsVisitor.doTraverse(JsVisitor.java:467)
com.google.gwt.dev.js.ast.JsModVisitor$NodeContext.traverse(JsModVisitor.java:153)
com.google.gwt.dev.js.ast.JsModVisitor.doAccept(JsModVisitor.java:176)
com.google.gwt.dev.js.ast.JsVisitor.accept(JsVisitor.java:109)
com.google.gwt.dev.js.ast.JsBinaryOperation.traverse(JsBinaryOperation.java:118)
com.google.gwt.dev.js.ast.JsVisitor.doTraverse(JsVisitor.java:467)
com.google.gwt.dev.js.ast.JsModVisitor$NodeContext.traverse(JsModVisitor.java:153)
com.google.gwt.dev.js.ast.JsModVisitor.doAccept(JsModVisitor.java:176)
com.google.gwt.dev.js.ast.JsVisitor.accept(JsVisitor.java:109)
com.google.gwt.dev.js.ast.JsExprStmt.traverse(JsExprStmt.java:42)
com.google.gwt.dev.js.ast.JsVisitor.doTraverse(JsVisitor.java:467)
com.google.gwt.dev.js.ast.JsModVisitor$ListContext.traverse(JsModVisitor.java:80)
com.google.gwt.dev.js.ast.JsModVisitor.doAcceptWithInsertRemove(JsModVisitor.java:197)
com.google.gwt.dev.js.ast.JsVisitor.acceptWithInsertRemove(JsVisitor.java:121)
com.google.gwt.dev.js.ast.JsBlock.traverse(JsBlock.java:53)
com.google.gwt.dev.js.ast.JsVisitor.doTraverse(JsVisitor.java:467)
com.google.gwt.dev.js.ast.JsModVisitor$NodeContext.traverse(JsModVisitor.java:153)
com.google.gwt.dev.js.ast.JsModVisitor.doAccept(JsModVisitor.java:176)
com.google.gwt.dev.js.ast.JsVisitor.accept(JsVisitor.java:109)
com.google.gwt.dev.js.ast.JsFunction.traverse(JsFunction.java:202)
com.google.gwt.dev.js.ast.JsVisitor.doTraverse(JsVisitor.java:467)
com.google.gwt.dev.js.ast.JsModVisitor$NodeContext.traverse(JsModVisitor.java:153)
com.google.gwt.dev.js.ast.JsModVisitor.doAccept(JsModVisitor.java:176)
com.google.gwt.dev.js.ast.JsVisitor.accept(JsVisitor.java:109)
com.google.gwt.dev.javac.JsniReferenceResolver$JsniReferenceResolverVisitor.resolve(JsniReferenceResolver.java:208)
com.google.gwt.dev.javac.JsniReferenceResolver$JsniDeclChecker.endVisit(JsniReferenceResolver.java:118)
org.eclipse.jdt.internal.compiler.ast.MethodDeclaration.traverse(MethodDeclaration.java:350)
org.eclipse.jdt.internal.compiler.ast.TypeDeclaration.traverse(TypeDeclaration.java:1379)
org.eclipse.jdt.internal.compiler.ast.CompilationUnitDeclaration.traverse(CompilationUnitDeclaration.java:745)
org.eclipse.jdt.internal.compiler.ast.CompilationUnitDeclaration.traverse(CompilationUnitDeclaration.java:706)
com.google.gwt.dev.javac.JsniReferenceResolver.resolve(JsniReferenceResolver.java:821)
com.google.gwt.dev.javac.JsniReferenceResolver.resolve(JsniReferenceResolver.java:755)
com.google.gwt.dev.javac.CompilationStateBuilder$CompileMoreLater$UnitProcessorImpl.process(CompilationStateBuilder.java:102)
com.google.gwt.dev.javac.JdtCompiler$CompilerImpl.process(JdtCompiler.java:384)
org.eclipse.jdt.internal.compiler.Compiler.compile(Compiler.java:470)
com.google.gwt.dev.javac.JdtCompiler.doCompile(JdtCompiler.java:985)
com.google.gwt.dev.javac.CompilationStateBuilder$CompileMoreLater.compile(CompilationStateBuilder.java:339)
com.google.gwt.dev.javac.CompilationStateBuilder.doBuildGeneratedTypes(CompilationStateBuilder.java:649)
   - locked com.google.gwt.dev.javac.CompilationStateBuilder@11193896
com.google.gwt.dev.javac.CompilationStateBuilder$CompileMoreLater.addGeneratedTypes(CompilationStateBuilder.java:242)
com.google.gwt.dev.javac.CompilationState.addGeneratedCompilationUnits(CompilationState.java:125)
com.google.gwt.dev.javac.StandardGeneratorContext.finish(StandardGeneratorContext.java:557)
com.google.gwt.dev.jjs.impl.UnifyAst$UnifyVisitor.createStaticRebindExpression(UnifyAst.java:491)
com.google.gwt.dev.jjs.impl.UnifyAst$UnifyVisitor.createRebindExpression(UnifyAst.java:443)
com.google.gwt.dev.jjs.impl.UnifyAst$UnifyVisitor.handleMagicMethodCall(UnifyAst.java:576)
com.google.gwt.dev.jjs.impl.UnifyAst$UnifyVisitor.endVisit(UnifyAst.java:306)
com.google.gwt.dev.jjs.ast.JMethodCall.traverse(JMethodCall.java:248)
com.google.gwt.dev.jjs.ast.JModVisitor.traverse(JModVisitor.java:381)
com.google.gwt.dev.jjs.ast.JModVisitor.accept(JModVisitor.java:293)
com.google.gwt.dev.jjs.ast.JModVisitor.accept(JModVisitor.java:285)
com.google.gwt.dev.jjs.ast.JVisitor.accept(JVisitor.java:128)
com.google.gwt.dev.jjs.ast.JCastOperation.traverse(JCastOperation.java:67)
com.google.gwt.dev.jjs.ast.JModVisitor.traverse(JModVisitor.java:381)
com.google.gwt.dev.jjs.ast.JModVisitor.accept(JModVisitor.java:293)
com.google.gwt.dev.jjs.ast.JModVisitor.accept(JModVisitor.java:285)
com.google.gwt.dev.jjs.ast.JVisitor.accept(JVisitor.java:128)
com.google.gwt.dev.jjs.ast.JBinaryOperation.traverse(JBinaryOperation.java:83)
com.google.gwt.dev.jjs.ast.JModVisitor.traverse(JModVisitor.java:381)
com.google.gwt.dev.jjs.ast.JModVisitor.accept(JModVisitor.java:293)
com.google.gwt.dev.jjs.ast.JModVisitor.accept(JModVisitor.java:285)
com.google.gwt.dev.jjs.ast.JVisitor.accept(JVisitor.java:128)
com.google.gwt.dev.jjs.ast.JExpressionStatement.traverse(JExpressionStatement.java:42)
com.google.gwt.dev.jjs.ast.JModVisitor$ListContext.traverse(JModVisitor.java:95)
com.google.gwt.dev.jjs.ast.JModVisitor.acceptWithInsertRemove(JModVisitor.java:351)
com.google.gwt.dev.jjs.ast.JBlock.traverse(JBlock.java:92)
com.google.gwt.dev.jjs.ast.JModVisitor.traverse(JModVisitor.java:381)
com.google.gwt.dev.jjs.ast.JModVisitor.accept(JModVisitor.java:293)
com.google.gwt.dev.jjs.ast.JVisitor.accept(JVisitor.java:149)
com.google.gwt.dev.jjs.ast.JIfStatement.traverse(JIfStatement.java:53)
com.google.gwt.dev.jjs.ast.JModVisitor$ListContext.traverse(JModVisitor.java:95)
com.google.gwt.dev.jjs.ast.JModVisitor.acceptWithInsertRemove(JModVisitor.java:351)
com.google.gwt.dev.jjs.ast.JBlock.traverse(JBlock.java:92)
com.google.gwt.dev.jjs.ast.JModVisitor.traverse(JModVisitor.java:381)
com.google.gwt.dev.jjs.ast.JModVisitor.accept(JModVisitor.java:293)
com.google.gwt.dev.jjs.ast.JVisitor.accept(JVisitor.java:149)
com.google.gwt.dev.jjs.ast.JVisitor.accept(JVisitor.java:145)
com.google.gwt.dev.jjs.ast.JMethodBody.traverse(JMethodBody.java:83)
com.google.gwt.dev.jjs.ast.JModVisitor.traverse(JModVisitor.java:381)
com.google.gwt.dev.jjs.ast.JModVisitor.accept(JModVisitor.java:293)
com.google.gwt.dev.jjs.ast.JModVisitor.accept(JModVisitor.java:285)
com.google.gwt.dev.jjs.ast.JMethod.visitChildren(JMethod.java:600)
com.google.gwt.dev.jjs.ast.JMethod.traverse(JMethod.java:569)
com.google.gwt.dev.jjs.ast.JModVisitor.traverse(JModVisitor.java:381)
com.google.gwt.dev.jjs.ast.JModVisitor.accept(JModVisitor.java:293)
com.google.gwt.dev.jjs.ast.JModVisitor.accept(JModVisitor.java:285)
com.google.gwt.dev.jjs.impl.UnifyAst.mainLoop(UnifyAst.java:1505)
com.google.gwt.dev.jjs.impl.UnifyAst.exec(UnifyAst.java:870)
com.google.gwt.dev.jjs.JavaToJavaScriptCompiler$Precompiler.unifyJavaAst(JavaToJavaScriptCompiler.java:1305)
com.google.gwt.dev.jjs.JavaToJavaScriptCompiler$Precompiler.constructJavaAst(JavaToJavaScriptCompiler.java:1038)
com.google.gwt.dev.jjs.JavaToJavaScriptCompiler$Precompiler.precompile(JavaToJavaScriptCompiler.java:954)
com.google.gwt.dev.jjs.MonolithicJavaToJavaScriptCompiler.precompile(MonolithicJavaToJavaScriptCompiler.java:303)
com.google.gwt.dev.jjs.JavaScriptCompiler.precompile(JavaScriptCompiler.java:38)
com.google.gwt.dev.Precompile.precompile(Precompile.java:286)
com.google.gwt.dev.Precompile.precompile(Precompile.java:229)
com.google.gwt.dev.Precompile.precompile(Precompile.java:145)
com.google.gwt.dev.Compiler.run(Compiler.java:206)
com.google.gwt.dev.codeserver.Recompiler.doCompile(Recompiler.java:333)
com.google.gwt.dev.codeserver.Recompiler.compile(Recompiler.java:161)
com.google.gwt.dev.codeserver.Recompiler.recompile(Recompiler.java:119)
   - locked com.google.gwt.dev.codeserver.Recompiler@ed2089f
com.google.gwt.dev.codeserver.Outbox.recompile(Outbox.java:128)
com.google.gwt.dev.codeserver.JobRunner.recompile(JobRunner.java:81)
com.google.gwt.dev.codeserver.JobRunner.access$100(JobRunner.java:34)
com.google.gwt.dev.codeserver.JobRunner$2.run(JobRunner.java:73)
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
java.util.concurrent.FutureTask.run(FutureTask.java:166)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:722)

Reported by helmut.reichhold@dematic.com on 2015-01-26 10:57:50

@dankurka
Copy link
Member Author 

The code is undoubtedly shared in both codepaths, however the lifetime of the object
that holds the cache might be different; pulling the cache up requires a little bit
of refactoring.

did you try with -strict to see why?

Reported by rluble@google.com on 2015-01-30 22:35:21

@dankurka
Copy link
Member Author 

I am not sure how to start SDM with -strict. I get the following error:
Unknown argument: -strict
Google Web Toolkit 2.7.0
DevMode [-[no]startServer] [-port port-number | "auto"] [-whitelist whitelist-string]
[-blacklist blacklist-string] [-logdir directory] [-logLevel level] [-gen dir] [-bindAddress
host-name-or-address] [-codeServerPort port-number | "auto"] [-[no]superDevMode] [-server
servletContainerLauncher[:args]] [-startupUrl url] [-war dir] [-deploy dir] [-extra
dir] [-modulePathPrefix ] [-workDir dir] [-XmethodNameDisplayMode NONE | ONLY_METHOD_NAME
| ABBREVIATED | FULL] [-sourceLevel [auto, 1.6, 1.7]] [-XjsInteropMode [NONE, JS, CLOSURE]]
[-[no]incremental] module[s] 

My GWT compile in Maven works with "-strict".
If I do a GWT compile in Eclipse with -strict, I get errors because of test classes
using JUnit and Mockito.

Is there anything else I could try?

Reported by helmut.reichhold@dematic.com on 2015-02-02 13:39:11

@dankurka
Copy link
Member Author 

Ah, looks like we might want to add -strict to DevMode. You'll have to run CodeServer
to be able to pass -strict.
When launching DevMode, it should tell you what arguments it passes to the CodeServer
so you can copy-paste. Or you could try "mvn gwt:run-codeserver"; make sure you set
<launcherDir> or pass -Dgwt.codeServer.launcherDir= on the command line to get "compile
on load"; just use the same value as the <hostedWebapp> for gwt:run (note: you'll also
have to use <failOnError> instead of <strict>, as I just spotted a typo in the gwt-maven-plugin's
gwt:run-codeserver that expects "struct" instead of "strict").

Reported by t.broyer on 2015-02-03 09:58:13

@dankurka
Copy link
Member Author 

We are currently trying to upgrade our 2.6.1 gwt to 2.7.0.
Our permutations processing with gwt 2.6.1 took 1155.690s for 29 permutations on a
8-core machine with 16GB of memory. (settings: localWorkers 5, mx2G)
With gwt 2.7.0 I had to decrease the amount of workers to 3 as I had to increase the
memory to mx3500M. (Otherwise the process dies with cannot allocate memory)
Now the permutation processing takes 2276.885s for 25 permutations.

This is currently a big hump we dont know how to tackle.
Any helpful idea is welcome.

Thanks,
Detlef

Reported by detlef.brendle@prospective.ch on 2015-02-03 15:02:08

@dankurka
Copy link
Member Author 

If I run 
mvn gwt:run-codeserver -Dgwt.codeServerPort=9876 -Dgwt.bindAddress=127.0.0.1 -Dgwt.codeServer.precompile=false
-Dgwt.module=com.dematic.demowarehouse.ui.Desktop -Dgwt.extraJvmArgs=-Xmx6G -Dgwt.codeServer.launcherDir=D:\Projekte\wms_trunk\dev\demo\target\demo
-Dgwt.compiler.strict=true
The code server is started with 
java -Xmx6G -classpath <classpath>  com.google.gwt.dev.codeserver.CodeServer -logLevel
INFO -noprecompile -sourceLevel auto -failOnError -bindAddress 127.0.0.1 -port 9876
-launcherDir D:\Projekte\wms_trunk\dev\demo\target\demo com.dematic.demowarehouse.ui.Desktop
This starts up the code server and after ~800 seconds the compile has been successfully
finished.
I get the same times regardless, whether the maven GWT plugin uses the original gwt-dev
jar or the patched one (patch 11300).
In both cases I see failed file system accesses I observed in the unpatched GWT compile
(And that is probably the reason for the almost tripled start up time of the compile
in the code server)
Starting SDM in IntelliJ IDEA still shows the same behavior as described in https://code.google.com/p/google-web-toolkit/issues/detail?id=9006#c12

Reported by helmut.reichhold@dematic.com on 2015-02-04 10:10:22

@dankurka
Copy link
Member Author 

When will GWT 2.7.1 be released with that fix?

Reported by juli1.bec on 2015-02-10 21:20:04

@dankurka
Copy link
Member Author 

GWT 2.8 will be released in the coming weeks/months, there won't be a 2.7.1.
That said, this patch is still not merged…

Reported by t.broyer on 2015-02-11 10:54:15

  • Labels added: Milestone-2_8

@dankurka
Copy link
Member Author 

For 2.8 can we expect performance improvements and memory consumption during compilation
? Or at least going back to the memory footprint of 2.6.1 would be nice to see. If
not we will surely stick to 2.6.1 - performance in 2.7.0 was not acceptable for us.

Reported by detlef.brendle@prospective.ch on 2015-02-19 16:51:40

@dankurka
Copy link
Member Author

Reported by t.broyer on 2015-02-23 07:19:50

  • Labels added: Priority-High

@dankurka
Copy link
Member Author 

Hi Gwt Team,

is there any news about that issue ? Would be nice.

Thanks,
Detlef

Reported by detlef.brendle@prospective.ch on 2015-03-10 19:05:00

@dankurka
Copy link
Member Author 

The patch has been submitted, however as it is noted in the bug it does not solve the
problem for SDM.

Reported by rluble@google.com on 2015-03-10 23:29:39

@dankurka
Copy link
Member Author 

We don't have a repro case of the slowdown at hand (it seems that is especially slow
only in Windows? we don't see the problem in our Linux workstations). I wonder if any
of our OS contributors that has access to a windows machine can take a deeper look
at the issue.

Most likely moving all the name resolution caches to CompilerContext from JdtCompiler
will solve the issue both in batch and SDM but it requires a bit of refactoring.

Reported by rluble@google.com on 2015-03-10 23:35:21

@dankurka
Copy link
Member Author 

Just as a data point: I have experienced the same ca. 2.5x slowdown on Linux (Fedora
19 64-bit) on recent hardware with 16GB of RAM.

Reported by rcalmbac on 2015-03-10 23:49:46

@dankurka
Copy link
Member Author 

We are running our builds mainly on Linux systems. The performance behaviour is the
same no matter where we build. Mac, Ubuntu, Amazon Linux. On all system we experience
this slow down.

Reported by detlef.brendle@prospective.ch on 2015-03-11 06:51:31

@dankurka
Copy link
Member Author 

I have another observation:
Executing a GWT compile in IntelliJ takes ~45 minutes instead of 5 minutes with maven
(the long compile time shows the same behavior I have described in https://code.google.com/p/google-web-toolkit/issues/detail?id=9006#c9)
. 
My project consists out of ~10 maven modules
The only significant difference I could find is, that maven uses source JAR files for
my base modules, IntelliJ uses source directories.
Perhaps this effect is also responsible for the very long compile times in SDM (https://code.google.com/p/google-web-toolkit/issues/detail?id=9006#c12)

Reported by helmut.reichhold@dematic.com on 2015-03-24 11:36:37

@dankurka
Copy link
Member Author 

Any chance we can change the status from ReviewPending to Accepted?

Reasoning:
ReviewPending is the right status for normal GWT compiles, but not for Super Dev Mode
compiles (see comment #12 and #13). So it makes sense to me if we revert back to Accepted.

(Alternatively we could keep it at ReviewPending and file a separate issue for Super
Dev Mode.)

Reported by brad.cupit@riptidesoftware.com on 2015-05-28 14:15:51

@dankurka
Copy link
Member Author 

Separate issue always makes more sense than dealing with different issues in the same
ticket.

Reported by rocky303 on 2015-05-28 14:17:39

@dankurka
Copy link
Member Author 

+1 for effecting both Windows and Linux machines.

Reported by hyeh@touchcommerce.com on 2015-05-28 15:27:28

@dankurka
Copy link
Member Author 

Can anyone provide a patched jar?

Reported by ryphoenix on 2015-05-28 17:05:21

@tbroyer tbroyer modified the milestone: 2.8 Jun 20, 2015
@HelmutReichhold
Copy link

I haven't seen any progress related to the SDM fix. Since Chrome is dropping classic DevMode support in the next version, I get a bit nervous...

@rluble
Copy link
Contributor

rluble commented Nov 10, 2015

I have a tentative patch that addresses the code path shown in the stack traces above: https://gwt-review.googlesource.com/13953

Can someone try it and confirm whether it fixes the problem or not.

@HelmutReichhold
Copy link

Hi Roberto,
This patch fixes my issues. Thank you very much :-)

hubot pushed a commit that referenced this issue Nov 10, 2015
@rluble
Avoid unnecessary path scanning during JSNI resolution.
0b0723f 
This patch:
- caches the set of unresolvable types to avoid file scanning, however the
  cache is cleared between generator invocations (generators may generate
  one of the missing files).
- resolves inner classes by binary name, avoiding looking for files in
  subdirectories.

Change-Id: Ic848d63f81400effdc92fa50f60e1e63200eed19
Bug-Link: #8960
@anton-tregubov
Copy link

Hi Roberto,
This patch make better only project with many JSNI methods. Other project still start SDM longer then 100 000 ms

@gkdn
Copy link
Contributor

gkdn commented Dec 5, 2015

Please open another bug with details if the you see other performance problems.

@gkdn gkdn closed this as completed Dec 5, 2015
@uhurusurfa
Copy link

Using the G1GC garbage collector on OSX made a big difference for me

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dankurka dankurka

Labels
Category-Compiler Priority-High
Projects
None yet
Milestone
2.8
Development

No branches or pull requests
7 participants
@tbroyer @dankurka @anton-tregubov @gkdn @rluble @uhurusurfa @HelmutReichhold