RFC: Introduce build rules for building GWT applications. #109
Comments
|
One argument that |
|
The Gerrit project (which uses Buck!) has defined |
|
I don't think JSNI is too much of an issue since it's designed so you can compile it as Java. It will be a native method with no implementation. In the build tool's implementation, you probably want to have separate, mostly-parallel DAG's of build actions to perform for Java and GWT, so that the GWT output doesn't have to be rebuilt when you only want Java. However, it's annoying to have to maintain separate rules for GWT and non-GWT library variants in build files. So I think java_library should handle both, but you will need special attributes for GWT. For example, there may be gwt-only dependencies, Java-only dependencies, and shared dependencies. It might also be good to distinguish GWT generator source code (typically in a "rebind" directory) and dependencies needed only by the generator. Dev Mode is slowly going away so I wouldn't worry about that too much. For the GWT compiler and Super Dev Mode, you need to construct a Java classpath containing all the dependencies and use that to start the GWT tool. (We are also working on incremental compilation for GWT, but that's not far enough along to worry about quite yet.) Another issue for (Super)DevMode is that you want to put the original source files into the classpath, so that when the user edits a file, the GWT compiler sees the changes without having to run buck. However, the GWT compiler doesn't recompile code used by generators, so the bytecode for generators has to be in the classpath. (In addition, some GWT generators assume Java bytecode exists so they can use Class.forName() and reflection.) |
|
There are three kind of GWT artifacts: shared libraries (shared between JVM -e.g. server, or Android client- that is), client-only libraries, and applications. The difference between client and shared libs is about where the classes go. Shared libs have most classes in a non-GWT jar, and another GWT-specific jar with GWT-specific files and classes and the gwt.xml. Client-only libs on the other hand do not need that separation. Shared libraries are the toughest ones to deal with, for the reasons you gave. My mind has been corrupted by Maven and the like, but I think a parallel tree is what works best: shared1-gwt depends on shared1, shared2 depends on shared1, shared2-gwt depends on shared2 and shared1-gwt. Also, if a GWT library (client or shared) has a single gwt.xml, it's possible to generate/merge from the rule dependencies. I started doing something along these lines for Maven (where separation is strict), an was thinking of doing the same for Gradle (where you could generate both jars if a shared lib from the same project). |
|
As for dev mode, first you'll want to use superdevmode nowadays. Size of the classpath can have a big impact on performance, so you don't want to put your source tree there. Following Buck's strategy for other things, you'll rather symlink files. For a better edit-refresh-cycle (including when adding files), maybe you'd rather symlink directories instead. And you'll of course need to use the gwt.xml too. |
|
On Tue, Apr 29, 2014 at 12:06 PM, bolinfest notifications@github.comwrote:
https://gerrit.googlesource.com/gerrit/+/HEAD/gerrit-gwtexpui/BUCK We define a java_library(name='server') that has some overlap in srcs with For us gwt_module() is defined as a simple rule to collect the sources into For runtime "edit-reload-test" cycle our application server knows how to https://gerrit.googlesource.com/gerrit/+/HEAD/gerrit-pgm/src/main/java/com/google/gerrit/pgm/http/jetty/JettyServer.java We have a genrule() in buck output a properties file that the server can |
|
Thanks everyone: this is all incredibly helpful! I haven't started playing around with SuperDevMode, but I certainly want to The reason I suggest this is that it would be a lot easier to do things "in java/com/example/stuff/Shared.java If I included java/com/example/stuff/ as a classpath entry, then that would java/com/example/stuff/Shared.java and then pass the path of that manifest file as an input to SuperDevMode. Thoughts? On Tue, Apr 29, 2014 at 3:02 PM, Shawn O. Pearce
|
|
Sure, adding support for a sourcefile manifest wouldn't be too hard. However, in a way, the gwt.xml files are already a manifest if you don't use globs. Would Buck be generating them? Also, you probably want to build using the GWT compiler's -strict flag (also known as -failOnErrors) by default; this will help ensure you get compile errors if your gwt.xml files include too much. |
|
I think we should change GWT to load from a dedicated class loader ultimately. It could be set to the system class loader by default for backwards compatibility, but if it's swappable then it becomes possible to pass a classpath as argument to the GWT compiler/superdevmode/etc. and have it construct a That said, for that to happen, we'd probably first have to remove classpath scanning from GWT (the custom class loader for Buck would then be about ensuring some file that isn't in the |
|
I'm learning as I go with all things GWT right now, so please bear with me. @tbroyer "Also, if a GWT library (client or shared) has a single gwt.xml, it's possible to generate/merge from the rule dependencies." Is it possible for a GWT library (or "module" in your parlance?) to have more than one gwt.xml file? From a build perspective, it appears that they are just resources in a JAR file, so I guess there is no reason why there cannot be more than one. Is there ever a case where additional metadata about the gwt.xml files needs to be written to the JAR's manifest or anywhere else? The answer to this question determines whether a @skybrian "However, the GWT compiler doesn't recompile code used by generators, so the bytecode for generators has to be in the classpath. (In addition, some GWT generators assume Java bytecode exists so they can use Class.forName() and reflection.)" Does the GWT compiler normally need the @skybrian After playing around with things myself, I agree that having separate java_library(
name = 'lib1',
srcs = glob(['lib1/**/*.java']),
)
java_library(
name = 'lib2',
srcs = glob(['lib2/**/*.java']),
deps = [ ':lib1', ],
)
gwt_binary(
name = 'app',
module = 'com.example.client.TheClient',
deps = [ ':lib2', ],
)And you run the following command: You might expect that this would not run We have worked hard to avoid a situation where a rule is allowed to "look upwards" to decide how to build its We have been able to avoid the "look upwards" thing using a technique we call graph_enhancement, which I won't get into here (it probably deserves its own write-up on http://facebook.github.io/buck/). Under the hood, for a The Though this would allow us to reuse What do folks think of this approach? Currently I'm skirting some issues like |
|
On Wed, Apr 30, 2014 at 4:50 PM, bolinfest notifications@github.com wrote:
For prebuilt jar files, you might want to have a gwt_import rule that takes
As a result, if you just compile using the GWT compiler, you won't catch For incremental compilation, we will need to clean this up. In Buck, it's
However, incremental compiles will be a problem; there's currently no way
|
|
On Thu, May 1, 2014 at 6:22 PM, Brian Slesinsky notifications@github.comwrote:
This is why I said earlier that you'd want to symlink files like you do for
If you put src/ on the classpath, then GWT will find lib1/* despite the Computing the GWT dependencies from Buck dependencies (see below) frees you
I tried to do something similar for Maven, but it has to look at a file
There's also the case of classes referenced by annotations (e.g.
You'll also need to javac the classes if you test them. Thomas Broyer |
|
Thanks for all of the info! I am actively working on a naive implementation of I think that it would make sense to have an argument to I think it's important for me to land the initial version and then iterate on it from there. It hear what you're saying and it definitely makes sense for Buck to generate the appropriate bits of the .gwt.xml files from its dependency graph, though the initial version that I land won't do that. Once I lay the groundwork, you could also feel free to start sending pull requests my way! |
|
I am still working on getting my diffs on However, while we're waiting for those, it seems like it currently takes ~20s for the GWT compiler to run on a modest size app (yes, that is with using For those of you who are using it, does it get edit-refresh-test cycles down from ~20s to 1s? Historically, I am a JavaScript developer, so although I love IDEs and static typing, I don't know if I would opt for that in favor of faster dev cycles. For the JS/Closure build tool that I wrote (http://plovr.com/), you can control the level of compilation that you want via query parameters. Most of the time, you want "raw" mode where the Closure Compiler does absolutely nothing and you have one I don't know how GWT works, but can it be designed to work this way? That is, can each input Admittedly, this might forgo a lot of error-checking, but I am relying on the IDE to do most of my static checking, as is. I could always run a full [slow] build if I am seeing things that I do not expect in the browser because I am playing fast and loose with the "raw" mode that I described. |
|
Super Dev Mode as it exists today is basically a wrapper around draft mode. (Note that -draft is not quite the same as -optimize 0; it sets some additional flags). It gets a bit of speed on subsequent compiles because it's running the compiler in the same JVM, so some things get cached. But it's fundamentally not that different from -draft. The compiler improvements you describe are basically what we call "incremental compilation". John Stalcup has been working on this for many months and is now adding it to Super Dev Mode; it's available on trunk behind the "-incremental" flag. However, many modules (including within the GWT SDK itself) don't work with it yet. Also, it's not always faster yet; we're currently focused on making the migration smoother. We did Super Dev Mode first because we control the entire build process that way, but it's definitely a goal to make incremental compilation work with other build systems. It would be great if we could support it in Buck. But it's a little early to get started, unless you want to jump into working on GWT's internals. (It might be worth looking into it and giving us feedback on the design.) There will be some significant improvements to Super Dev Mode in GWT 2.7 which we're planning on releasing around June. Also, we're currently releasing GWT 2.6.1 but that's just a bugfix release. (The main change for Super Dev Mode in 2.6.1 is getting it to work on Windows.) |
|
Also, this isn't going to help get you from 20s to 1s, but it's an important speedup for larger GWT projects: it's possible to optimize each permutation in a separate JVM, in parallel. The entry points for that are com.google.gwt.dev.Precompile for the first phase, com.google.gwt.dev.CompileOnePerm to compile a permutation (this can be run in parallel), and com.google.gwt.dev.Link to assemble the permutations into a single output directory. In Google's build system, we use this to compile each permutation on a different machine. |
Summary: Implement `gwt_binary()` as a result of the discussion at: #109 Using `gwt_binary()` looks like the following: ``` gwt_binary( name = 'my_app', modules = [ 'com.example.client.TheClient', # GWT module to build. ], style = 'PRETTY', draft_compile = True, optimize = 0, local_workers = 4, module_deps = [ # List of java_library() build targets that function as GWT modules. ], deps = [ # Traditional deps. # One of these should be a java_library() (or prebuilt_jar()) rule that # has the definition of the com.google.gwt.dev.Compiler class. ], ) ``` For each Java library (i.e., `java_library()` or `prebuilt_jar()`) in `module_deps`, a source JAR is generated and is included on the classpath when running the GWT compiler. Therefore, building a `gwt_binary()` does not entail running `javac` to build each of the rules listed in `module_deps`. If you want to make sure that `javac` is run for your library, you can always list your Java library in `deps`. (In the future, I will probably add an argument to `gwt_binary()` to specify whether `module_deps` get built like regular deps.) Note that the rules in the `module_deps` list will likely want to include their `.gwt.xml` files as Java resources. As such, you might want to define the following macro in your build: ``` import copy def gwt_module( gwt_xml=None, **kwargs ): """A GWT library is defined by its .java source code, a .gwt.xml module file, and resources.""" java_kwargs = copy.copy(kwargs) if not 'resources' in java_kwargs: java_kwargs['resources'] = [] if gwt_xml: java_kwargs['resources'] += [gwt_xml] java_library(**java_kwargs) ``` Going forward, whether we support a `gwt_module()` rule or just add a `gwt_xml` argument to `java_library()` is open for debate. For now, we would like to add support for GWT while introducing as few new concepts as possible to Buck. This diff also introduces a `gwt_jar` argument to `prebuilt_jar`. This is useful for defining Guava: ``` prebuilt_jar( name = 'guava', binary_jar = 'guava-17.0.jar', source_jar = 'guava-17.0-sources.jar', gwt_jar = 'guava-gwt-17.0.jar', visibility = [ 'PUBLIC', ], ) ``` This is imperative when you have an alternative GWT implementation of a Java library. Specifically, when you have one implementation that contains JSNI, and one implementation that contains an implementation that cannot be translated to JS by the GWT compiler. Finally, here is an example of using macros to create "debug" and "release" targets for the same GWT application: ``` java_library( name = 'debug_module', resources = [ 'Example_debug.gwt.xml', ], ) java_library( name = 'release_module', resources = [ 'Example_release.gwt.xml', ] ) def create_gwt_binary_rules(name): for is_debug in [True, False]: gwt_binary( name = name + '_' + ('debug' if is_debug else 'release'), style = 'PRETTY' if is_debug else 'OBF', draft_compile = is_debug, optimize = 0 if is_debug else 9, local_workers = 3 if is_debug else 8, modules = [ 'com.example.Example_' + ('debug' if is_debug else 'release'), ], module_deps = [ # List the common module_deps here. ] + [':debug_module' if is_debug else ':release_module'], deps = [ # List the common deps here. ], ) # Creates :example_debug and :example_release. create_gwt_binary_rules('example') ``` Test Plan: Retrofit an existing GWT application with `BUCK` build files and build it using Buck.
|
I apologize that this took so long, but I just landed an implementation of |
|
Are relative paths in Is this because I am passing source jars to I'm hoping to make Buck build GWT apps quickly by avoiding |
|
It seems that if I use an "absolute path" with no leading slash, then things work. |
|
I just tried using 2af62f4 to build Gerrit Code Review. It appears to work, but we lost some speed. Gerrit passes We used to pass I am confused about why the The resulting .war is 2M larger for us. It looks like the There is no way to set the heap memory of the JVM running the GWT compiler. Gerrit had to pass |
|
Gerrit also used to pass |
|
All helpful feedback. Thank you. I think some of the points you raise are The alternative to the "jvm_flags" would be to have a parameter in your Regards, Simon On Wed, May 14, 2014 at 12:09 AM, Shawn O. Pearce
|
|
On Tue, May 13, 2014 at 4:37 PM, Simon Stewart notifications@github.com wrote:
I think a single entry makes sense, but maybe some of the other GWT
Right these are not common so a generic flag parameter for the
Yes, but the global setting may be better:
+1, this may be cleaner. |
You can pass several modules to the GWT compiler. It outputs each one's output to a different directory within the
You're far from unique. symbolMaps are useful for deobfuscating stacktraces logged by the client, but a) I suspect not many people actually send logs to the server and b) it doesn't work out of the box as it needs some configuration: both GWT-RPC-based
Brian proposed this should be the default. |
|
@spearce Awesome: thanks for the feedback! I will try to add options for these things and land this today. I have been doing specific arguments instead of a general "compiler args" list because: (1) When I finally write up documentation, it is easier for users to see all of the available options. What do you think? |
Summary: This is in response to Shawn Pearce's feedback on #109. Hopefully this addresses all of the issues. Test Plan: Retrofit an existing GWT application with `BUCK` build files and build it using Buck.
|
I think we do need a catch-all for various GWT compiler arguments, which should be appended at the end. (In the GWT compiler, flags that come later in the list override flags that come earlier.) There are lots of flags, some are experimental, and we do change them in new GWT releases (while trying to remain backward compatible). You shouldn't have to update Buck (and its documentation) every time we do a GWT release; instead link to the GWT compiler's docs. [1] Also, Buck users should be able to use nightly builds of GWT if they like and try out experimental compiler flags without having to change Buck, so they can give the GWT team feedback on release candidates. Of course there are some compiler options that Buck needs to know about, so I suggest just adding attributes for those. (In particular, controlling the compiler's inputs and outputs.) Regarding JVM flags: we do have a compiler_jvm_flags attribute in blaze. It seems to be used solely for tinkering with JVM performance to make GWT compiles go faster, so I don't have a strong opinion; it could start out as a global flag for now. [1] http://www.gwtproject.org/doc/latest/DevGuideCompilingAndDebugging.html at the bottom. |
|
@bolinfest Thanks. Looks good. Performance is OK now. One question, If i'm reading the code correctly the default for Before switch to but now i think it's more cleaner to do it from |
|
@bolinfest Thanks for adding
I have an issue with recovering from compile errors in GWT code:
To recover from this situation, i've found that only The output file seems not to be created from the cache after GWT error. We are using the following caching strategy: |
|
I don't follow: in your first example, I only see one command run.
|
OK, first successful buck run was missing:
|
|
Why shouldn't it output a file if you removed the error?
|
|
@davido I think I misread. So you're saying that after the third call to |
|
OK, I can repro this. Investigating.. |
|
@bolinfest Exactly. That's what i am saying. It looks like it isn't get compiled new, normal compile time would be around 30 sec. but the third call after revert of my error change is ready after 1 sec. So i think that something is going wrong with cache handling. If you prefer i could create a reproducer. |
|
What is really weird is that I looked at the cache key from the build trace ( |
|
Unfortunately it doesn't work (yet). |
|
@davido I believe this is not specific to Buck. I suspect these lines in https://github.com/facebook/buck/blob/master/src/com/facebook/buck/rules/CachingBuildEngine.java are to blame: Or rather, this doing the right thing, but if the build rule fails to build, then we need to make sure that we delete the existing |
|
Ha: look at the TODO I wrote for myself on line 224. |
You mean it's not specific to the
;-) |
Summary: A concrete failure case was reported in the wild: #109. Test Plan: Very straightforward repro base: * Build a GWT app, verify the .zip file is generated. * Introduce a breaking change and try to rebuild the GWT app. * Verify that the .zip file, `metadata/RULE_KEY` and `metadata/RULE_KEY_NO_DEPS` files are deleted. * Revert the breaking change so it matches the original rule key. * Build the GWT app again. * Verify that the .zip file, `metadata/RULE_KEY` and `metadata/RULE_KEY_NO_DEPS` files are restored. Also ran `buck test --all` locally and verified that all tests passed.
|
@bolinfest Thanks, works like a charm: |
We have only very few GWT specific tests and we are using GWT test utils library [1] : |
|
@bolinfest Just to let you know: The only three custom Buck hacks in Gerrit itself and its eco systems are:
|
Migrate gwt build tool chain to built in gwt_binary() rule [1]. [1] facebook/buck#109 Change-Id: I1f13f4d29864bc7e7278f8a2b2e39c882acf44aa
|
I've spent some time to understand why small extension to Gerrit GWT Plugin API worked as expected in standalone build mode, but failed to compile in Gerrit tree mode: [1]. Classpath of GWT compiler invocation was polluted with dozens of pseudo GWT modules, that interfered with this new extension. What happened? The easiest way to see the problem is to check the source code: [2]. It turns out (it wasn't really clear to me) that the implementation reconstructs GWT modules from the dependency graph. In Gerrit tree build In standalone mode, that makes use of I think that the reachability resolution algorithm should differentiate between provided deps and normal deps. Another option to consider is to make dependency module resolution configurable. [1] https://gerrit-review.googlesource.com/#/c/63489 |
As explained in [1], [2] in tree build mode is currently broken for plugins that expose GWT modules. To rectify, isolate the libraries that used for building regular and GWT modules. While regular modules need the whole Plugin API as their dependency, GWT modules should only consume pre-defined set of GWT API modules. [1] facebook/buck#109 [2] https://gerrit-review.googlesource.com/63489 Change-Id: I4694745254ec0f2d9578d79b62110241d4b8b429
|
“ I don't want to document it on http://facebook.github.io/buck/ until I get some external confirmation that this is the right way to go.” @bolinfest：Is it the time to document it? |
|
FYI, I copied from Buck how you configure GWT for my gwt-maven-plugin (very few properties exposed, and Please note however that |
As explained in [1], [2] in tree build mode is currently broken for plugins that expose GWT modules. To rectify, isolate the libraries that used for building regular and GWT modules. While regular modules need the whole Plugin API as their dependency, GWT modules should only consume pre-defined set of GWT API modules. [1] facebook/buck#109 [2] https://gerrit-review.googlesource.com/63489 Change-Id: I5943bec40e659421b746de92397468b8ad7420ab
|
We now have integration tests for this provided by @shs96c as well. If someone wanted to submit a pull requests for documentation for these rules, we'd happily take it. |
Port Buck gwt_binary() native rule: [1] to Skylark. TEST PLAN: bazel build gerrit-gwtui:ui_optdbg * [1] facebook/buck#109 Change-Id: I24d11f10928a8000304bc4233156ddc50fad8931
I'm looking into adding build rules for building GWT applications. I'm curious what folks think the arguments to the rules should look like.
gwt_library()rule, or can we just usejava_library()? My intuition is that if you use JSNI in your Java code, then you need to classify that as agwt_library().gwt_binary()target and ajava_binary()target? How do you ensure that thegwt_binary()pulls inguava-gwt-17.0.jarwhile thejava_binary()pulls inguava-17.0.jar? One option is to have parallel versions of each subgraph, but that seems messy.gwt_binary()(or this whole system of rules) so that it is easy to run the GWT app in developer mode? Ideally, a developer should not have to runbuck buildafter every code modification in order to reload the GWT app. The existing edit-refresh-test cycle needs to be preserved.The text was updated successfully, but these errors were encountered: