🏗🚀✨ Significantly speed up gulp dist and gulp check-types

[rsimha]

Background:

• We use closure compiler (a Java binary with a slow start up time) to minify the AMP runtime.
• About two years ago, gulp dist used to take ~2 minutes on Linux, ~3 minutes on a Macbook Pro, and ~11 minutes on Travis to minify the runtime and all extensions.
• Today, it takes ~7 minutes on Linux, ~13 minutes on a Macbook pro, and ~35 minutes on Travis. (gulp dist is the top cause of slow CI builds.)
• An effort was started about a year ago to minify the runtime and all extensions with a single invocation of closure compiler. This effort, known as single-pass (Integrate single_pass into gulp #16922), has been around for a few months, but has not yet replaced the default multi-pass build due to concerns around correctness (single pass closure compiler issue tracker #16847).
• More recently, we stopped using the gulp-closure-compiler plugin (defunct, hasn't seen an update in 3 years) in favor of the native gulp plugin of google-closure-compiler (official repo for closure compiler, updated monthly) (🏗 Replace gulp-closure-compiler with google-closure-compiler (reprise) #21900).

What this PR does:

Adopts nailgun, which can start up any java binary in the background in server mode, and allow it to be reused during successive calls while eliminating the long start-up time.

Changes:

• Switch build-system/runner/dist/runner.jar to a single jar instead of a jar-in-jar (to work with nailgun)
• Copy the v1.0 nailgun binaries from https://github.com/facebook/nailgun to build-system/runner/nailgun/
• Update gulpfile.js and build-system/tasks/compile.js to start up and use a nailgun server during gulp dist and gulp check-types (on supported platforms)

Running time improvements:

• gulp dist (all extensions):

  • Linux: ~7 minutes → ~2.6 minutes
  • Macbook pro: ~13 minutes → ~3.5 minutes
  • Travis CI: ~35 minutes → ~8.2 minutes

• gulp dist (no extensions):

  • Linux: ~1 minute → ~30 seconds
  • Macbook pro: ~2 minutes → ~45 seconds
  • Travis CI: ~4 minutes → ~1.6 minutes

• gulp check-types:

  • Linux: ~40 seconds → ~25 seconds
  • Macbook pro: ~1 minute → ~40 seconds
  • Travis CI: ~2 minutes → ~1.3 minutes

References:

• Original nailgun project: http://www.martiansoftware.com/nailgun
• Working version of nailgun: of https://github.com/facebook/nailgun
• Implementation that uses it for closure: https://www.npmjs.com/package/@teppeis/closure-gun
• Creating a singe-jar version of runner.jar: https://stackoverflow.com/q/515428

Current release:

• https://github.com/facebook/nailgun/releases/tag/nailgun-all-v1.0.0

Fixes #10277

[rsimha]

Okay, I've tested this PR a bunch of different ways, and it's yielding a significant speed up with no apparent downsides. (So far 🤞)

Ready for review!

/to @cramforce @erwinmombay @jridgewell @choumx

[rsimha]

The numbers are in. On average, this PR results in a:

• ~4x speed up of gulp dist (all extensions)
• ~2x speed up of gulp dist (no extensions)
• ~1.5x speed up of gulp check-types

See #21939 (comment) for more details.

[cramforce]

Very cool! Can this also improve perf for local dev by keeping the compiler running in the background?

[cramforce]

Amazing! Two little comments

[rsimha]

Comments addressed.

Re: using this for local dev:

• As written, a nailgun server will be used forgulp dist or gulp check-types on linux or macos, irrespective of whether it's local dev or CI. (The server is cleaned up whether or not the gulp task is successful.)
• If you were asking about using this to start up an always-running compiler instance, it's definitely possible:

# Start runner.jar as a nailgun server
# You can drop the '&' and use a separate terminal for the server
$ java -XX:+TieredCompilation -server -cp \
third_party/nailgun/nailgun-server.jar:build-system/runner/dist/runner.jar \
com.facebook.nailgun.NGServer [PORT] &
NGServer 1.0.0-SNAPSHOT started on all addresses, port 2113.

# Use nailgun to compile code with runner.jar
$ echo "const foo = 42; function bar() { return foo * 2; }" > foo.js
$ third_party/nailgun/nailgun-runner [--nailgun-port PORT] \
org.ampproject.AmpCommandLineRunner -- --js foo.js
var foo=42;function bar(){return 2*foo};

# Stop nailgun server
# Ctrl+C works if the server is started without an '&'
$ third_party/nailgun/nailgun-runner [--nailgun-port PORT] ng-stop
NGServer shut down.

[jridgewell]

This brought gulp dist on my 2015 Macbook Pro from 34min to 6.58min. Also, my computer is actually usable during this time, unlike the old setup. 😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍😍

[rsimha]

I've tested this PR in a bunch of ways:

• Tested timing / reliability on my Linux workstation, my 2017 Macbook Pro, and Travis CI
• Manually tested several component pages using gulp dist --fortesting and gulp serve --compiled
• Made sure sourcemaps line up correctly by running gulp dist --noextensions --type canary --full_sourcemaps and comparing v0.js and v0.js.map using @jridgewell's sourcemaps comparison tool
• Timed gulp dist on @jridgewell's 2015 Macbook Pro (~6 mins) and @danielrozenberg's Linux laptop (~2 mins)

[cramforce]

RE: Local dev: Could you wrap that in a gulp task?

[rsimha]

@cramforce Sent you #21977