Build Script Rewrite

[sourrust]

Jumping from #1369, it is time to layout a plan for writing the build script to be easier to maintain, remove the complexity within the script, and making it easier for people new to the project to jump in and start contributing.

So what is wrong with our the current script?

Issues

• The current "bus factor" of the build script
• Unmaintained gear.js
• Security of dependencies in gear.js
• Complexity within the script

Bus Factor

I was the one to convert the old python build script to node with the goal being to make it easier to include continuous integration via Travis CI and reducing the amount of dependencies need to start to just node.js. Now that I'm not contributed as much as I use to, the concern of maintaining this script was in the forefront of mind of a while. Using libraries that actually are maintained and have good documentation would make the build script less of a black box and easier to change if there was a new requirement for it. Actual suggestions of libraries will be continued in the "Task Management" and "Module Bunder" sections.

Unmaintained gear.js

For what we got out of it, gear.js was a good start for what our build script needed previously. Unfortunately, with the more recent security issues and the last commit for gear.js being almost a year ago, it is safe to say that we need to move away from it. It was a poor choice on my part, but it was the only build system I found that would allow me to keep the same command line interface for run certain types of builds. While others like gulp, grunt, webpack, or any of the several different build systems out them required the build file to be in the root directory.

Complexity

Most of the complexity of the script comes from having to write our own of the following:

• Module bundler
• Parser for comment headers
• Mangler for long keyword names

There are a handful of module bundlers that alleviate this burden off our plate, but this change would make it more difficult for the site's version of the build script, which isn't using node. The parsing of starting comment header could be moved to a separate JSON file or with the help of whatever module bundler we choose, could make this part a bit easier. I haven't put much thought into this issue, but it would be nice to get some assistance on this parsing delima. And last is the mangler that can simply shift the responsibility of uglify and it's properties mangler.

Module Bundler

• browserify
• rollup

This is probably the biggest change as it will remove the comment header field Required and we would start using either common.js or ES6 styled importing. browserify would be the one that is using the common.js pattern and the most obvious benefit would be being able to use the project as is with node. The downside is that the bundled size would more than likely increase significantly.

The other module bundler, rollup, is something I use at work and is the one I personally prefer. Working in ES6 is the eventual future, making it an easier transition when the majority of browsers support features in ES6. The other thing I like about it is, all the module import/export information gets removed resulting in a smaller bundled size and still use the small UMD export we have currently in the core. The only downside would be actually learning ES6 modules.

And of course, like I said previously, there will be an issue with the site's differing build script not being able to correctly parse and bundle the library for either option.

Task Management

• async
• bluebird
• co

To continue to have fast build times, we need to think about which asynchronous type we are going to use to have a consistent style throughout the build script. Both bluebird and co are libraries that handle a specific type of asynchronous flow, bluebird is focused on promise and co is on coroutines. While async is a library for cleaning up asynchronous behaviour via utility functions.

I've used both async and bluebird pretty heavily so those are the two I'm leaning toward. However, if there are other libraries out in the wild that I haven't mentioned, feel free mention them and I'll consider them in the rewrite.

In opening this issues before starting the rewrite, I'm hoping to get ideas from the team, other contributors, and people simply using highlight.js. Feel free to discuss the ideas I have laid out, and don't consider any of them set in stone. Thank you for reading.

---

cc: @isagalaev @Sannis @marcoscaceres

[marcoscaceres]

Just minor thing: for "Task Management", I don't think you need any of those libraries any more (even if we were targeting node 4.x). You have generators and promises, which basically give you all the async machinery you need to create a task runner. For example, the following 30 odd lines of code implement Async/await:

https://github.com/marcoscaceres/async/blob/gh-pages/lib/async.js

Example of usage (see any async.task(function*(){):
https://github.com/w3c/respec/blob/develop/tools/release.js

[sourrust]

That may be true, but the reason to use these libraries is a matter of convenience. For example, if there was a non-promise interface, I would have to make sure to wrap a node callback styled function into one that returns a promise in order to use these generator functions. Bluebird help get rid of that with the promisify and promisifyAll interface. Not only that, but I can continue to use this library with generator function support via coroutines.

The suggestion of the async module might be less relevant now, but bluebird is still something I would consider.

[marcoscaceres]

The suggestion of the async module might be less relevant now, but bluebird is still something I would consider.

Makes sense, thanks for the clarification. Might also look into Bluebird - as I also end up promesify'ing all the things.

[sourrust]

@isagalaev and @Sannis, any feedback on this? My main concerns about the rewrite is the change of module bundling and header parsing. Everything else isn't really going to bring any breaking changes -- as far as the site's build script is concerned.

[isagalaev]

I hope to find some time to look into this and provide my promised input as well in a few days after I'm back from a family vacation. My time is a little sparse to pay full attention right now :-)

[marcoscaceres]

Some unsolicited input from me, but please take a look at how we currently build highlight.js in our project (have to navigate to the install folder, call npm install in there, etc.):

"build:highlight": "cd node_modules/highlight.js/ && npm install && node ./tools/build.js -n xml javascript css http markdown xquery json && cd ../../",

https://github.com/w3c/respec/blob/develop/package.json#L51

That's not ideal, IMHO. It would be nice if once hljs is installed, I can request it to build as a command line tool - so it just generates the highlighting code:

hljs --build -n xml json --output some/path 

Hope that makes sense!

[marcoscaceres]

(fixed comment above)

[saschanaz]

I tried reading the build script and found gearjs.org is now gone. No more docs for gearjs.

Edit: The official repository is gone but the author still has his own one, with the docs! CNAME still points to the dead URL so I forked it: https://saschanaz.github.io/gear

Update: https://twobit.github.io/gear/ is now alive.

[marcoscaceres]

ouch :(

On a different note... maybe it's time for this project to consider the pull request hack? This project is heavily relied on by a lot of people, so maybe it's time to open it up a bit more to trusted members of the community to maintain it?

[isagalaev]

Closing in favor of #1759.