An experimental Rust parser for WordPress Gutenberg post format
Clone or download
Hywan feat(wasm) Use a cache for name decoding.
It boosts performance by 13% on `moby_dick_parsed`.
Latest commit 9dc6a63 Nov 2, 2018

README.md

The Gutenberg logo


The Gutenberg post parser

Gutenberg is a new post editor for the WordPress ecosystem. A post has always been HTML, and it continues to be. The difference is that the HTML is now annotated. Like most annotation languages, it is located in comments, like this:

<h1>Famous post</h1>

<!-- wp:component {attributes: "as JSON"} -->
lorem ipsum
<!-- /wp:component -->

The parser analyses a post and generates an Abstract Syntax Tree (AST) of it. The AST is then accessible to many languages through bindings.

Platforms and bindings, aka targets

The parser aims at being used on different platforms, such as: the Web (within multiple browsers), Web applications like Electron, native applications like macOS, iOS, Windows, Linux etc.

Thus, the parser can be compiled as:

This project uses Justfile as an alternative to Makefile. Every following command will use just, you might consider to install it. To learn about all the commands, just just --list.

Note: Right now, this project needs rustc nightly to compile the WASM target. This target should switch to stable in a couple of months. Since then, be sure to run the latest nightly version with rustup update nightly.

Binary

To compile the parser to a binary, run:

$ just build-binary
$ ./target/release/gutenberg-post-parser --emit-json tests/fixtures/gutenberg-demo.html

Static library

To compile the parser to a static library, run:

$ just build-library
$ ls target/release/

WebAssembly

To compile the parser to a WebAssembly binary, run:

$ just build-wasm
$ ./bindings/wasm/bin/gutenberg-post-parser --emit-json tests/fixtures/gutenberg-demo.html

If you would like to test directly in your browser, run:

$ just build-wasm
$ just start-wasm-server
$ open localhost:8888

Learn more about the WebAssembly binding.

ASM.js

To compile the parser to an ASM.js module, run:

$ just build-asmjs
$ just start-asmjs-server
$ open localhost:8888

The ASM.js module is slower than the WebAssembly binary, but it is useful for Internet Explorer compatibility, or any browser that does not support WebAssembly. Remember that ASM.js is just a JavaScript file.

Learn more about the ASM.js binding.

NodeJS

To compile the parser to a NodeJS native module, run:

$ just build-nodejs
$ ./bindings/nodejs/bin/gutenberg-post-parser --emit-json tests/fixtures/gutenberg-demo.html

Learn more about the NodeJS binding.

C

To compile the parser to a C header, run:

$ just build-c
$ ./bindings/c/bin/gutenberg-post-parser tests/fixtures/gutenberg-demo.html

PHP

To compile the parser to a PHP extension, run:

$ just build-php
$ ./bindings/php/bin/gutenberg-post-parser --emit-debug tests/fixtures/gutenberg-demo.html

To load the extension, add extension=gutenberg_post_parser in the php.ini file (hint: Run php --ini to locate this configuration file), or run PHP such as php -d extension=gutenberg_post_parser file.php.

Learn more about the PHP binding.

Performance and guarantee

The parser guarantees to never copy the data in memory while analyzing, which makes it fast and memory efficient.

WASM binary

A yet-to-be-official benchmark is used to compare the performance of the actual Javascript parser against the Rust parser compiled as a WASM binary so that it can run in the browser. Here are the results:

file Javascript parser (ms) Rust parser as a WASM binary (ms) speedup
demo-post.html 13.167 0.137 × 96
shortcode-shortcomings.html 26.784 0.225 × 119
redesigning-chrome-desktop.html 75.500 0.905 × 83
web-at-maximum-fps.html 88.118 0.698 × 126
early-adopting-the-future.html 201.011 0.927 × 217
pygmalian-raw-html.html 311.416 1.016 × 307
moby-dick-parsed.html 2,466.533 14.673 × 168

The WASM binary of the Rust parser is in average 159 times faster than the actual Javascript implementation. The median of the speedup is 126.

Learn more with this blog post: From Rust to beyond: The WebAssembly galaxy.

ASM.js module

ASM.js is a fallback for environments that do not support WebAssembly, like Internet Explorer. The same benchmark is used for ASM.js than for WASM, and compares the performance of the actual Javascript parser against the Rust parser compiled as a ASM.js module so that it can run in the browser. Here are the results:

file Javascript parser (ms) Rust parser as an ASM.js module (ms) speedup
demo-post.html 15.368 2.718 × 6
shortcode-shortcomings.html 31.022 8.004 × 4
redesigning-chrome-desktop.html 106.416 19.223 × 6
web-at-maximum-fps.html 82.920 27.197 × 3
early-adopting-the-future.html 119.880 38.321 × 3
pygmalian-raw-html.html 349.075 23.656 × 15
moby-dick-parsed.html 2,543.750 361.423 × 7

The ASM.js module version of the Rust parser is in average 6 times faster than the actual Javascript implementation. The median of the speedup is 6.

Learn more with this blog post: From Rust to beyond: The ASM.js galaxy.

PHP native extension

Another benchmark has been used to compare the performance of the actual PHP parser against the Rust parser compiled as a PHP native extension. Here are the results:

file PHP parser (ms) Rust parser as a PHP extension (ms) speedup
demo-post.html 30.409 0.0012 × 25341
shortcode-shortcomings.html 76.39 0.096 × 796
redesigning-chrome-desktop.html 225.824 0.399 × 566
web-at-maximum-fps.html 173.495 0.275 × 631
early-adopting-the-future.html 280.433 0.298 × 941
pygmalian-raw-html.html 377.392 0.052 × 7258
moby-dick-parsed.html 5,437.630 5.037 × 1080

The PHP extension of the Rust parser is in average 5230 times faster than the actual PHP implementation. The median of the speedup is 941.

Note that memory limit has been hit very quickly with the PHP parser, while the Rust parser as a PHP native extension has a small memory footprint.

Learn more with this blog post: From Rust to beyond: The C galaxy.

License

The license is a classic BSD-3-Clause:

New BSD License

Copyright © Ivan Enderlin. All rights reserved.

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

  • Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.

  • Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.

  • Neither the name of this project nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDERS AND CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.