Use WebAssembly to speed up SourceMapConsumer #306

fitzgen · 2018-01-08T19:58:00Z

…parse in benchmark

…method

Easier to graph / work with after benchmarking this way.

paulirish · 2018-01-16T16:56:49Z

This is awesome.

I just tried it out in https://github.com/danvk/source-map-explorer and my particular usecase went from 20.5s to 8.2s. Nicely done. cc @danvk

fitzgen · 2018-01-16T19:14:54Z

Expect a detailed blog post in the coming days... ;)

tromey · 2018-01-16T19:48:23Z

README.md

 ```js
-var consumer = new sourceMap.SourceMapConsumer(rawSourceMapJsonData);
+consumer.destroy();


Bummer but I guess there's no good way around it.

I have ideas to improve this in follow ups:

Either add an async RAII-ish function like:

SourceMapConsumer.with = async function (rawSourceMap, f) { const consumer = await new SourceMapConsumer(rawSourceMap); try { await f(consumer); } finally { consumer.destroy(); } };

Or alternatively give every SourceMapConsumer its own wasm module instance, make the Mappings a global/implicit in the wasm heap, and let the GC manage lifetimes. Requires further investigation.

tromey · 2018-01-16T20:09:20Z

bench/bench-shell-bindings.js

  };
 }

 if (typeof print !== "function") {
-  print = console.log.bind(console);
+  print = function (x = "") {
+    console.log(`${x}`);


Not important but I thought console.log was self-bound already.

I know it differed across engines at one point in time, unsure if it still does or not.

In fact, the shell version of the benchmarks doesn't work anymore (added too many options that are only exposed in the Webpage), so I should just remove it.

tromey · 2018-01-16T20:22:32Z

lib/read-wasm.js

+  module.exports = function readWasm() {
+    if (typeof mappingsWasmUrl !== "string") {
+      throw new Error("You must provide the URL of lib/mappings.wasm by calling " +
+                      "SourceMapConsumer.initialize({ 'lib/mappings.wasm': ... }) " +


Would it be crazy to just have the wasm be a giant string constant in some .js file?

It would bloat the code size, which I took quite a bit of effort in keeping small.

tromey · 2018-01-16T20:37:55Z

lib/source-map-consumer.js

+          // The offset fields are 0-based, but we use 1-based indices when
+          // encoding/decoding from VLQ.
+          generatedLine: offsetLine + 1,
+          generatedColumn: offsetColumn + 1


Not introduced here, but elsewhere it says that columns are 0-based (what I recall as well), so I wonder if this is a latent bug.

Perhaps? I had to do this to get the tests passing with the wasm. I think we can investigate further in a follow up.

tromey · 2018-01-16T20:41:10Z

lib/source-map-consumer.js

      for (var j = 0; j < sectionMappings.length; j++) {
        var mapping = sectionMappings[j];

-        var source = section.consumer._sources.at(mapping.source);
-        source = util.computeSourceURL(section.consumer.sourceRoot, source, this._sourceMapURL);
+        var source = util.computeSourceURL(section.consumer.sourceRoot, source, this._sourceMapURL);
        this._sources.add(source);
        source = this._sources.indexOf(source);

        var name = null;


I didn't examine more context but I suspect this line should be removed.

The var name = null; line? The name variable is assigned to and used a little bit down this method.

tromey · 2018-01-16T20:41:45Z

lib/source-map-consumer.js

+    var sourceRoot = this.sourceRoot;
+    mappings.map(function (mapping) {
+      var source = null;
+      if(mapping.source !== null) {


Trivia: space after the "if".

tromey · 2018-01-16T20:49:11Z

test/test-source-map-generator.js

-exports['test .fromSourceMap'] = function (assert) {
-  var map = SourceMapGenerator.fromSourceMap(new SourceMapConsumer(util.testMap));
+exports['test .fromSourceMap'] = async function (assert) {
+  var map = SourceMapGenerator.fromSourceMap(await new SourceMapConsumer(util.testMap));


I suppose it's not a big deal to leak the consumers in the tests.

Hm, I thought I caught them all, did I miss some?

I didn't see where these consumers were destroyed. But maybe fromSourceMap does and I missed it.

tromey

This is big enough and covers enough different things that it is going to suffer from the other side of bikeshedding -- that is, I tried to read it all but I'm not sure I can really do it justice.

I do wonder if it's possible to remove the need for a separate wasm URL and thus (IIUC) the need for the constructor to return a promise.

Otherwise I'm inclined to push ahead. Perhaps I ought to go address #291 and #305...

fitzgen · 2018-01-17T22:31:30Z

I do wonder if it's possible to remove the need for a separate wasm URL and thus (IIUC) the need for the constructor to return a promise.

Compiling wasm is still done off-thread, and returns a promise, so even if we already had the blob, the constructor would still return a promise.

tromey · 2018-01-17T23:05:52Z

Compiling wasm is still done off-thread, and returns a promise, so even if we already had the blob, the constructor would still return a promise.

Thanks. Never mind then.

…-generator.js`

fitzgen · 2018-01-17T23:17:44Z

Ok, I think this is just about ready to land. Waiting on OSX jobs in Travis CI.

fitzgen · 2018-01-18T17:34:47Z

FYI, https://hacks.mozilla.org/2018/01/oxidizing-source-maps-with-rust-and-webassembly/ is live with details on the experience.

Going to go ahead and merge this PR!

Kagami · 2018-01-22T14:13:22Z

CONTRIBUTING.md

+```
+
+The resulting `wasm` file will be located at
+`source-map-mappings-c-api/target/wasm32-unknown-unknown/release/source_map_mappings.wasm`.


s/c-api/wasm-api/

Nice catch!

Kagami · 2018-01-22T14:16:54Z

bench/README.md

-Open `bench.html` in a browser and click on the appropriate button.
+```
+$ cd source-map/
+$ python -m SimpleHTTPServer


Better python2 -m SimpleHTTPServer or python3 -m http.server

Kagami · 2018-01-22T14:28:46Z

lib/read-wasm.js

+  module.exports = function readWasm() {
+    if (typeof mappingsWasmUrl !== "string") {
+      throw new Error("You must provide the URL of lib/mappings.wasm by calling " +
+                      "SourceMapConsumer.initialize({ 'lib/mappings.wasm': ... }) " +


initialize() accepts just string though?

The public SourceMapConsumer.initialize function takes an object, the interal initialize function in lib/read-wasm.js just takes a string.

This is a leftover from #306.

fitzgen added 30 commits December 19, 2017 15:15

Make the SourceMapConsumer constructor return a promise

f0e7420

All tests passing with Rust+wasm

45c04ed

Unbreak the benchmark

38e98f4

Use allGeneratedPositionsFor instead of _generatedMappings to ensure …

084ba13

…parse in benchmark

Fix function name

e652d65

Update lib/mappings.wasm to 0.2.0

367356c

Re-add support for using on the web

f76c60b

Update bench for running in a webpage

6ca73c7

Update webpack dependency

c4aa0a3

Add tests for out-of-range line numbers

e80a40b

Document SourceMapConsumer.initialize

b90fa03

Document asynchronicity of new SourceMapConsumer and its destroy …

d395699

…method

Breaking version bump

bd9fdc4

Add CONTRIBUTING.md instructions for updating lib/mappings.wasm

8b10e38

Convert the wasm's error codes into meaningful error messages

fa87d4b

Remove VLQ decoding code that is now handled in wasm

e222ef6

Add more variety of benchmarks

2fbfb52

Reuse the existing mapping object in eachMapping

b2b4adc

Make bench iterations easily tunable

7b8e6da

Add wasm profiling support

2621835

Update mappings.wasm

488686a

Avoid slow script dialogs during warm up

f676051

Update lib/mappings.wasm

338b341

Add another source map to run benchmarks on

f25ff92

Save individual samples

15eab8b

Easier to graph / work with after benchmarking this way.

Add an angular js source map as benchmarking input

b3b00f0

Update lib/mappings.wasm to source-map-mappings @ 0.3.0

3b61129

Add CHANGELOG entry for 0.7.0

353d481

Small README docs fixes for wasm stuff

ab311fd

Allow multiplying the size of benchmark inputs

ab43cc4

fitzgen added 2 commits January 15, 2018 11:51

bench: Add implementation and browser field to CSV generation

53c566a

dist: Re-build

a3d7d94

tromey reviewed Jan 16, 2018

View reviewed changes

tromey approved these changes Jan 16, 2018

View reviewed changes

fitzgen added 3 commits January 17, 2018 14:39

bench: remove broken benchmarking with a JS shell

3b664b3

I think node >= 8 has WebAssembly

59fab43

Add space after "if" before "("

ef24a04

fitzgen added 2 commits January 17, 2018 15:06

Properly destroy SourceMapConsumer instances in `test/test-source-map…

80d2199

…-generator.js`

Also run CI tests on node 8

f878008

fitzgen mentioned this pull request Jan 18, 2018

Large speed ups in iterating over all mappings #308

Merged

fitzgen merged commit 4a047e7 into mozilla:master Jan 18, 2018

This was referenced Jan 18, 2018

See if we can utilize WebAssembly source-map stuff broccolijs/broccoli-concat#120

Open

See if we can utilize WebAssembly source-map stuff ef4/fast-sourcemap-concat#45

Open

Kagami reviewed Jan 22, 2018

View reviewed changes

jridgewell mentioned this pull request Jul 3, 2019

Remove unused fromVLQSigned function #402

Merged

jasonLaster pushed a commit that referenced this pull request Oct 29, 2019

Remove unused fromVLQSigned function (#402)

0f3e955

This is a leftover from #306.

Use WebAssembly to speed up SourceMapConsumer #306

Use WebAssembly to speed up SourceMapConsumer #306

fitzgen commented Jan 8, 2018

paulirish commented Jan 16, 2018

fitzgen commented Jan 16, 2018

tromey Jan 16, 2018

fitzgen Jan 17, 2018

tromey Jan 16, 2018

fitzgen Jan 17, 2018

tromey Jan 16, 2018

fitzgen Jan 17, 2018

tromey Jan 16, 2018

fitzgen Jan 17, 2018

tromey Jan 16, 2018

fitzgen Jan 17, 2018

tromey Jan 17, 2018

tromey Jan 16, 2018

tromey Jan 16, 2018

fitzgen Jan 17, 2018

tromey Jan 17, 2018

tromey left a comment

fitzgen commented Jan 17, 2018

tromey commented Jan 17, 2018

fitzgen commented Jan 17, 2018

fitzgen commented Jan 18, 2018

Kagami Jan 22, 2018

fitzgen Jan 22, 2018

Kagami Jan 22, 2018

fitzgen Jan 22, 2018

Kagami Jan 22, 2018

fitzgen Jan 22, 2018

Use WebAssembly to speed up SourceMapConsumer #306

Use WebAssembly to speed up SourceMapConsumer #306

Conversation

fitzgen commented Jan 8, 2018

paulirish commented Jan 16, 2018

fitzgen commented Jan 16, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tromey left a comment

Choose a reason for hiding this comment

fitzgen commented Jan 17, 2018

tromey commented Jan 17, 2018

fitzgen commented Jan 17, 2018

fitzgen commented Jan 18, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment