Source Map support

[rvagg]

This issue is for discussion of adding Source Map support to Ender so we can ship map data with the ender.min.js for easier debugging. Since (I believe) source maps can map a single minified file to multiple original source files it might enable some neat debugging foo where the original npm package source files can be referenced so you'd even get line-numbers back to your original source file(s).

Resources:

• Source Map Revision 3 Proposal
• Mozilla Dev Tools Source Map documentation
• Google Closure Inspector Guide (this is a bit out of date, refers to Firebug but I think Chrome Dev Tools have better support for Source Maps now, the important bit is using Closure Compiler's --create_source_map)
• Closure Compiler SourceMap.java
• Mozilla's JavaScript (Node) source-map project - with both SourceMapConsumer and SourceMapGenerator (might be a good way to get started)

Nothing in UglifyJS yet, but perhaps this is something we can encourage for their 2.0 rewrite/rework. Alternatively perhaps we can switch to Closure Compiler if you need an Ender build with Source Map output.

[rvagg]

This video just popped up yesterday: https://plus.google.com/u/1/110412141990454266397/posts/Nvr6Se6eAPh

Using source maps to debug GWT (Java) code that's compiled into JavaScript, i.e. it's just a generic line mapping so it can point to any source language file.

[rvagg]

@tobie is working on this too for https://github.com/tobie/modulr-node has a lot of similarities to Ender, I suspect there will be a lot of overlap with his work. A couple of resources from him:

• UglifyJS issue for tracking this: Compression of CoffeeScript output mishoo/UglifyJS#315
• HTML5Rocks tutorial: http://www.html5rocks.com/en/tutorials/developertools/sourcemaps/

[tobie]

Yup. Would love to see support for this.

[ded]

yeah. soo.... 12 days later. I'll +1 this....

[download13]

If you want a source map from the modules to the end minified file it would probably have to be done in stages. Basically the process would look like this:

Module 1 >--\
Module 2 >--+[map segment 1]-->-- ender.js -->[map segment 2]-- ender.min.js
Module 3 >--/

Whatever gets used for minification, it will have to support source maps to produce map segment 2. Luckily, it also does all the heavy lifting mapping-wise.

All we need to worry about is map segment 1, which is (thankfully) much simpler. Since we're not refactoring the code in this stage it's essentially just mapping the indentation offsets for columns (which can be done away with if we don't care about prettifying the indentations in ender.js) and the line number offsets caused by placing the module code below the ender-js closure. For example, let's say that ender-js is 50 lines long, the closure header for each module is 5 lines, and the closure footer is 3 lines. Then all the information we need for map segment 1 is as easy as:

var sourceMap = {'mod1.js': 'mod1source', 'mod2.js': 'fizz', 'mod3.js': 'buzz'};
var offsetMap = {};
var offset = 50;
for(var name in sourceMap) {
  offset += 5;
  offsetMap[name] = offset;
  offset += sourceMap[name].split('\n').length - 1; // Get number of lines
  offset += 3;
}

offsetMap is now our map segment 1. It's not an actual source map, but for our purposes it will work as one. This pseudocode demonstrates:

names = Object(offsetMap).keys()
source_map.sources = names

foreach(segment in source_map):
  enderjs_line = segment.field3 // Line of the source file according to map segment 2

  for(i=0;i<names.length;i++):
    // If this source line falls within the range we can attribute to this module
    if(enderjs_line > offsetMap[names[i]] && enderjs_line < offsetMap[names[i + 1]]):
      segment.field3 -= offsetMap[names[i]] // The source line can changed
      segment.field2 = i // Field 2 references which source file it came from

It would need some debugging, but you get the idea. We have a list of the modules we want to map from so we assign it to the sources field in the source map. Then we iterate through each segment in the source map checking which module it belongs in and removing the line number offset it got when we put it into ender.js.

As I said, this ignores the indentation corrections that would be needed if you wanted to make ender.js pretty (which the builder does at the moment), but the same basic concept could be applied to the column numbers as well.

It's now 2am here, so I'm going to bed. Tell me if I made any horrendous mistakes and I'll try to fix them tomorrow.

[download13]

So I tried implementing a little demo of this, but to be honest the build code path in Ender is rather complex and I'm not really sure how to modify it without breaking something or pissing everyone off, so hopefully someone who's more familiar with how the internals work can have a go at it.

[rvagg]

Feel free to put in a PR for broken code, even if it's just a start, we can shunt it to a different branch as a play-area for this. Others may want to join in on what you've tried.

[download13]

Alright. One thing though. Do you know what they mean in the Source Map v3 document by "LSB first"? I was under the impression that meant that the LSB is in the left-most slot and increasing significance goes to the right, but the Mozilla implementation that you referenced in your initial post doesn't seem to translate to MSB-first for use by JS as a Number.

Were there recent changes made, or am I confused as to what LSB-first means?

[rvagg]

eeek, that's a bit odd, I'd assume LSB-first would mean the left is the least significant.

might also be worth peeking into the CoffeeScript source, it's not too hard to understand, as of 1.3 I think they've got SourceMap support.

[download13]

After looking through the WebKit code for their source map reader in the developer toolbar, I think I've figured out what they meant. When they say "least significant first" they mean that the groups of bits are in order of least significant group to most significant group. The individual bits are still in MSB-first order.
That makes a lot more sense now that I think about it, but hopefully the wording gets a little clearer before the document is finalized.

[download13]

Do you know if the WebKit Inspector is working correctly with names fields from source maps yet? As far as I can tell my serializer is sending them out correctly but the inspector isn't translating variable names from their minified versions. I'm not sure if I should keep looking for bugs in my code or just assume it's WebKit's fault.

On that subject, Source Maps are mostly working. The code is rather hacky at the moment, but that can be fixed later. I'll put in a pull request so you can see what it looks like so far and try it out.

I couldn't get the unit tests to work on Windows, so I'm not sure if it passes. Actually, I broke so much to get the maps working I'm almost sure it wouldn't pass.

[rvagg]

Perhaps @paulirish would be the one to ask about WebKit's SourceMap names support. How does it go in Firebug?

[paulirish]

I am not, unfortunately. cc @ryanseddon in case he knows.

The traceur source maps support was done via the mozilla js generator.. whereas I only have experience using closure compiler to generate maps. But both work with webkit inspector just fine. As to the specifics of names fields I have no idea. :/

[ryanseddon]

If you're referring to adding unminified names as a watch expression in the dev tools it won't work and would require some sort of reverse mapping to do a lookup.

[download13]

Not necessarily as a watch expression, but being able to see variables by their unminified names in the local scope section when execution is paused from a breakpoint. Also, hovering over a variable in the code while paused.

[download13]

@rvagg: Firebug doesn't support source maps as far as I know. It needs Closure Inspector, which I'm having a hard time with right now. It's also deprecated so I doubt it would support Source Map v3 anyway.

[rvagg]

I'll pull your code into a separate branch and have a play with it during the week when I can find some time. Good work on implementing a Closure minifier path, the plan is to make the minifier plug gable anyway so I guess if you want SourceMaps on your build then you'd have to run with a minifier that'll do it for you. Hopefully Uglify will get it sooner rather than later tho.
BTW, it'd be good if you could emulate the code style of the rest of the project as much as possible, I know it's not what most are used to. I'll get a code style doc up on the wiki soon as we have had some discussions about basic rules, which are reflected mostly in the 1.0-wip branch.

[download13]

I wasn't going to bother just for a mockup, but for project code I assumed as much.
How are those class-like things supposed to work anyway? They look like they're supposed to act like classes, but they don't have constructors and just use binds everywhere. Is there some way to map features between prototypal constructors and those? That's the sort of thing that would be good to have in the style doc.

[ryanseddon]

@download13 yeah wouldn't bother with closure inspector. I got it working but you need ff3.6, firebug 1.5 (thats the only combination that works) and like you said it only works with v1 source maps.

[rvagg]

re resources, @ryanseddon's article is quite a good reference on source maps too: http://www.thecssninja.com/javascript/source-mapping thanks Ryan

[rvagg]

more from @ryanseddon, I wasn't aware that UglifyJS2 was usable yet, great stuff! http://www.thecssninja.com/javascript/multi-level-sourcemaps

[amccollum]

Source maps are now being generated in ender@2.0.x. Finally.