Source Map support #127

Closed
rvagg opened this Issue Mar 15, 2012 · 22 comments

Comments

Projects
None yet
7 participants
@rvagg
Member

rvagg commented Mar 15, 2012

This issue is for discussion of adding Source Map support to Ender so we can ship map data with the ender.min.js for easier debugging. Since (I believe) source maps can map a single minified file to multiple original source files it might enable some neat debugging foo where the original npm package source files can be referenced so you'd even get line-numbers back to your original source file(s).

Resources:

Nothing in UglifyJS yet, but perhaps this is something we can encourage for their 2.0 rewrite/rework. Alternatively perhaps we can switch to Closure Compiler if you need an Ender build with Source Map output.

@rvagg

This comment has been minimized.

Show comment Hide comment
@rvagg

rvagg Mar 20, 2012

Member

This video just popped up yesterday: https://plus.google.com/u/1/110412141990454266397/posts/Nvr6Se6eAPh

Using source maps to debug GWT (Java) code that's compiled into JavaScript, i.e. it's just a generic line mapping so it can point to any source language file.

Member

rvagg commented Mar 20, 2012

This video just popped up yesterday: https://plus.google.com/u/1/110412141990454266397/posts/Nvr6Se6eAPh

Using source maps to debug GWT (Java) code that's compiled into JavaScript, i.e. it's just a generic line mapping so it can point to any source language file.

@rvagg

This comment has been minimized.

Show comment Hide comment
@rvagg

rvagg Mar 22, 2012

Member

@tobie is working on this too for https://github.com/tobie/modulr-node has a lot of similarities to Ender, I suspect there will be a lot of overlap with his work. A couple of resources from him:

Member

rvagg commented Mar 22, 2012

@tobie is working on this too for https://github.com/tobie/modulr-node has a lot of similarities to Ender, I suspect there will be a lot of overlap with his work. A couple of resources from him:

@tobie

This comment has been minimized.

Show comment Hide comment
@tobie

tobie Mar 22, 2012

Yup. Would love to see support for this.

tobie commented Mar 22, 2012

Yup. Would love to see support for this.

@ded

This comment has been minimized.

Show comment Hide comment
@ded

ded Mar 27, 2012

Member

yeah. soo.... 12 days later. I'll +1 this....

Member

ded commented Mar 27, 2012

yeah. soo.... 12 days later. I'll +1 this....

@download13

This comment has been minimized.

Show comment Hide comment
@download13

download13 May 1, 2012

Contributor

If you want a source map from the modules to the end minified file it would probably have to be done in stages. Basically the process would look like this:

Module 1 >--\
Module 2 >--+[map segment 1]-->-- ender.js -->[map segment 2]-- ender.min.js
Module 3 >--/

Whatever gets used for minification, it will have to support source maps to produce map segment 2. Luckily, it also does all the heavy lifting mapping-wise.

All we need to worry about is map segment 1, which is (thankfully) much simpler. Since we're not refactoring the code in this stage it's essentially just mapping the indentation offsets for columns (which can be done away with if we don't care about prettifying the indentations in ender.js) and the line number offsets caused by placing the module code below the ender-js closure. For example, let's say that ender-js is 50 lines long, the closure header for each module is 5 lines, and the closure footer is 3 lines. Then all the information we need for map segment 1 is as easy as:

var sourceMap = {'mod1.js': 'mod1source', 'mod2.js': 'fizz', 'mod3.js': 'buzz'};
var offsetMap = {};
var offset = 50;
for(var name in sourceMap) {
  offset += 5;
  offsetMap[name] = offset;
  offset += sourceMap[name].split('\n').length - 1; // Get number of lines
  offset += 3;
}

offsetMap is now our map segment 1. It's not an actual source map, but for our purposes it will work as one. This pseudocode demonstrates:

names = Object(offsetMap).keys()
source_map.sources = names

foreach(segment in source_map):
  enderjs_line = segment.field3 // Line of the source file according to map segment 2

  for(i=0;i<names.length;i++):
    // If this source line falls within the range we can attribute to this module
    if(enderjs_line > offsetMap[names[i]] && enderjs_line < offsetMap[names[i + 1]]):
      segment.field3 -= offsetMap[names[i]] // The source line can changed
      segment.field2 = i // Field 2 references which source file it came from

It would need some debugging, but you get the idea. We have a list of the modules we want to map from so we assign it to the sources field in the source map. Then we iterate through each segment in the source map checking which module it belongs in and removing the line number offset it got when we put it into ender.js.

As I said, this ignores the indentation corrections that would be needed if you wanted to make ender.js pretty (which the builder does at the moment), but the same basic concept could be applied to the column numbers as well.

It's now 2am here, so I'm going to bed. Tell me if I made any horrendous mistakes and I'll try to fix them tomorrow.

Contributor

download13 commented May 1, 2012

If you want a source map from the modules to the end minified file it would probably have to be done in stages. Basically the process would look like this:

Module 1 >--\
Module 2 >--+[map segment 1]-->-- ender.js -->[map segment 2]-- ender.min.js
Module 3 >--/

Whatever gets used for minification, it will have to support source maps to produce map segment 2. Luckily, it also does all the heavy lifting mapping-wise.

All we need to worry about is map segment 1, which is (thankfully) much simpler. Since we're not refactoring the code in this stage it's essentially just mapping the indentation offsets for columns (which can be done away with if we don't care about prettifying the indentations in ender.js) and the line number offsets caused by placing the module code below the ender-js closure. For example, let's say that ender-js is 50 lines long, the closure header for each module is 5 lines, and the closure footer is 3 lines. Then all the information we need for map segment 1 is as easy as:

var sourceMap = {'mod1.js': 'mod1source', 'mod2.js': 'fizz', 'mod3.js': 'buzz'};
var offsetMap = {};
var offset = 50;
for(var name in sourceMap) {
  offset += 5;
  offsetMap[name] = offset;
  offset += sourceMap[name].split('\n').length - 1; // Get number of lines
  offset += 3;
}

offsetMap is now our map segment 1. It's not an actual source map, but for our purposes it will work as one. This pseudocode demonstrates:

names = Object(offsetMap).keys()
source_map.sources = names

foreach(segment in source_map):
  enderjs_line = segment.field3 // Line of the source file according to map segment 2

  for(i=0;i<names.length;i++):
    // If this source line falls within the range we can attribute to this module
    if(enderjs_line > offsetMap[names[i]] && enderjs_line < offsetMap[names[i + 1]]):
      segment.field3 -= offsetMap[names[i]] // The source line can changed
      segment.field2 = i // Field 2 references which source file it came from

It would need some debugging, but you get the idea. We have a list of the modules we want to map from so we assign it to the sources field in the source map. Then we iterate through each segment in the source map checking which module it belongs in and removing the line number offset it got when we put it into ender.js.

As I said, this ignores the indentation corrections that would be needed if you wanted to make ender.js pretty (which the builder does at the moment), but the same basic concept could be applied to the column numbers as well.

It's now 2am here, so I'm going to bed. Tell me if I made any horrendous mistakes and I'll try to fix them tomorrow.

@download13

This comment has been minimized.

Show comment Hide comment
@download13

download13 May 2, 2012

Contributor

So I tried implementing a little demo of this, but to be honest the build code path in Ender is rather complex and I'm not really sure how to modify it without breaking something or pissing everyone off, so hopefully someone who's more familiar with how the internals work can have a go at it.

Contributor

download13 commented May 2, 2012

So I tried implementing a little demo of this, but to be honest the build code path in Ender is rather complex and I'm not really sure how to modify it without breaking something or pissing everyone off, so hopefully someone who's more familiar with how the internals work can have a go at it.

@rvagg

This comment has been minimized.

Show comment Hide comment
@rvagg

rvagg May 3, 2012

Member

Feel free to put in a PR for broken code, even if it's just a start, we can shunt it to a different branch as a play-area for this. Others may want to join in on what you've tried.

Member

rvagg commented May 3, 2012

Feel free to put in a PR for broken code, even if it's just a start, we can shunt it to a different branch as a play-area for this. Others may want to join in on what you've tried.

@download13

This comment has been minimized.

Show comment Hide comment
@download13

download13 May 3, 2012

Contributor

Alright. One thing though. Do you know what they mean in the Source Map v3 document by "LSB first"? I was under the impression that meant that the LSB is in the left-most slot and increasing significance goes to the right, but the Mozilla implementation that you referenced in your initial post doesn't seem to translate to MSB-first for use by JS as a Number.

Were there recent changes made, or am I confused as to what LSB-first means?

Contributor

download13 commented May 3, 2012

Alright. One thing though. Do you know what they mean in the Source Map v3 document by "LSB first"? I was under the impression that meant that the LSB is in the left-most slot and increasing significance goes to the right, but the Mozilla implementation that you referenced in your initial post doesn't seem to translate to MSB-first for use by JS as a Number.

Were there recent changes made, or am I confused as to what LSB-first means?

@rvagg

This comment has been minimized.

Show comment Hide comment
@rvagg

rvagg May 3, 2012

Member

eeek, that's a bit odd, I'd assume LSB-first would mean the left is the least significant.

might also be worth peeking into the CoffeeScript source, it's not too hard to understand, as of 1.3 I think they've got SourceMap support.

Member

rvagg commented May 3, 2012

eeek, that's a bit odd, I'd assume LSB-first would mean the left is the least significant.

might also be worth peeking into the CoffeeScript source, it's not too hard to understand, as of 1.3 I think they've got SourceMap support.

@download13

This comment has been minimized.

Show comment Hide comment
@download13

download13 May 4, 2012

Contributor

After looking through the WebKit code for their source map reader in the developer toolbar, I think I've figured out what they meant. When they say "least significant first" they mean that the groups of bits are in order of least significant group to most significant group. The individual bits are still in MSB-first order.
That makes a lot more sense now that I think about it, but hopefully the wording gets a little clearer before the document is finalized.

Contributor

download13 commented May 4, 2012

After looking through the WebKit code for their source map reader in the developer toolbar, I think I've figured out what they meant. When they say "least significant first" they mean that the groups of bits are in order of least significant group to most significant group. The individual bits are still in MSB-first order.
That makes a lot more sense now that I think about it, but hopefully the wording gets a little clearer before the document is finalized.

@download13

This comment has been minimized.

Show comment Hide comment
@download13

download13 May 5, 2012

Contributor

Do you know if the WebKit Inspector is working correctly with names fields from source maps yet? As far as I can tell my serializer is sending them out correctly but the inspector isn't translating variable names from their minified versions. I'm not sure if I should keep looking for bugs in my code or just assume it's WebKit's fault.

On that subject, Source Maps are mostly working. The code is rather hacky at the moment, but that can be fixed later. I'll put in a pull request so you can see what it looks like so far and try it out.

I couldn't get the unit tests to work on Windows, so I'm not sure if it passes. Actually, I broke so much to get the maps working I'm almost sure it wouldn't pass.

Contributor

download13 commented May 5, 2012

Do you know if the WebKit Inspector is working correctly with names fields from source maps yet? As far as I can tell my serializer is sending them out correctly but the inspector isn't translating variable names from their minified versions. I'm not sure if I should keep looking for bugs in my code or just assume it's WebKit's fault.

On that subject, Source Maps are mostly working. The code is rather hacky at the moment, but that can be fixed later. I'll put in a pull request so you can see what it looks like so far and try it out.

I couldn't get the unit tests to work on Windows, so I'm not sure if it passes. Actually, I broke so much to get the maps working I'm almost sure it wouldn't pass.

@rvagg

This comment has been minimized.

Show comment Hide comment
@rvagg

rvagg May 6, 2012

Member

Perhaps @paulirish would be the one to ask about WebKit's SourceMap names support. How does it go in Firebug?

Member

rvagg commented May 6, 2012

Perhaps @paulirish would be the one to ask about WebKit's SourceMap names support. How does it go in Firebug?

@paulirish

This comment has been minimized.

Show comment Hide comment
@paulirish

paulirish May 6, 2012

I am not, unfortunately. cc @ryanseddon in case he knows.

The traceur source maps support was done via the mozilla js generator.. whereas I only have experience using closure compiler to generate maps. But both work with webkit inspector just fine. As to the specifics of names fields I have no idea. :/

I am not, unfortunately. cc @ryanseddon in case he knows.

The traceur source maps support was done via the mozilla js generator.. whereas I only have experience using closure compiler to generate maps. But both work with webkit inspector just fine. As to the specifics of names fields I have no idea. :/

@ryanseddon

This comment has been minimized.

Show comment Hide comment
@ryanseddon

ryanseddon May 6, 2012

If you're referring to adding unminified names as a watch expression in the dev tools it won't work and would require some sort of reverse mapping to do a lookup.

If you're referring to adding unminified names as a watch expression in the dev tools it won't work and would require some sort of reverse mapping to do a lookup.

@download13

This comment has been minimized.

Show comment Hide comment
@download13

download13 May 6, 2012

Contributor

Not necessarily as a watch expression, but being able to see variables by their unminified names in the local scope section when execution is paused from a breakpoint. Also, hovering over a variable in the code while paused.

Contributor

download13 commented May 6, 2012

Not necessarily as a watch expression, but being able to see variables by their unminified names in the local scope section when execution is paused from a breakpoint. Also, hovering over a variable in the code while paused.

@download13

This comment has been minimized.

Show comment Hide comment
@download13

download13 May 6, 2012

Contributor

@rvagg: Firebug doesn't support source maps as far as I know. It needs Closure Inspector, which I'm having a hard time with right now. It's also deprecated so I doubt it would support Source Map v3 anyway.

Contributor

download13 commented May 6, 2012

@rvagg: Firebug doesn't support source maps as far as I know. It needs Closure Inspector, which I'm having a hard time with right now. It's also deprecated so I doubt it would support Source Map v3 anyway.

@rvagg

This comment has been minimized.

Show comment Hide comment
@rvagg

rvagg May 6, 2012

Member

I'll pull your code into a separate branch and have a play with it during the week when I can find some time. Good work on implementing a Closure minifier path, the plan is to make the minifier plug gable anyway so I guess if you want SourceMaps on your build then you'd have to run with a minifier that'll do it for you. Hopefully Uglify will get it sooner rather than later tho.
BTW, it'd be good if you could emulate the code style of the rest of the project as much as possible, I know it's not what most are used to. I'll get a code style doc up on the wiki soon as we have had some discussions about basic rules, which are reflected mostly in the 1.0-wip branch.

Member

rvagg commented May 6, 2012

I'll pull your code into a separate branch and have a play with it during the week when I can find some time. Good work on implementing a Closure minifier path, the plan is to make the minifier plug gable anyway so I guess if you want SourceMaps on your build then you'd have to run with a minifier that'll do it for you. Hopefully Uglify will get it sooner rather than later tho.
BTW, it'd be good if you could emulate the code style of the rest of the project as much as possible, I know it's not what most are used to. I'll get a code style doc up on the wiki soon as we have had some discussions about basic rules, which are reflected mostly in the 1.0-wip branch.

@download13

This comment has been minimized.

Show comment Hide comment
@download13

download13 May 6, 2012

Contributor

I wasn't going to bother just for a mockup, but for project code I assumed as much.
How are those class-like things supposed to work anyway? They look like they're supposed to act like classes, but they don't have constructors and just use binds everywhere. Is there some way to map features between prototypal constructors and those? That's the sort of thing that would be good to have in the style doc.

Contributor

download13 commented May 6, 2012

I wasn't going to bother just for a mockup, but for project code I assumed as much.
How are those class-like things supposed to work anyway? They look like they're supposed to act like classes, but they don't have constructors and just use binds everywhere. Is there some way to map features between prototypal constructors and those? That's the sort of thing that would be good to have in the style doc.

@ryanseddon

This comment has been minimized.

Show comment Hide comment
@ryanseddon

ryanseddon May 6, 2012

@download13 yeah wouldn't bother with closure inspector. I got it working but you need ff3.6, firebug 1.5 (thats the only combination that works) and like you said it only works with v1 source maps.

@download13 yeah wouldn't bother with closure inspector. I got it working but you need ff3.6, firebug 1.5 (thats the only combination that works) and like you said it only works with v1 source maps.

@rvagg

This comment has been minimized.

Show comment Hide comment
@rvagg

rvagg May 9, 2012

Member

re resources, @ryanseddon's article is quite a good reference on source maps too: http://www.thecssninja.com/javascript/source-mapping thanks Ryan

Member

rvagg commented May 9, 2012

re resources, @ryanseddon's article is quite a good reference on source maps too: http://www.thecssninja.com/javascript/source-mapping thanks Ryan

@rvagg

This comment has been minimized.

Show comment Hide comment
@rvagg

rvagg Nov 2, 2012

Member

more from @ryanseddon, I wasn't aware that UglifyJS2 was usable yet, great stuff! http://www.thecssninja.com/javascript/multi-level-sourcemaps

Member

rvagg commented Nov 2, 2012

more from @ryanseddon, I wasn't aware that UglifyJS2 was usable yet, great stuff! http://www.thecssninja.com/javascript/multi-level-sourcemaps

@amccollum

This comment has been minimized.

Show comment Hide comment
@amccollum

amccollum Feb 28, 2014

Contributor

Source maps are now being generated in ender@2.0.x. Finally.

Contributor

amccollum commented Feb 28, 2014

Source maps are now being generated in ender@2.0.x. Finally.

@amccollum amccollum closed this Feb 28, 2014

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment