sourcemaps out-of-line #322

Closed
pmuellr opened this Issue Mar 12, 2013 · 12 comments

Projects

None yet

4 participants

@pmuellr
pmuellr commented Mar 12, 2013

The sourcemaps via data-url is a neat trick, which allows you to deliver a debuggable wad to the browser without requiring subsequent network accesses to get the map.

Seems a little non-standard.

Also, it increases the amount of data that needs to be sent by 33%, since it's base64'd.

But a bigger problem is that when some editors try to open files with these in them, they die - TextMate and Sublime Text 2 anyway. Click on the file, now you gots to Force Quit your editor. Locks up with 100% CPU for longer than I was willing to wait for it to "finish" (30 sec). :-( My test is with a sourcemap wad that's 242KB (base64'd).

I'd like to see an option to split the sourcemap data into a separate file, and then also let me indicate what the URL to that file will be. The default file name would be [outputFile].map, and the default URL would be that string as well.

Please do keep using the sourcesContent field to ship the source IN the sourcemap, because I'd like to limit the # of URLs I have to keep in sync between server and dev to 1 - the sourcemap itself, and don't really care how big the sourcemap file is.

@substack
Owner

This is planned! The stuff @thlorenz is working on already has the hooks to build a separate source map file, it's just that --debug to build self-contained blobs makes for a really convenient debugging workflow to start with.

@thlorenz
Collaborator

This is indeed planned I discussed those with @substack previously.
However, we still need to come up with a good API for this.

I'm thinking something along the lines of (showing defaults):

{
  debug: true,
  sourceMaps: {
    inlinineSourcesContent: true, // if false no sources will be added, but will be loaded from soureRootURL/sourceFile by the browser when dev tools are opened
    sourceRootURL: 'file://',     // this can be changed to load sources from a specific server that may be remote
    sourceFileRoot: '/' ,        // sourceFile paths will be determined relative to this root
    sourceMapFile: null          // if supplied, source maps won't be inlined, but '//@ sourceMappingURL=/path/to/file.js.map' is added instead
  }
}

Note: when sourceMapFile is given:

  • all other sourceMaps options will be ignored
  • the source map file will be generated by browserify and stored at the given path

Concerning the command line this may start to look a little ugly, what I would suggest would be:

--sm-inline-sources-content, --sm-source-root-url, etc.

Possibly we should only make these advanced features available via the API.

@pmuellr
pmuellr commented Mar 12, 2013

For anyone interested in splitting the existing wad out to a separate file (for 2.5.1), I wrote a little thingee: https://gist.github.com/pmuellr/5143384

@pmuellr
pmuellr commented Mar 12, 2013

My default use case is:

  • always keep the source map data in a separate .map file
  • that .map file will be available alongside the corresponding .js file
  • that .map file should have all the source inlined, so that I only have to deal with one file and not all the little source files as separate URLs
  • I'll always use the --debug option because I can deploy in the field with the sourcemap line in the cat'd file, and it shouldn't hurt anyone - and it may even help someone!

This lets me get away with a single build, instead of "debug" and "release" builds I had to do back in the ol' days of eval()ing //sourceURL gunk. Keeps the map in a single file, easy to either move around, or delete, as needed. WIll have overhead in the debugger as ALL the source will be downloaded for EVERYTHING, instead of just what you need, but - until someone finds a problem - performance does not seem to be a problem with a 200KB source map for me right now. And if it's a problem, then you can split your wad of modules into multiple wads, presumably.

I'd like a way to have the default happen with just a single switch, no names, or anything.

I'd also like a way to "name" some of the modules - we shouldn't be using absolute paths to the files, as these are too large given the way the browser show them in nav lists (which is broken, so we have to work-around by shortening, for now). My own modules come out fine, especially if I generate names which are relative to my cwd when I do a build. What'd I'd like to see changed are both the built-in node modules (path, url, etc) as well as anything else in a node_modules directory - perhaps removing the node_modules path segment altogether - or replacing with $ or something - is an easy enough cheat. I made an attempt at massaging some of the built-ins in the gist I reference above - s'ok, could use some work.

@thlorenz
Collaborator

Lots of interesting points here:

that .map file will be available alongside the corresponding .js file

We don't always know where your bundle is. This needs to work even if you don't give us an outfile (i.e. you may want to serve the bundle from memory). So we have no notion of where 'alongside' would be.
IMO a better solution is to specify the map location -- if I'm not mistaken, if you just give a filename (without a path), chrome will try to find it right next to where the generated file came from.

However we'd still need to know where to store the generated map file, so I'd suggest a change to take two properties concerning the map file: path (where browserify will write it) url which will result in //@ sourceMappingURL=url.

that .map file should have all the source inlined

I agree that this is a sensible default and browserify should generate the map file that way.

I'd like a way to have the default happen with just a single switch, no names, or anything

As I outlined this is not always possible, so you'll need to give browserify the necessary info here.

we shouldn't be using absolute paths to the files

If you pass a sensible sourceFileRoot, that won't be a problem, i.e. make it the root of your project to see paths like public/js/foo.js. I tried this already (was using process.cwd()), so it works as long as sources are inlined. So once this API is fleshed out you should get better results.

perhaps removing the node_modules path segment altogether

I believe that might be more confusing than helpful and if all paths are relative to project root, seeing things like node_modules/backbone/backbone.js seems ok to me.

Updated API to include more details regarding map files:

{
  debug: true,
  sourceMaps: {
    inlinineSourcesContent: true, // if false no sources will be added, but will be loaded from soureRootURL/sourceFile by the browser when dev tools are opened
    sourceRootURL: 'file://',     // this can be changed to load sources from a specific server that may be remote
    sourceFileRoot: '/' ,        // sourceFile paths will be determined relative to this root
    // if supplied, source maps won't be inlined, but '//@  sourceMappingURL=url' is added instead and file generated at given path    
    sourceMapFile: { path: null, url: null } 
  }
}
@substack
Owner

I think the best way to address this is with a completely separate tool that can read browserify bundle files, moving the inline source maps into a separate file. If we can keep browserify core minimal by only addressing the inline use-case I think this will be a better approach to keep scope creep in browserify core under control.

@substack substack closed this Mar 17, 2013
@thlorenz
Collaborator

@pmuellr Take a look at mold-source-map.

It does exactly what you need.
Try the browserify-transform-sources example and open index.html to see the *.js files all relative to js inside dev tools.

More functionality is coming soon ( including the ability to replace the source map with a pointer to a map file and specify where that map file will be written to).

@thlorenz
Collaborator

@pmuellr The promised extra functionality is in.

Please review this example which demonstrates how to achieve all changes that you desired:

  • external source map file
  • adjust sourceRootUrl and sources paths to improve file display in the browser sources tab

Any other modifications you'd want to make to the source map are now possible as well.
I'll expose more helper functions, but the underlying sourcemap (which exposes the convert-source-map api) property can be modified via setProperty - see here.

I need to improve the documentation, but for now, just know that this sourcemap is accessible in the transform via sourcemap.sourcemap.

@pmuellr
pmuellr commented Mar 19, 2013

cool. I'll take a look at it when I get a chance to breathe.

issue #326 is preventing me from using the coffeeify transformer, which means I have some other juggling to do. eg, compiling CS before running browserify. I'll be interested to see if I can get the CS compiler to generate sourcemaps which browserify itself can consume! That should, in theory, no? I believe it generates them as standalone files beside the generated .js, with an appropriate sourcemap comment in the generated .js.

@thlorenz
Collaborator

You could very well do that or you could just insert a transformer in front of coffeeify that adjusts your requires to include the .coffee.

Another option is to wrap coffeeify inside another transformer that calls thru to the coffeeify transformer with adapted filename. I'm planning on implementing a wrapper that will cache compiled files to prevent recompiling files that didn't change.
Once I have that ready I'll shoot you a link and you can use some of its ideas to create your wrapper.

@pmuellr
pmuellr commented Mar 19, 2013

I'll be interested to see your work there. Presumably a "require rewriter" would need to have some smarts to figure out whether to suffix-ize a required coffee module - the path of the module being rewritten at a minimum. Then it has to do the resolve itself; handling simple "it's a relative module" is easy, but looking back up the node package search path seems like it would be dying for a helper of some kind.

Seems a bit round-about. Node already has some support to augment module resolution via require.extensions. Why not have a way of allowing a transformer to opt-in to your module resolution in a similar fashion. As simple as adding a property to an "extensions" property of the transform function itself. eg, at the end of coffeeify, add:

module.exports.extensions = ".coffee .litcoffee .coffee.md".split(" ")

Then YOU do all the work :-)

@lucastschmidt

Hey guys, can anyone give me a status on this? Are you guys being able to work with coffeeify sourceMaps?

Thanks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment