Support for streaming source maps alongside files #303

Closed
theefer opened this Issue Feb 23, 2014 · 24 comments

Comments

Projects
None yet
7 participants
@theefer

theefer commented Feb 23, 2014

Are there any plans to provide better support for source maps with Gulp? It seems that Gulp elegantly removes the need for intermediate files by letting resources flow through transformations directly, but there doesn't seem to be a standard way of handling the source map associated with each transformation. As a result, many Gulp plugins seem to either not support source maps (gulp-requirejs, gulp-concat, etc), or only support them as inlined data-urls (gulp-less, gulp-sass). Those plugins that do support standalone source maps typically pipe them as another file in the stream (gulp-coffee, gulp-uglify), which only works as the last transformation. If there is any more transformation after it (say, uglify after coffee, or concat after uglify), it's unlikely to give the expected result.

Even gulp-uglify doesn't support input source maps (say from a previous transformation), and it's unclear how it could, seeing as there's no explicit link between the source map and the file it maps in the stream.

Is there any will or items on the Gulp roadmap aiming at making it easier for users and plugin developers to serve source maps for their transformed files?

Full disclaimer: I'm the author of Plumber, a build tool based on declarative pipelines with many similarities with Gulp. One of the differences though is that Plumber and all its operations support source maps natively by default, by attaching a source map to each resource and letting each operation provide the source map for the transformation it applied.

The reason I'm inquiring about Gulp plans regarding source maps is that Gulp is already a great project, and I want to understand to what extent my efforts may be redundant, if features similar to Plumber's are also envisioned for Gulp. So please consider this as an attempt for constructive feedback between similar projects!

@terinjokes

This comment has been minimized.

Show comment
Hide comment
@terinjokes

terinjokes Feb 23, 2014

Contributor

I'm the author of gulp-uglify and while it current has terrible support for source maps, I'm working hard to change that, as well as contributing plugin guidelines and tools to consistently support source maps all the way through a pipeline.

Using my local versions of various gulp plugins, source maps and the original source are preserved from coffeescript through concat and uglify.

My plan is to get these resources out in the new few days, perhaps even today.

Contributor

terinjokes commented Feb 23, 2014

I'm the author of gulp-uglify and while it current has terrible support for source maps, I'm working hard to change that, as well as contributing plugin guidelines and tools to consistently support source maps all the way through a pipeline.

Using my local versions of various gulp plugins, source maps and the original source are preserved from coffeescript through concat and uglify.

My plan is to get these resources out in the new few days, perhaps even today.

@theefer

This comment has been minimized.

Show comment
Hide comment
@theefer

theefer Feb 23, 2014

Oh great, thanks for the reply, I'd love to see how that looks!

theefer commented Feb 23, 2014

Oh great, thanks for the reply, I'd love to see how that looks!

@contra

This comment has been minimized.

Show comment
Hide comment
@contra

contra Feb 24, 2014

Member

Thank you @terinjokes for fixing this problem single handedly.

@everyone https://www.gittip.com/terinjokes/

Solving this requires no changes within gulp, but I'll leave this issue open as a reminder to update the plugin guidelines

Member

contra commented Feb 24, 2014

Thank you @terinjokes for fixing this problem single handedly.

@everyone https://www.gittip.com/terinjokes/

Solving this requires no changes within gulp, but I'll leave this issue open as a reminder to update the plugin guidelines

@maboiteaspam

This comment has been minimized.

Show comment
Hide comment
@maboiteaspam

maboiteaspam Feb 25, 2014

Hi, very intersted to know more too,

@terinjokes

Is this the way you currently fixed the problem or did you find any better solution yet ?

if (options.outSourceMap) {
            sourceMap = JSON.parse(mangled.map);
            sourceMap.sources = [ file.relative ];
            map = new Vinyl({
                cwd: file.cwd,
                base: file.base,
                path: file.path + '.map',
                contents: new Buffer(JSON.stringify(sourceMap))
            });
            this.push(map);
        }

Hi, very intersted to know more too,

@terinjokes

Is this the way you currently fixed the problem or did you find any better solution yet ?

if (options.outSourceMap) {
            sourceMap = JSON.parse(mangled.map);
            sourceMap.sources = [ file.relative ];
            map = new Vinyl({
                cwd: file.cwd,
                base: file.base,
                path: file.path + '.map',
                contents: new Buffer(JSON.stringify(sourceMap))
            });
            this.push(map);
        }
@terinjokes

This comment has been minimized.

Show comment
Hide comment
@terinjokes

terinjokes Feb 25, 2014

Contributor

@maboiteaspam that's an old implementation that I was testing with, please don't accept it as the way going forward, it is not.

Edit: That is an even older way of handling source maps than I previously thought you were referencing, though unfortunately from the current release of gulp-uglify. Under absolutely no circumstances should you implement that as the correct way either.

Contributor

terinjokes commented Feb 25, 2014

@maboiteaspam that's an old implementation that I was testing with, please don't accept it as the way going forward, it is not.

Edit: That is an even older way of handling source maps than I previously thought you were referencing, though unfortunately from the current release of gulp-uglify. Under absolutely no circumstances should you implement that as the correct way either.

@maboiteaspam

This comment has been minimized.

Show comment
Hide comment
@maboiteaspam

maboiteaspam Feb 25, 2014

hmmm yes you totally right this is ain t good at all, unless this process is the very last one.

But it seems to me that by design gulp fails to handle that case properly, as in the input i put an src such **.js, and that meanwhile we have to output both js and map file, it become incompatible. map files can not support eventual process we could put after the minify proess.

Thus, regarding gulp, my understanding is that we should have two process, one to do the minify, with only JS content, another one to do the map file only => total waste of performance.

That s so sad, the most practical way i can find to fix it is to patch any post process in order to make them check the file type before they are processed and ignore those which are incompatible.

IRL, that s fine for me, but talking about software design it is dirty : /

Hope Contra can find out some good way to handle that.

hmmm yes you totally right this is ain t good at all, unless this process is the very last one.

But it seems to me that by design gulp fails to handle that case properly, as in the input i put an src such **.js, and that meanwhile we have to output both js and map file, it become incompatible. map files can not support eventual process we could put after the minify proess.

Thus, regarding gulp, my understanding is that we should have two process, one to do the minify, with only JS content, another one to do the map file only => total waste of performance.

That s so sad, the most practical way i can find to fix it is to patch any post process in order to make them check the file type before they are processed and ignore those which are incompatible.

IRL, that s fine for me, but talking about software design it is dirty : /

Hope Contra can find out some good way to handle that.

@maboiteaspam

This comment has been minimized.

Show comment
Hide comment
@maboiteaspam

maboiteaspam Feb 25, 2014

another idea,

gulp.task('scripts', function() {
  // Minify and copy all JavaScript (except vendor scripts)
  return gulp.src(paths.scripts)
    .pipe(uglify())
    .pipe(filter("*.map",gulp.dest('build/js'))
    .pipe(concat('all.min.js'))
    .pipe(gulp.dest('build/js'));
});

?

another idea,

gulp.task('scripts', function() {
  // Minify and copy all JavaScript (except vendor scripts)
  return gulp.src(paths.scripts)
    .pipe(uglify())
    .pipe(filter("*.map",gulp.dest('build/js'))
    .pipe(concat('all.min.js'))
    .pipe(gulp.dest('build/js'));
});

?

@terinjokes

This comment has been minimized.

Show comment
Hide comment
@terinjokes

terinjokes Feb 25, 2014

Contributor

It is done very nicely, one just needs to be patient.
On Feb 25, 2014 5:01 AM, "Clément" notifications@github.com wrote:

another idea,

gulp.task('scripts', function() {
// Minify and copy all JavaScript (except vendor scripts)
return gulp.src(paths.scripts)
.pipe(uglify())
.pipe(filter("*.map",gulp.dest('build/js'))
.pipe(concat('all.min.js'))
.pipe(gulp.dest('build/js'));
});

?


Reply to this email directly or view it on GitHubhttps://github.com/gulpjs/gulp/issues/303#issuecomment-36004739
.

Contributor

terinjokes commented Feb 25, 2014

It is done very nicely, one just needs to be patient.
On Feb 25, 2014 5:01 AM, "Clément" notifications@github.com wrote:

another idea,

gulp.task('scripts', function() {
// Minify and copy all JavaScript (except vendor scripts)
return gulp.src(paths.scripts)
.pipe(uglify())
.pipe(filter("*.map",gulp.dest('build/js'))
.pipe(concat('all.min.js'))
.pipe(gulp.dest('build/js'));
});

?


Reply to this email directly or view it on GitHubhttps://github.com/gulpjs/gulp/issues/303#issuecomment-36004739
.

@maboiteaspam

This comment has been minimized.

Show comment
Hide comment
@maboiteaspam

maboiteaspam Feb 26, 2014

Hi,

Well after some sleep on it, I have a big doubt on that solution, let s say that for 1 js file you have 1000 map (just for example), i wonder how gulp.task method could detect correctly the end of both stream.

Indeed, the main stream which is not filtered, and containing only one remaining item, would end before the filtered one :/

Obviously we could play with strem.on("end') + async callback, but it ain t going to be that nice for end user source code.

Looking forward for ideas !

Bye

Actually found that one : https://github.com/sindresorhus/gulp-filter

Hi,

Well after some sleep on it, I have a big doubt on that solution, let s say that for 1 js file you have 1000 map (just for example), i wonder how gulp.task method could detect correctly the end of both stream.

Indeed, the main stream which is not filtered, and containing only one remaining item, would end before the filtered one :/

Obviously we could play with strem.on("end') + async callback, but it ain t going to be that nice for end user source code.

Looking forward for ideas !

Bye

Actually found that one : https://github.com/sindresorhus/gulp-filter

@theefer

This comment has been minimized.

Show comment
Hide comment
@theefer

theefer Feb 27, 2014

@maboiteaspam You normally have one map per file. Either way, as far as I understand, @terinjokes plans to record the sourcemaps inside (at the end of) the file itself, as a data-URL, rather than emitting it as a separate resource.

I'm not convinced by this approach but I'd first like to see how it looks like and works, so I'll just wait and see :-)

theefer commented Feb 27, 2014

@maboiteaspam You normally have one map per file. Either way, as far as I understand, @terinjokes plans to record the sourcemaps inside (at the end of) the file itself, as a data-URL, rather than emitting it as a separate resource.

I'm not convinced by this approach but I'd first like to see how it looks like and works, so I'll just wait and see :-)

@terinjokes

This comment has been minimized.

Show comment
Hide comment
@terinjokes

terinjokes Feb 27, 2014

Contributor

I prefer source maps to be inlined, but as it looks right now you'll have
an option to go to a separate file
On Feb 27, 2014 3:57 AM, "Sébastien Cevey" notifications@github.com wrote:

@maboiteaspam https://github.com/maboiteaspam You normally have one map
per file. Either way, as far as I understand, @terinjokeshttps://github.com/terinjokesplans to record the sourcemaps inside (at the end of) the file itself, as a
data-URL, rather than emitting it as a separate resource.

I'm not convinced by this approach but I'd first like to see how it looks
like and works, so I'll just wait and see :-)


Reply to this email directly or view it on GitHubhttps://github.com/gulpjs/gulp/issues/303#issuecomment-36234771
.

Contributor

terinjokes commented Feb 27, 2014

I prefer source maps to be inlined, but as it looks right now you'll have
an option to go to a separate file
On Feb 27, 2014 3:57 AM, "Sébastien Cevey" notifications@github.com wrote:

@maboiteaspam https://github.com/maboiteaspam You normally have one map
per file. Either way, as far as I understand, @terinjokeshttps://github.com/terinjokesplans to record the sourcemaps inside (at the end of) the file itself, as a
data-URL, rather than emitting it as a separate resource.

I'm not convinced by this approach but I'd first like to see how it looks
like and works, so I'll just wait and see :-)


Reply to this email directly or view it on GitHubhttps://github.com/gulpjs/gulp/issues/303#issuecomment-36234771
.

@tkellen

This comment has been minimized.

Show comment
Hide comment
@tkellen

tkellen Feb 28, 2014

Member

@jmeas @mzgoddard is the general-use sourcemap work you two are doing useful here?

Member

tkellen commented Feb 28, 2014

@jmeas @mzgoddard is the general-use sourcemap work you two are doing useful here?

@maboiteaspam

This comment has been minimized.

Show comment
Hide comment
@maboiteaspam

maboiteaspam Mar 2, 2014

@terinjokes, yep, and it will let me do more than just manage map files, houra :D
@theefer, let s try to do softwares that can do a little more than just what is expected, We nerver know ;)

@terinjokes, yep, and it will let me do more than just manage map files, houra :D
@theefer, let s try to do softwares that can do a little more than just what is expected, We nerver know ;)

@terinjokes

This comment has been minimized.

Show comment
Hide comment
@terinjokes

terinjokes Mar 2, 2014

Contributor

@maboiteaspam yes, yes. with the current API you'll have the option to go to a separate file, or to inline, or do whatever else you want to do with them. :)

Contributor

terinjokes commented Mar 2, 2014

@maboiteaspam yes, yes. with the current API you'll have the option to go to a separate file, or to inline, or do whatever else you want to do with them. :)

@floridoo

This comment has been minimized.

Show comment
Hide comment
@floridoo

floridoo Mar 2, 2014

Another possible approach:

How about adding an optional property sourceMap to the vinyl file object? This way plugins can pass along the source map with the file, without changing its content. At the end of the pipe the source maps can be written by a plugin.

In the gulpfile that could look like this:

var gulp = require('gulp');
var concat = require('gulp-concat');
var uglify = require('gulp-uglify');
var sourcemaps = require('gulp-sourcemaps');

gulp.task('javascript', function() {
    gulp.src('src/**/*.js')
    .pipe(concat('all.js'))
    .pipe(uglify({outSourceMap: true}))
    .pipe(sourcemaps())
    .pipe(gulp.dest('dist'));
});

I experimented with this idea by writing a plugin gulp-source-maps, and forking gulp-uglify and gulp-concat to use the described approach.

If you want to try them out in a sample project, just clone this repository and npm install.
The order of uglify and concat doesn't matter, both plugins combine their source map with the one they get from higher in the pipeline.

In my opinion this would be a way that is clean for the user and API, without breaking current plugins. Plugins currently supporting source maps could be adapted quite easily, as you can see in the gulp-uglify example.

What do you think?

floridoo commented Mar 2, 2014

Another possible approach:

How about adding an optional property sourceMap to the vinyl file object? This way plugins can pass along the source map with the file, without changing its content. At the end of the pipe the source maps can be written by a plugin.

In the gulpfile that could look like this:

var gulp = require('gulp');
var concat = require('gulp-concat');
var uglify = require('gulp-uglify');
var sourcemaps = require('gulp-sourcemaps');

gulp.task('javascript', function() {
    gulp.src('src/**/*.js')
    .pipe(concat('all.js'))
    .pipe(uglify({outSourceMap: true}))
    .pipe(sourcemaps())
    .pipe(gulp.dest('dist'));
});

I experimented with this idea by writing a plugin gulp-source-maps, and forking gulp-uglify and gulp-concat to use the described approach.

If you want to try them out in a sample project, just clone this repository and npm install.
The order of uglify and concat doesn't matter, both plugins combine their source map with the one they get from higher in the pipeline.

In my opinion this would be a way that is clean for the user and API, without breaking current plugins. Plugins currently supporting source maps could be adapted quite easily, as you can see in the gulp-uglify example.

What do you think?

@terinjokes

This comment has been minimized.

Show comment
Hide comment
@terinjokes

terinjokes Mar 2, 2014

Contributor

@floridoo you've pretty much gotten it, but there's more stuff to do within the plugin.

What I've been doing is iterating on developing plugins that support source maps and abstracting out the common workflows to make working with source maps easy for plugin authors. In your modifications, there's still a lot of boilerplate that needs to happen.

Sorry everyone for being busy with work this past week, going to try to finish it up and get it out today.

Contributor

terinjokes commented Mar 2, 2014

@floridoo you've pretty much gotten it, but there's more stuff to do within the plugin.

What I've been doing is iterating on developing plugins that support source maps and abstracting out the common workflows to make working with source maps easy for plugin authors. In your modifications, there's still a lot of boilerplate that needs to happen.

Sorry everyone for being busy with work this past week, going to try to finish it up and get it out today.

@floridoo

This comment has been minimized.

Show comment
Hide comment
@floridoo

floridoo Mar 2, 2014

@terinjokes Sounds good. Looking forward to see your plugins, guides, etc.!

BTW: I hacked this together before seeing this thread. Just when I wanted to open an issue suggesting better source map handling I found this thread. Glad to see you already had the same idea and that this is being improved soon.

Let me know if I can help with anything.

floridoo commented Mar 2, 2014

@terinjokes Sounds good. Looking forward to see your plugins, guides, etc.!

BTW: I hacked this together before seeing this thread. Just when I wanted to open an issue suggesting better source map handling I found this thread. Glad to see you already had the same idea and that this is being improved soon.

Let me know if I can help with anything.

@theefer

This comment has been minimized.

Show comment
Hide comment
@theefer

theefer Mar 5, 2014

@floridoo I like the idea of attaching the source map to the vinyl object. In fact, this is exactly how I've done it in Plumber, as a property of the Resource (equivalent of vinyl object), and it's been working really well.

@terinjokes In case you're interested, I wrote a helper library called mercator for dealing with source maps, essentially a wrapper for the Mozilla source-map package with helpers to deal with common operations and combinations (concatenate, identity source map, apply onto existing source map). If each individual operation produces a source map of the transform it's responsible for, it's then easy to combine the source maps sequentially to produce a start-to-finish map.

Would love it if this stuff could be supported natively in Gulp and Vinyl too!

theefer commented Mar 5, 2014

@floridoo I like the idea of attaching the source map to the vinyl object. In fact, this is exactly how I've done it in Plumber, as a property of the Resource (equivalent of vinyl object), and it's been working really well.

@terinjokes In case you're interested, I wrote a helper library called mercator for dealing with source maps, essentially a wrapper for the Mozilla source-map package with helpers to deal with common operations and combinations (concatenate, identity source map, apply onto existing source map). If each individual operation produces a source map of the transform it's responsible for, it's then easy to combine the source maps sequentially to produce a start-to-finish map.

Would love it if this stuff could be supported natively in Gulp and Vinyl too!

@jamesplease

This comment has been minimized.

Show comment
Hide comment
@jamesplease

jamesplease Mar 5, 2014

@tkellen, we've discussed supporting streaming but haven't even scratched the surface on the exact implementation.

@tkellen, we've discussed supporting streaming but haven't even scratched the surface on the exact implementation.

@tkellen

This comment has been minimized.

Show comment
Hide comment
@tkellen

tkellen Mar 5, 2014

Member

@jmeas I was referring to the general use libs ya'll are working on, not the grunt specific stuff.

On Tuesday, March 4, 2014 at 7:39 PM, Jmeas Smith wrote:

@tkellen (https://github.com/tkellen), we've discussed supporting streaming but haven't even scratched the surface on the exact implementation.


Reply to this email directly or view it on GitHub (#303 (comment)).

Member

tkellen commented Mar 5, 2014

@jmeas I was referring to the general use libs ya'll are working on, not the grunt specific stuff.

On Tuesday, March 4, 2014 at 7:39 PM, Jmeas Smith wrote:

@tkellen (https://github.com/tkellen), we've discussed supporting streaming but haven't even scratched the surface on the exact implementation.


Reply to this email directly or view it on GitHub (#303 (comment)).

@terinjokes

This comment has been minimized.

Show comment
Hide comment
@terinjokes

terinjokes Mar 5, 2014

Contributor

@theefer i've been using thlorenz's convert-source-maps, inline-source-map and combine-source-map

Contributor

terinjokes commented Mar 5, 2014

@theefer i've been using thlorenz's convert-source-maps, inline-source-map and combine-source-map

@theefer

This comment has been minimized.

Show comment
Hide comment
@theefer

theefer Mar 5, 2014

@terinjokes Ah yes, thanks I saw those too and they seemed quite handy, but I couldn't find the equivalent of Mozilla source-map's applySourceMap, to apply a source map onto another, which ends up quite useful when pipelining operations. Could probably be spun as a simple standalone lib too I guess.

theefer commented Mar 5, 2014

@terinjokes Ah yes, thanks I saw those too and they seemed quite handy, but I couldn't find the equivalent of Mozilla source-map's applySourceMap, to apply a source map onto another, which ends up quite useful when pipelining operations. Could probably be spun as a simple standalone lib too I guess.

@contra

This comment has been minimized.

Show comment
Hide comment
@contra

contra Mar 20, 2014

Member

Moving this to #356

Member

contra commented Mar 20, 2014

Moving this to #356

@contra contra closed this Mar 20, 2014

@terinjokes

This comment has been minimized.

Show comment
Hide comment
@terinjokes

terinjokes Apr 8, 2014

Contributor

@floridoo can you email me (address on my profile)? thanks.

Contributor

terinjokes commented Apr 8, 2014

@floridoo can you email me (address on my profile)? thanks.

@simshanith simshanith referenced this issue in jorrit/gulp-requirejs Dec 23, 2014

Closed

Support for Source Maps #6

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment