Support for streaming source maps alongside files

[theefer]

Are there any plans to provide better support for source maps with Gulp? It seems that Gulp elegantly removes the need for intermediate files by letting resources flow through transformations directly, but there doesn't seem to be a standard way of handling the source map associated with each transformation. As a result, many Gulp plugins seem to either not support source maps (gulp-requirejs, gulp-concat, etc), or only support them as inlined data-urls (gulp-less, gulp-sass). Those plugins that do support standalone source maps typically pipe them as another file in the stream (gulp-coffee, gulp-uglify), which only works as the last transformation. If there is any more transformation after it (say, uglify after coffee, or concat after uglify), it's unlikely to give the expected result.

Even gulp-uglify doesn't support input source maps (say from a previous transformation), and it's unclear how it could, seeing as there's no explicit link between the source map and the file it maps in the stream.

Is there any will or items on the Gulp roadmap aiming at making it easier for users and plugin developers to serve source maps for their transformed files?

Full disclaimer: I'm the author of Plumber, a build tool based on declarative pipelines with many similarities with Gulp. One of the differences though is that Plumber and all its operations support source maps natively by default, by attaching a source map to each resource and letting each operation provide the source map for the transformation it applied.

The reason I'm inquiring about Gulp plans regarding source maps is that Gulp is already a great project, and I want to understand to what extent my efforts may be redundant, if features similar to Plumber's are also envisioned for Gulp. So please consider this as an attempt for constructive feedback between similar projects!

[terinjokes]

I'm the author of gulp-uglify and while it current has terrible support for source maps, I'm working hard to change that, as well as contributing plugin guidelines and tools to consistently support source maps all the way through a pipeline.

Using my local versions of various gulp plugins, source maps and the original source are preserved from coffeescript through concat and uglify.

My plan is to get these resources out in the new few days, perhaps even today.

[theefer]

Oh great, thanks for the reply, I'd love to see how that looks!

[yocontra]

Thank you @terinjokes for fixing this problem single handedly.

@everyone https://www.gittip.com/terinjokes/

Solving this requires no changes within gulp, but I'll leave this issue open as a reminder to update the plugin guidelines

[maboiteaspam]

Hi, very intersted to know more too,

@terinjokes

Is this the way you currently fixed the problem or did you find any better solution yet ?

if (options.outSourceMap) {
            sourceMap = JSON.parse(mangled.map);
            sourceMap.sources = [ file.relative ];
            map = new Vinyl({
                cwd: file.cwd,
                base: file.base,
                path: file.path + '.map',
                contents: new Buffer(JSON.stringify(sourceMap))
            });
            this.push(map);
        }

[terinjokes]

@maboiteaspam that's an old implementation that I was testing with, please don't accept it as the way going forward, it is not.

Edit: That is an even older way of handling source maps than I previously thought you were referencing, though unfortunately from the current release of gulp-uglify. Under absolutely no circumstances should you implement that as the correct way either.

[maboiteaspam]

hmmm yes you totally right this is ain t good at all, unless this process is the very last one.

But it seems to me that by design gulp fails to handle that case properly, as in the input i put an src such **.js, and that meanwhile we have to output both js and map file, it become incompatible. map files can not support eventual process we could put after the minify proess.

Thus, regarding gulp, my understanding is that we should have two process, one to do the minify, with only JS content, another one to do the map file only => total waste of performance.

That s so sad, the most practical way i can find to fix it is to patch any post process in order to make them check the file type before they are processed and ignore those which are incompatible.

IRL, that s fine for me, but talking about software design it is dirty : /

Hope Contra can find out some good way to handle that.

[maboiteaspam]

another idea,

gulp.task('scripts', function() {
  // Minify and copy all JavaScript (except vendor scripts)
  return gulp.src(paths.scripts)
    .pipe(uglify())
    .pipe(filter("*.map",gulp.dest('build/js'))
    .pipe(concat('all.min.js'))
    .pipe(gulp.dest('build/js'));
});

?

[terinjokes]

It is done very nicely, one just needs to be patient.
On Feb 25, 2014 5:01 AM, "Clément" notifications@github.com wrote:

another idea,

gulp.task('scripts', function() {
// Minify and copy all JavaScript (except vendor scripts)
return gulp.src(paths.scripts)
.pipe(uglify())
.pipe(filter("*.map",gulp.dest('build/js'))
.pipe(concat('all.min.js'))
.pipe(gulp.dest('build/js'));
});

?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/303#issuecomment-36004739
.

[maboiteaspam]

Hi,

Well after some sleep on it, I have a big doubt on that solution, let s say that for 1 js file you have 1000 map (just for example), i wonder how gulp.task method could detect correctly the end of both stream.

Indeed, the main stream which is not filtered, and containing only one remaining item, would end before the filtered one :/

Obviously we could play with strem.on("end') + async callback, but it ain t going to be that nice for end user source code.

Looking forward for ideas !

Bye

Actually found that one : https://github.com/sindresorhus/gulp-filter

[theefer]

@maboiteaspam You normally have one map per file. Either way, as far as I understand, @terinjokes plans to record the sourcemaps inside (at the end of) the file itself, as a data-URL, rather than emitting it as a separate resource.

I'm not convinced by this approach but I'd first like to see how it looks like and works, so I'll just wait and see :-)

[terinjokes]

I prefer source maps to be inlined, but as it looks right now you'll have
an option to go to a separate file
On Feb 27, 2014 3:57 AM, "Sébastien Cevey" notifications@github.com wrote:

@maboiteaspam https://github.com/maboiteaspam You normally have one map
per file. Either way, as far as I understand, @terinjokeshttps://github.com/terinjokesplans to record the sourcemaps inside (at the end of) the file itself, as a
data-URL, rather than emitting it as a separate resource.

I'm not convinced by this approach but I'd first like to see how it looks
like and works, so I'll just wait and see :-)

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/303#issuecomment-36234771
.

[tkellen]

@jmeas @mzgoddard is the general-use sourcemap work you two are doing useful here?

[maboiteaspam]

@terinjokes, yep, and it will let me do more than just manage map files, houra :D
@theefer, let s try to do softwares that can do a little more than just what is expected, We nerver know ;)

[terinjokes]

@maboiteaspam yes, yes. with the current API you'll have the option to go to a separate file, or to inline, or do whatever else you want to do with them. :)

[floridoo]

Another possible approach:

How about adding an optional property sourceMap to the vinyl file object? This way plugins can pass along the source map with the file, without changing its content. At the end of the pipe the source maps can be written by a plugin.

In the gulpfile that could look like this:

var gulp = require('gulp');
var concat = require('gulp-concat');
var uglify = require('gulp-uglify');
var sourcemaps = require('gulp-sourcemaps');

gulp.task('javascript', function() {
    gulp.src('src/**/*.js')
    .pipe(concat('all.js'))
    .pipe(uglify({outSourceMap: true}))
    .pipe(sourcemaps())
    .pipe(gulp.dest('dist'));
});

I experimented with this idea by writing a plugin gulp-source-maps, and forking gulp-uglify and gulp-concat to use the described approach.

If you want to try them out in a sample project, just clone this repository and npm install.
The order of uglify and concat doesn't matter, both plugins combine their source map with the one they get from higher in the pipeline.

In my opinion this would be a way that is clean for the user and API, without breaking current plugins. Plugins currently supporting source maps could be adapted quite easily, as you can see in the gulp-uglify example.

What do you think?

[terinjokes]

@floridoo you've pretty much gotten it, but there's more stuff to do within the plugin.

What I've been doing is iterating on developing plugins that support source maps and abstracting out the common workflows to make working with source maps easy for plugin authors. In your modifications, there's still a lot of boilerplate that needs to happen.

Sorry everyone for being busy with work this past week, going to try to finish it up and get it out today.

[floridoo]

@terinjokes Sounds good. Looking forward to see your plugins, guides, etc.!

BTW: I hacked this together before seeing this thread. Just when I wanted to open an issue suggesting better source map handling I found this thread. Glad to see you already had the same idea and that this is being improved soon.

Let me know if I can help with anything.

[theefer]

@floridoo I like the idea of attaching the source map to the vinyl object. In fact, this is exactly how I've done it in Plumber, as a property of the Resource (equivalent of vinyl object), and it's been working really well.

@terinjokes In case you're interested, I wrote a helper library called mercator for dealing with source maps, essentially a wrapper for the Mozilla source-map package with helpers to deal with common operations and combinations (concatenate, identity source map, apply onto existing source map). If each individual operation produces a source map of the transform it's responsible for, it's then easy to combine the source maps sequentially to produce a start-to-finish map.

Would love it if this stuff could be supported natively in Gulp and Vinyl too!

[jamesplease]

@tkellen, we've discussed supporting streaming but haven't even scratched the surface on the exact implementation.

[tkellen]

@jmeas I was referring to the general use libs ya'll are working on, not the grunt specific stuff.

On Tuesday, March 4, 2014 at 7:39 PM, Jmeas Smith wrote:

@tkellen (https://github.com/tkellen), we've discussed supporting streaming but haven't even scratched the surface on the exact implementation.

—
Reply to this email directly or view it on GitHub (#303 (comment)).

[terinjokes]

@theefer i've been using thlorenz's convert-source-maps, inline-source-map and combine-source-map

[theefer]

@terinjokes Ah yes, thanks I saw those too and they seemed quite handy, but I couldn't find the equivalent of Mozilla source-map's applySourceMap, to apply a source map onto another, which ends up quite useful when pipelining operations. Could probably be spun as a simple standalone lib too I guess.

[yocontra]

Moving this to #356

[terinjokes]

@floridoo can you email me (address on my profile)? thanks.