Adds file back to the stream with wrong path (i.e. path changes that happened after gulp-remember are already applied)

[smessmer]

[Using gulp 3.9.0, gulp-cached 1.1.0, gulp-remember 0.3.0, gulp-rename 1.2.2]
[stackoverflow post: http://stackoverflow.com/questions/33072113/gulp-remember-seems-to-output-wrong-path]

We're using gulp as a build tool and gulp-cached together with gulp-remember to allow fast incremental rebuilds. Part of the files under build have to be moved to a different output directory and this has to happen in-stream (i.e. not in gulp.dest()), because we're zipping the results afterwards. We use gulp-rename for this.

However, when the build script is called multiple times (gulp.watch() for incremental rebuilds), then it seems to apply the gulp-rename transformation multiple times. It looks like gulp-remember is not actually outputting the files with the path it saw them last time, but with the path they got after the gulp-rename step.

I've narrowed the problem down to the following script:

var gulp = require('gulp');
var cached = require('gulp-cached');
var remember = require('gulp-remember');
var rename = require('gulp-rename');

function build() {
    gulp.src("src/**")
        .pipe(cached())
        .pipe(remember())
        .pipe(rename(function(path) {path.dirname = "resources/" + path.dirname;}))
        .pipe(gulp.dest('build/'));
}

gulp.task('default', [], function() {
    build();
    build();
});

Running this on a source directory src with just one file "myfile.js" produces the output file:

/build/resources/resources/myfile.js

If the second call to build() is removed, it produces correctly

/build/resources/myfile.js

And if we insert a third call to build(), it produces

/build/resources/resources/resources/myfile.js

I think gulp-remember should output the files with the path they had when they passed through it last time, not the path the files got after being processed further.

[ahaurw01]

Thanks for the ticket, @smessmer. This kind of problem can occur when your pipe to remember() comes before another pipe that mutates the files. When you rename() the files, this mutates the file object and thus affects the files in the remember cache.

You can confirm this by putting some logging in node_modules/gulp-remember/index.js if you are curious. For example,

• util.log('REMEMBERING FILE', file.path); inside the transform function.
• util.log('RECALLING FILE', path, '(' + cache[path].path + ')') inside the flush loop.

You'll see that the recalled path inside parens differs from the original path because it was mutated.

There are a few options for you:

• Split apart your processing a little bit and pipe to gulp.dest('build/resources').
• Simply swap your pipes to remember() and rename(). This is probably easiest.

In a similar example, you wouldn't want to remember() a .coffee file and then send it through the coffeescript transpiler afterward. This would risk re-transpiling an already transpiled file. You would instead transpile it, remember it, then do anything with it that does not require mutating the file, like concatenate with other files, for example.

Let me know if this helps. If so, I'll leave it to you to answer your stack overflow post with the gist of the concept here and the solution that worked for you.

[smessmer]

Hey,

thank you for the fast answer :)
Unfortunately, I can't swap remember() and rename(), because remember() is called inside a module and rename() outside of that. Our build is quite complex, using multiple remember caches in different sub modules. And we can't split the build, because in the end, we zip all files together, which needs us to have all files in one stream.

Our current workaround is a check in the rename module that checks whether the resource directory name has already been prepended. That works, but is a (quite hacky) workaround and would not work for other instances (for example when there are input files in a resource/* subfolder that should be copied to resource/resource/* correspondingly).

Is there a specific reason for gulp-remember behaving this way? I'd find it much more logical if it remembered the files as they actually passed through it. Since the output stream of gulp-remember always passes the same future processing, no matter whether it is the first run or a later run, gulp-remember also should output the files in the same way in each run. Later runs should not output different files than the first run remembered.

[alexander-akait]

@ahaurw01 I have some problem. May be add options base to calculation path.

[ahaurw01]

As mentioned in PR #19, gulp-clone may be a good option here.