Force gulp.src to fix order of files

[fesor]

If i just pass mainBowerFiles result, which ordered based on dependencies for each module, into gulp's stream, then all files will be reordered and it broke app for example for inject or concat.

For example, mainBowerFiles returns:

[
    '/some/path/to/src/lib/angular/angular.js',
    '/some/path/to/src/lib/angular-animate/angular-animate.js',
    '/some/path/to/src/lib/angular-sanitize/angular-sanitize.js',
    '/some/path/to/src/lib/angular-ui-router/release/angular-ui-router.js',
]

after gulp-inject we'll get:

<script src="scripts/angular-animate.js"></script>
<script src="scripts/angular-sanitize.js"></script>
<script src="scripts/angular-ui-router.js"></script>
<script src="scripts/angular.js"></script>

Since ur-router, angular-sanitize and angular-animate depends of angular, application crashes.

Can this be solved?

[sindresorhus]

gulp.src reads the files in order, but plugins are not ordered as it's beneficial for files to not block each other through the pipeline for maximum performance.

You can use a plugin like https://github.com/sirlantis/gulp-order to order the file. But depending on file order is usually an anti-pattern anyways.

[yairEO]

I have just updated gulp the latest version (5 minutes ago) and files order definitely changed.

This simple task just creates random code instead of filename-ordered concatenated code.

gulp.src('./js/*.js')
    .pipe(concat('_all.js'))
    .pipe(gulp.dest('./js/'));

Really.. this happens right after I ran npm install gulp. So I think it's something wrong with the gulp src function which generates the stream.

Update

I had to revert gulp back to (older) version 3.5 until this issue is resolved.
Please open the issue...Thanks!

[heikki]

@yairEO Minimal example which reproduces the issue would be helpful.

[yairEO]

you are absolutely right. I will produce a test now (but this should be automatically tested by a test suit..)

[yairEO]

Here, a test project that shows the bug:

https://github.com/yairEO/gulp-src-test/

[heikki]

I get the expected (a, b, c) order on osx 10.10.1.

ps. You should remove node_modules & put it to .gitignore. Add package.json instead.

[yairEO]

yes you are right i should have put just package.json file.
Anyway, it does happen to me on windows 7 and it never happened before with other gulp version. This bug started recently, and it is totally confirmed by the test case.

[heikki]

I'm seeing the wrong order (c, a, b) on VirtualBox windows 7. It started happening with gulp 3.7.0.

module	diff
gulp	3.6.2 <-> 3.7.0
vinyl-fs	0.1.4 <-> 0.2.1
graceful-fs	2.0.3 <-> 3.0.4

I have no clue why but downgrading graceful-fs to 2.0.3 fixes it.

--edit

Narrowed down to this commit: isaacs/node-graceful-fs@08471b2

[yairEO]

this is a major major bug..someone should re-open this issue.

I absolutely love Gulp and am using it in all my projects and this caused quite a mess

[heikki]

It looks like node-glob doesn't emit match events in sorted order. glob-stream listens those events and older version of graceful-fs has masked this issue somehow.

I guess it's up to @contra to decide if and where this should be handled.

[heikki]

I have a test & fix here: https://github.com/heikki/glob-stream/compare/master...fix-order

• Test would show the problem on windows if mocha didn't use an old version of glob that in turn uses old version of graceful-fs 🙀
• Here's benchmark that shows why the fix is no good (streaming starts a lot later)

I think the original cause is that graceful-fs readdir (or something) is sorted. When monkey patching the global was changed to eval then behaviour changed to normal where graceful-fs wasn't directly used. Normal for windows is different order.

[yocontra]

@heikki That fix will mess up the speed of gulp. One of the great things is that glob matches get emitted immediately, not after the whole fs has been traversed. Has anyone opened an issue on graceful-fs that they broke something here?

[heikki]

I haven't opened any issues yet. Maybe the right place to fix this would be in node-glob? There's no need to sort all the matches at once. Just sort the result of each readdir (or however it works) like it happened when graceful-fs was messing things.

[yocontra]

@heikki Yeah it should go on glob if they introduced a regression without realizing it

[heikki]

Opened issue in node-glob. ^

[heikki]

Works now.

[yocontra]

Closing, an npm update should fix this

[mbcooper]

@contra
I see this as closed. However, i have all the latest and still have files coming back in odd orders ...

[yocontra]

@mbcooper Run npm cache rm && npm update and if it still isn't working run npm ls glob and put the results here

[heikki]

FYI, fix has been reverted in node-glob. isaacs/node-glob#143 (comment)

[heikki]

Hm, what now? Should we leave this as is & add docs or do something in glob-stream if sort-option is specified?

[phated]

What about a config option that globs all before beginning to emit?

[heikki]

Sync mode results are sorted by default so this ugly tweak probably works:

if (opt.sync) {
  globber.found.forEach(function(filename) {
    stream.write({
      cwd: opt.cwd,
      base: basePath,
      path: path.resolve(opt.cwd, filename)
    });
  });
  stream.end();
} else {
  globber.on('error', stream.emit.bind(stream, 'error'));
  globber.on('end', function(/* some args here so can't use bind directly */){
    stream.end();
  });
  globber.on('match', function(filename) {
    stream.write({
      cwd: opt.cwd,
      base: basePath,
      path: path.resolve(opt.cwd, filename)
    });
  });
}

https://github.com/wearefractal/glob-stream/blob/58358ff102e2c5b379ec2f21c938f4b9706385ba/index.js#L29-L39

[yocontra]

I think the cleaner way to do this is to let people specify a sort option, and if given it will tell glob-stream to pipe itself through a stream sorter

[phated]

I like async ordering to belonging in userland. Why build it in core?

[yocontra]

@phated What plugin do you use?

[phated]

@contra I don't use one, but https://github.com/sirlantis/gulp-order doesn't look bad. I think it should be updated to through2, but no other gripes really.

[yocontra]

@phated Doesn't seem to support the features needed to solve this specific ordering problem (alphabetical)

[phated]

@contra it can be adapted. I think userland should have a gulp-reorder that accepts a sort function.

[yocontra]

gulp-community/gulp-order#13

[phated]

@contra https://www.npmjs.com/package/sort-stream2 should work with a custom sort function

[heikki]

Depending on the use case use either gulp-sort (alphabetical etc.) or gulp-order (globs)

[nicholas-johnson]

I'd just interject that non-deterministic ordering causes unnecessary conflicts when managing a deploy via Git. It's cool, yet painful.

[musterknabe]

Why not let the user decide via flag if gulp.src should return reliably ordered results (slower) or random results (faster)?

[mattsawyer77]

totally agree with @musterknabe.

[vniche]

@musterknabe +1