Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Is gulp.src too strict about input globs? (gulp 4) #1298

Closed
vbud opened this issue Oct 1, 2015 · 44 comments
Labels

Comments

@vbud
Copy link

@vbud vbud commented Oct 1, 2015

In #712, a decision was made to allow an empty array to be passed into gulp.src based on the case made by @wesleytodd. That decision was reverted recently in gulp 4 in is-valid-glob, which is used by vinyl-fs.

Elsewhere, another vinyl-fs dependency, glob-stream, is now also enforcing that individual file references in glob arrays or in singular globs must exist, otherwise a 'File not found with singular glob' error is thrown.

Both of these changes make gulp.src stricter and harder to use. I will sometimes hand arrays to gulp.src that can be empty in some circumstances, or an array of globs, some of which do not match any files (but may in the future).

I started some discussions in a vinyl-fs issue stating the problems I have encountered with the new behavior:
gulpjs/vinyl-fs#40 (comment)
gulpjs/vinyl-fs#40 (comment)
gulpjs/vinyl-fs#40 (comment)

It seems there are some anti-patterns in gulpfiles that were the cause of these changes, as @contra mentioned in:
gulpjs/vinyl-fs#40 (comment)

I would like to make the case for returning to the more forgiving gulp.src. This would require behavior changes to is-valid-glob and glob-stream.

Here's some examples of the added complexity required now:

// this
function moveVendorCssFiles() {
    return gulp.src(mainBowerFiles({filter: '**/*.css'}), {base: paths.bowerComponents})
    ...
}
// becomes
function moveVendorCssFiles(cb) {
    var bowerCss = mainBowerFiles({filter: '**/*.css'});
    if(bowerCss.length === 0) {
        cb();
        return;
    }
    return gulp.src(bowerCss, {base: paths.bowerComponents})
    ...
}
// this
gulp.src([
    'app/*.module.js',
    'app/**/*.js',
    '.tmp/index.js'
])
...
// becomes
gulp.src(globby.sync([
    'app/*.module.js',
    'app/**/*.js',
    '.tmp/index.js'
]), {base: 'app'})
...

Commence discussion!

@contra

This comment has been minimized.

Copy link
Member

@contra contra commented Oct 1, 2015

I'm curious about your use case, why aren't you having gulp.src resolve the globs?

@phated

This comment has been minimized.

Copy link
Member

@phated phated commented Oct 1, 2015

Edit: wrong points removed

@contra

This comment has been minimized.

Copy link
Member

@contra contra commented Oct 1, 2015

@phated allowEmpty is for ignoring the error when a file is not found in the stream, not for ignoring the error on invalid glob

@phated

This comment has been minimized.

Copy link
Member

@phated phated commented Oct 1, 2015

Oops, I was thinking the wrong thing, removed.

@phated

This comment has been minimized.

Copy link
Member

@phated phated commented Oct 1, 2015

@contra somehow it became commonplace (I think through some tutorial series) for people to not specify their globs, but instead rely on that main-bower-files module to build their array of paths (it doesn't actually have any globs, just paths). I don't see the appeal to this, but I've done away with bower so very long ago.

@vbud

This comment has been minimized.

Copy link
Author

@vbud vbud commented Oct 1, 2015

@contra if I allow gulp.src to resolve the globs, I get an error from glob-stream. In this particular case, .tmp/index.js does not always exist. If I could configure gulp.src to use glob-stream's allowEmpty (can I do that?) that may solve that part of the problem.

@phated regarding mainBowerFiles, that's just an example of a case where something else is used to resolve globs into an array of files, then the resulting array of files is handed to gulp. Don't get hung up on bower - it could just as easily be globby or some other tool handing gulp.src an array of files.

To shed more light on the globby use case - we have a lazy-loading setup that manages several individual modules. So lots of non-gulp code runs to handle determining which files make up each module, and then those files are handed to gulp. At that point, we no longer have globs; we have arrays of already-resolved files. In some cases, these arrays can be empty.

Keep in mind also the broader use case of application generators, where some operations are stubbed out for handling files that could exist in a project, but don't necessarily exist at the beginning of a project.

Hopefully that helps you guys understand my use cases a bit better.

@phated

This comment has been minimized.

Copy link
Member

@phated phated commented Oct 1, 2015

I thought about this more last night.

The 2 reasons this was added were:

  1. To avoid the gulp.src([]).pipe(exec()) usage of vinyl streams
  2. To help users debug why a pipeline wasn't processing anything (since it would throw telling them it is an invalid glob)

What if we just add a validateGlob option that defaults to true and use that option as part of the isValidGlob conditional?

@phated phated added the question label Oct 1, 2015
@contra

This comment has been minimized.

Copy link
Member

@contra contra commented Oct 1, 2015

@phated too many options imo

@vbud

This comment has been minimized.

Copy link
Author

@vbud vbud commented Oct 1, 2015

The validateGloboption would work.

For 2, I usually use gulp-debug to see what is in the pipeline. Is there a reason you guys don't recommend that?

@contra

This comment has been minimized.

Copy link
Member

@contra contra commented Oct 1, 2015

We should just revert that commit of invalidating the array glob since it had unintended consequences.

Solution will happen at micromatch/is-valid-glob#3 - thanks for clarifying!

@contra contra closed this Oct 1, 2015
@phated

This comment has been minimized.

Copy link
Member

@phated phated commented Oct 1, 2015

I think that we should still keep the logic of checking for an empty array in vinyl-fs but just log a warning. This is an important debugging tool

@phated

This comment has been minimized.

Copy link
Member

@phated phated commented Oct 1, 2015

I am going to reopen this because we've only had feedback from 2 people so far and I'd like to hear from others.

@phated phated reopened this Oct 1, 2015
@tunnckoCore

This comment has been minimized.

Copy link

@tunnckoCore tunnckoCore commented Oct 2, 2015

Okey, just digging around of these issues and read all comments.

I can't realize what and why you (@vbud and the others against this change) expect something to happen when there's no such actual file on the file system? And actually, why even you would run gulp "on nothing". I can't accept their state. I understand the case, but can't accept it, it's pointless for me.

There's no logic in the https://github.com/vbud/gulp-testing and the examples on the first post of this issue. It's total overkill to pass nothing (empty thing-ish a.k.a empty array) to gulp and do nothing. And it's more overkill and more error prone than just writing simple if statement and executes the cb.
There's no logic to run all this hard and slow mechanisms of checks, foreachs, globbing, concat/merge/combine streams, passthroughs and etc stuffs - i mean one word - gulp. It will take 100x more time and more possibilities of errors than just one "if" which only job will be to just check some case (e.g. checking is empty array, then call cb)
It's also overkill to use globby (and even sync calling it.. nah) and then to pass the paths to gulp.src

I'm absolutely agreed with gulpjs/vinyl-fs#40 (comment). There are a tons of tutorials and cases which use gulp for everything, instead of just use "normal js code to do it". They make their lifes harder.

If someone is lazy or don't know sometimes how to trick some things exactly, it isn't problem of gulp, or google or whatever. Talking generally. And as you can see there's always have some workaround for everything. It's not "harder, stricter, or trickier", it's better or at least - correct. It's a lot better to do it with if statement instead of running something which will do nothing, i mean.. absolute no logic. Most of the times it's better to be trickier instead of be "x, y,z -way", or "simple/easy".

That's why I love strictness and use it everywhere - allows me to do more than crazy things and to have a lot more success and lower issues.

Cheers.

@jonschlinkert

This comment has been minimized.

Copy link
Member

@jonschlinkert jonschlinkert commented Oct 2, 2015

I'm not sure if this helps or hinders, but node-glob itself will throw an error if an empty string is passed.

@vbud

This comment has been minimized.

Copy link
Author

@vbud vbud commented Oct 2, 2015

@phated logging a warning sounds like a reasonable balance to me - it helps debugging while allowing the behavior.

@vbud

This comment has been minimized.

Copy link
Author

@vbud vbud commented Oct 2, 2015

To me, this situation is really similar to running a map operation on an empty array - if there is nothing in the source array, there is nothing in the output array. The map operation does not throw an error because the array is empty.

In the same way, if I hand gulp.src nothing, it should do nothing, but it shouldn't throw an error. Is this a reasonable way of looking at the situation or am I missing something?

@tunnckoCore It seems like we have a philosophical difference :). But more importantly I think you are completely mis-representing my position. The gulp-testing repo was for testing current gulp 4.0 behavior, but it is not an example of the kind of thing I want to do. I never want to hand a file to gulp that will never exist - the whole point of my examples is that the file could or will exist, but does not always. The examples I gave at the beginning of this issue are more representative of what I'm trying to do, so please react to those instead.

Also, please don't get hung up on globby.sync or mainBowerFiles or whatever - they are just examples of some other code resolving files and then handing to gulp. And globby is what I am having to use after these changes were made to pre-resolve files since gulp's behavior is now stricter.

To @tunnckoCore's last point about strictness - I'm all for strictness when it is preventing dangerous outcomes. Do you have examples of where this particular strictness in gulp has helped you?

@callumacrae

This comment has been minimized.

Copy link
Member

@callumacrae callumacrae commented Oct 2, 2015

What's the harm in just passing the allowEmpty option when you think the file might not exist?

@phated

This comment has been minimized.

Copy link
Member

@phated phated commented Oct 2, 2015

I have been thinking about this more and more and the way I see it (especially in light of @jonschlinkert's comment) is that an invalid glob is something that will never match a file; e.g. an empty string, an empty array, an array of empty strings, etc. I think we should continue to throw on invalid globs because there is no reason to take on the overhead of building a pipeline if there will never be any files in it. And to @vbud's point about map, you wouldn't give a map function a number, so why give a pipeline no files? That example is a little absurd but a JS map is much lighter weight than constructing an entire stream pipeline.

gulp 4.0 is all about function composition so I think we should defer this use case to userland. Something like the following:

var gulp = require('gulp');
var isValidGlob = require('is-valid-glob');
var mainBowerFiles = require('main-bower-files');

// this would become a module
function ifGlob(paths, task){
  return function(cb){
    if(isValidGlob(paths)){
      return task(paths, cb);
    } else {
      cb();
    }
  }
}

gulp.task('build', ifGlob(mainBowerFiles(), function(paths, cb){
  return gulp.src(paths)
    .pipe(gulp.dest('./out/');
});

Also, note that we are throwing inside a task, so with gulp 4.0, you can use the --continue flag to continue upon errors inside tasks.

@tunnckoCore

This comment has been minimized.

Copy link

@tunnckoCore tunnckoCore commented Oct 2, 2015

@vbud i can agree with your first paragraph, but also support the @phated question and state about it

you wouldn't give a map function a number, so why give a pipeline no files? That example is a little absurd but a JS map is much lighter weight than constructing an entire stream pipeline.

I understand that "the files could or will exist", okey, but so .. you know in what case (when) they exist, so it's enough also your code to know, so perfect case for ifs and workarounds. In js specially you can do one thing in ton of different ways, respect that.

It seems that these cases can be tricked and can be as @phated say "in userland", it's not so hard.

Do you have examples of where this particular strictness in gulp has helped you?

I don't use it and won't use it until we write better file globbing library than node-glob. But I actively following it and using ideas and libraries behind it. Okey, I have one-two big projects with gulp. But yea, core behind gulp4 is awesome and thanks to @phated.

@jonschlinkert: I'm not sure if this helps or hinders, but node-glob itself will throw an error if an empty string is passed.

👍 cool, this is the way we should move. :) one more reason why this is the correct way.

@vbud

This comment has been minimized.

Copy link
Author

@vbud vbud commented Oct 5, 2015

@phated when you say:

That example is a little absurd but a JS map is much lighter weight than constructing an entire stream pipeline.

Could you educate me a bit more here? What all happens behind the scenes when a stream pipeline is constructed, and what happens when it is run on an empty array?

@salarmehr

This comment has been minimized.

Copy link

@salarmehr salarmehr commented Oct 6, 2015

I think gulp should show a warning when there is no corresponding file for a plural pattern. It's rare that the developer specify a pattern without any corresponding files. A warning had no deleterious effect but can save huge debugging time.

Also, currently gulp does not return error for invalid plural paths.

var gulp = require('gulp');

gulp.task('test', function() {
    return gulp.src(['fakeDirectoy/*'])
        .pipe(gulp.dest('path'));
});
@storafrid

This comment has been minimized.

Copy link

@storafrid storafrid commented Oct 13, 2015

Gulp user here - I don't know the gulp internals. And I'm voting for a warning, if anything.

Passing an empty array to a function doesn't generally throw an error in most programming languages, including JavaScript. Due to this, I suspect that A) users will expect a silent exit, and B) code in the pipeline will exit fairly quickly.

As a naive example, my initial thought was to look at grunt-contrib-copy and it predictably starts with a forEach loop, which will accept an empty array (and obviously traverse it very quickly). This to me is a good overall design that should be honored in this case as well.

@phated

This comment has been minimized.

Copy link
Member

@phated phated commented Oct 13, 2015

A forEach loop isn't a fair example in this case, as the callbacks won't get executed and nothing gets built up. A stream pipeline is a much more complex set of objects that take a fair amount of time to construct (especially for large pipelines).

I still think throwing on invalid globs is the correct choice here, with a userland module to detect empty globs and not call the callback. It removes the incorrect usage completely and improves performance where people might get an empty glob due to some other thing they are doing.

@rxgx

This comment has been minimized.

Copy link

@rxgx rxgx commented Oct 13, 2015

For the example with bower, I've had better success using the right module that outputs globs.

var bower = require('bower-main');
var bowerFiles = bower('js', 'min.js');

function bowerBuild () {
  return merge2(
    gulp.src(bowerFiles.minified),
    gulp.src(bowerFiles.minifiedNotFound)
      .pipe(concat('tmp.min.js'))
      .pipe(uglify())
    )
    .pipe(concat('vendor.min.js'))
    .pipe(gulp.dest('./public/dist/js')
  );
}

function bowerDebug () {
  return gulp.src(bowerFiles.normal)
    .pipe(concat('vendor.js'))
    .pipe(gulp.dest('./public/dist/js')
  );
}

If any of the globs, such as bowerFiles.minifiedNotFound were empty, I don't expect to get a warning or error. I would, however, like it to avoid generating the pipeline. In the case that it's invalid, then go ahead and throw error!

@vbud

This comment has been minimized.

Copy link
Author

@vbud vbud commented Oct 30, 2015

It looks like @phated has decided on a direction and we just need a userland module :). Next time I'm faced with this I'll create one if someone else does not.

I'm happy to close this issue unless any of the core gulp contributors would like to keep it open.

@phated

This comment has been minimized.

Copy link
Member

@phated phated commented Oct 30, 2015

@vbud meant to follow up this week. There was more discussion about the pattern this stopped on the gulp-shell plugin and it reinforced my opinion that the throwing needs to stay in. I think your edge case isn't really a problem but keeping people from the bad use case has more benefit and we can defer your's to userland.

@vbud

This comment has been minimized.

Copy link
Author

@vbud vbud commented Oct 31, 2015

Sounds good. Thanks for the lively discussion.

@vbud vbud closed this Oct 31, 2015
@dgwaldo

This comment has been minimized.

Copy link

@dgwaldo dgwaldo commented Nov 4, 2015

I normally don't have an opinion, but I have to say this has caught my eye. I wrote a glob '../Views//.html' that blows up with File not found with singular glob. If I do '../Views/Home/.html' it's all good. So why does the glob pattern blow up? I'm assuming it's because there is no .html file directly under views, and when the array of paths is constructed from the glob pattern (or whatever happens) the first level down fails to match and boom.

Or is something else going on here? I'd expect that if something matches, I shouldn't get an error...

@callumacrae

This comment has been minimized.

Copy link
Member

@callumacrae callumacrae commented Nov 4, 2015

You probably wanted ../Views/**/.html

@phated

This comment has been minimized.

Copy link
Member

@phated phated commented Nov 4, 2015

That's really weird. I thought we had a different error for a valid glob that resolved to no files.

@dgwaldo

This comment has been minimized.

Copy link

@dgwaldo dgwaldo commented Nov 4, 2015

@callumacrae I think I got hosed by markup it is two ** in my code

@dgwaldo

This comment has been minimized.

Copy link

@dgwaldo dgwaldo commented Nov 4, 2015

@phated shouldn't it not throw any errors at all if the glob finds at least one file?

@callumacrae

This comment has been minimized.

Copy link
Member

@callumacrae callumacrae commented Nov 5, 2015

It just shouldn't throw the error, if it's a non-singular glob. That error makes no sense with that glob.

@contra

This comment has been minimized.

Copy link
Member

@contra contra commented Nov 5, 2015

@dgwaldo definitely a bug with the singular glob identifier, can you open an issue?

@dgwaldo

This comment has been minimized.

Copy link

@dgwaldo dgwaldo commented Nov 5, 2015

@contra This happened while trying to use https://github.com/jonkemp/gulp-useref finally decided to try and figure out what the heck was going on, and I arrived at the conclusion that it's actually a bug in that library. It takes a searchPath which is concatenated with the glob pattern provided to gulp.src() which I think is what was actually blowing to bits. If you didn't include a searchPath, it still didn't work... Anyhow switched to gulp-usemin and the exact same glob is now working perfectly... Sorry I blew up ;)

@mctolentino

This comment has been minimized.

Copy link

@mctolentino mctolentino commented Nov 5, 2015

I'm also having the same issue, has the issue been raised?

@phated

This comment has been minimized.

Copy link
Member

@phated phated commented Nov 5, 2015

@mctolentino as stated 3 hours ago, it was due to a poorly constructed plugin

@mctolentino

This comment has been minimized.

Copy link

@mctolentino mctolentino commented Nov 5, 2015

@phated weird. dgwaldo's comment wasn't there when I posted a few minutes ago. Only saw it when I refreshed the page. I'll just be using usemin then. Thanks!

@vdh

This comment has been minimized.

Copy link

@vdh vdh commented Nov 14, 2016

I'm sending gulp.src an array of ['*.map', 'manifest.json'], where manifest.json may not exist in some circumstances, but there are always going to be files matching *.map. It's incredibly frustrating to get a Error: File not found with singular glob error thrown in my face just because that one file is missing.

If it's really so important to throw an error for this kind of "issue", why not instead throw an error if gulp.src has no files after all of the globs have been evaluated?

@phated

This comment has been minimized.

Copy link
Member

@phated phated commented Nov 15, 2016

@vdh that isn't supposed to be happening (you have a non-singular glob in the patterns) and is likely a bug. Please create a reproduction case and submit an issue to vinyl-fs.

Edit: It's not possible to handle this with our stream interface. Use the allowEmpty option.

@vdh

This comment has been minimized.

Copy link

@vdh vdh commented Nov 16, 2016

@phated Okay, is this an adequate reproduction? (Gist attached in description) gulpjs/vinyl-fs#217

@Munter

This comment has been minimized.

Copy link

@Munter Munter commented Jan 9, 2018

@vdh @phated I ran into the same issue and submitted an issue to vinyl-fs with a reproduction

@gittac

This comment has been minimized.

Copy link

@gittac gittac commented Sep 13, 2018

I'm having trouble with the watch command. I'm using gulp v4 gulp-watch v5.0.1
Watch seems to be working correctly until I try to add a file while watch is running. I'm following a youtube vidio It suggests removing the dot Before the path and it works on the video, but it was using an older version of gulp. When I do it and it cannot find the files.

//Variables for the sources
var styleSRC = './src/scss/style.scss';
var jsSRC = './src/js/script.js';

//Variables for the watch command
var styleWatch = './src/scss//*.scss';
var jsWatch = './src/js/
/*.js';

gulp.task('default',gulp.series(['style', 'js']));

//Watch will keep running working for changes in src/js or src/js folders then run either style or js tasks when a change happens
gulp.task('w',gulp.series(['default'], function(){
gulp.watch(styleWatch,gulp.parallel(['style']));
gulp.watch(jsWatch,gulp.parallel(['js']));
}));

@demurgos

This comment has been minimized.

Copy link
Member

@demurgos demurgos commented Sep 13, 2018

@gittac
Thanks for the message. Posting on stale closed issues is not the best way to deal with bugs.
If you believe that you found a Gulp bug (I had some issues with watch in the past), please open a new issue following the issue template to get more information about your situation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
You can’t perform that action at this time.