changing this

[robrich]

Test the Beta

Swap your gulp dependency reference in package.json to github.com/robrich/gulp#develop and take the next version of gulp for a ride. Give us your feedback, and we'll continue polishing.

How to test the beta

Switch this:

  "devDependencies": {
    "gulp": "^3.0.0"
  }

to this:

  "devDependencies": {
    "gulp": "git://github.com/robrich/gulp#develop"
  }

then run npm update and you'll have the latest version.

Does everything still work? Give us your feedback.

Then switch it back so you can get the latest from NPM when it's released.

What's new

• gulp.run works very differently, now using task builders like so:

var builder = gulp.parallel('task1', 'task2', 'task3');
gulp.run(builder, function (err, stats) {

You can also use gulp.runParallel('task1', 'task2', 'task3', function (err, stats) {

• In addition to running in maximum concurrency (after respecting all dependencies), you can run a series. These two examples run identically:

Using dependencies:

gulp.task('one', function (cb) {
  console.log('running task one');
  cb(null);
});
// notice the dependency: `two` won't begin until `one` is done
gulp.task('two', ['one'], function (cb) {
  console.log('running task two, task 1 is done already');
  cb(null);
});
gulp.runParallel('one', 'two', function (err, stats) {
// this could also be `gulp.runParallel('two', function (err, stats) {
...

Using runSeries:

gulp.task('one', function (cb) {
  console.log('running task one');
  cb(null);
});
// notice the dependency is removed here:
gulp.task('two', function (cb) {
  console.log('running task two, task 1 is done already');
  cb(null);
});
gulp.runSeries('one', 'two', function (err, stats) {

runSeries ensures the prior task is complete before the subsequent task begins.

• You can also nest these builders to produce interesting workflows:

gulp.run(gulp.series('clean', gulp.parallel('css', 'js', 'img'), 'deploy'), function (err, stats) {

• run can accept an options argument, and if passed {continueOnError: false} it won't blow up if a task fails. This is great for watch tasks, and gulp.watch now uses this when you run tasks.

gulp.watch(['**/*.less'],['less']);
// this no longer crashes node when the less compiler fails

• Nested orchestrations now run as expected, and you can use them to create a series:

gulp.runParallel('task1', function (err) {
  gulp.runParallel('task2', function (err) {
    gulp.runParallel('task3', function (err) {

but why would you bother doing it that way when you can do the same thing like this:

gulp.runSeries('task1', 'task2', 'task3', function (err) {

• You can now not only specify dependencies that must run before the task, you can also specify follow-up tasks and other properties, and the task runs scoped to a copy of your configuration object:

gulp.task('mytask', {
  before: ['clean'],
  after: ['deploy'],
  other: 'configuration',
  passed: 'to your task'
}, function (cb) {
  // this.other === 'configuration'
  cb(null);
});

• Many of the old crufty API details are removed, and this yields many breaking changes in places you probably weren't using anyway. See Deprecation warning, what say ye? robrich/orchestrator#30. Of particular note is that synchronous tasks are no longer supported. You must call the callback, return a promise, or return a stream. In a future version, you'll also be able to return an array of promises or an array of streams.

How did we do? Do you like these new features? Are they ready for prime-time? Give us your feedback, and we'll continue polishing this release.

[yocontra]

I'm not a fan of the runSeries/runParallel fns, its more API surface than is needed IMO

[robrich]

You'd rather run(parallel('task1', 'task2'))? Intriguing.

[yocontra]

Yeah its more intuitive that way. You have to weigh the value of these shortcuts vs. adding more surface to the project

[robrich]

I'm good with pulling them. They're just sugar.

[jasonrhodes]

+1 on removing some of this, it feels like a lot of stuff going on. Is the "before/after" option syntax solving a different problem than the series/parallel builder syntax? It seems like there are a lot of ways to do the same thing, with task dependencies, builders, before/after hook syntax…

(Nice work on all of this overall, @robrich btw)

[robrich]

Yes, before/after is from robrich/orchestrator#21 and series/parallel/builder is from robrich/orchestrator#15. Thanks for the kudos.

[yocontra]

Yeah we should spend some time trimming features here. I don't like specifying task options or deps ahead of time, I'd rather do everything via the chaining syntax. I think most if not all task options can be removed

[robrich]

Most dependencies can be resolved when you define the tasks. It's quite
rare that you need to define them at run time.
On Mar 16, 2014 11:58 AM, "Eric Schoffstall" notifications@github.com
wrote:

Yeah we should spend some time trimming features here. I don't like
specifying task options or deps ahead of time, I'd rather do everything via
the chaining syntax. I think most if not all task options can be removed

Reply to this email directly or view it on GitHubhttps://github.com//issues/347#issuecomment-37766374
.

[phated]

I think we are going about this in quite the wrong way. @contra has said before that he no longer wants the array of task dependencies to exist anymore and I agree with him based on some APIs I have been working on. Like such:

gulp.task('task1', ['some', 'deps', 'here'], function(cb){
    cb();
});

My proposed solution is to remove that array of deps, but the same effect can be achieved as such:

gulp.task('task1', gulp.parallel('some', 'deps', 'here', function(cb){
  cb();
});

As you can see, this removes the run APIs all together, as we are just composing functions into a single callable function.

var aParallelFunction = gulp.parallel('some', 'task', 'strings', 'here');
aParallelFunction(function(err, res){
    // err will be the errors that occurred
    // res will be the results of called functions
});

An API like this would allow for things like:

// an alias
gulp.task('default', gulp.series('clean', gulp.parallel('js', 'css', 'img'), 'watch'));

Which would run clean first, then js, css, and img, in parallel, then watch last.

A watch task could look like:

gulp.task('watch', function(){
    return gulp.watch(['**'], gulp.series('default', 'notify'));
});

The main concept behind these APIs are composability and that they are just callable functions. You might think it is adding extra verbosity, but I feel like it is less than the current beta APIs are adding.

An added bonus to this approach is that the task management layer doesn't need to account for AOP concepts.

var gulp = require('gulp');
var meld = require('meld');

function before(){
    // do stuff before
}

gulp.task('default', meld.before(function(cb){
    // our task definition
    cb();
}, before);

[jasonrhodes]

I'm not sure if this solves or causes more problems as far as what these changes were meant to address, but just based on first reaction, @phated's proposed syntax feels much more natural and easier to reason about as far as integrating into my current gulp workflow.

[yocontra]

@phated has it on point, this is exactly what I want. composable functions instead of config is exactly what gulp is about. Every config option and complex function signature kills me a little inside

[phated]

I already have the function composition stuff done through bach which uses async-done and async-settle internally to solve some things that async doesn't do as gulp requires.

The last remaining piece would be a task registry. Hopefully, we can keep orchestrator as that. I have some really interesting ideas on a pluggable task registry, but I think @robrich would have to add some maintainers to the project to keep it moving ahead rapidly.

[yocontra]

With the task composition do we even have to do a dependency graph to figure out which order to run stuff in? The user is in charge of specifying the order which makes things infinitely simpler

[phated]

@contra correct, but I think that we still need the key -> function stuff, also, we need the string (key) -> function resolution. I think those things should live in a registry module.

Bach takes function definitions (not strings/keys).

[yocontra]

@phated This goes back to the original gulp task system which was

gulp.task = function(name, fn){
  gulp.tasks[name] = fn;
  return gulp;
}

I think this is a lot more intuitive, good work!

[robrich]

I like where this thought is going -- simpler is better. There's a few
things async-done does really well that I'd love to leverage, and a few
places where it doesn't go far enough. Like people have asked for gulp to
support returning arrays of promises or streams, but that seems trivial to
add.

[phated]

@robrich async-done does just the minimal required. Other things should probably happen in other small modules, but I do like the idea of returning an array of promises/streams. I will think on those APIs today.

[ScottWeinstein]

I can't get this to work, unless the changes are intended to be breaking changes for existing code.

Using the following Gulpfile.js

var gulp = require('gulp');
var gutil = require('gulp-util');
var PluginError = gutil.PluginError;

gulp.task('default', function () {
    console.log('default');
});

gulp.task('errTask', function(done) {
    throw new PluginError('sample-error',  'an error');
});

I get the following message on both 'gulp default' and 'gulp errTask'

tester> .\node_modules.bin\gulp
[gulp] Using gulpfile tester\Gulpfile.js
gulp.start() has been deprecated. Use task dependencies or gulp.run(gulp.parallel()) or gulp.run(gulp.series()) instead.

[robrich]

@ScottWeinstein Good catch. There was one more runParallel() reference that got left behind as that feature got pulled. Try it now.

[ScottWeinstein]

@robrich the first errors are resolved.

It looks like errors thrown or passed into the done CB still produce big stack traces, perhaps that fix isn't part of this beta?

But... this this code

gulp.task('errTask-stream', function() {
    var raiseError = function() {
        return this.emit('error', new PluginError('sample-error',  'err'));
    }
    return gulp.src(['*.js'])
               .pipe(through(raiseError, raiseError));
});

produces the following

tester> .\node_modules.bin\gulp errTask-stream
[gulp] Starting 'errTask-stream'...
[gulp] Finished 'errTask-stream' after
Error: task 'errTask-stream' timed out, waited 20000 ms
at null._onTimeout ? >(gulp\node_modules\orchestrator\lib\runOne\timeoutTask.js:14:16)
at Timer.listOnTimeout as ontimeout

[robrich]

Timeout is unexpected (unless you specifically didn't call the callback)
but huge stack traces are expected.

[tom--]

I like @phated's API. Action verbs such as 'run' do not feel like they belong in gulpfiles. Action verbs are for the command line only.

[tom--]

@phated mentioned a registry for "string (key) -> function resolution". I assume these strings are task names.

Yesterday on #gulpjs I asked why not use JavaScript function names instead. Task name strings in a global namespace doesn't feel right in a functional system. And they can be inconvenient.

For example, my build has a number of tasks ("a", "b", ...), each of which has its own "build", "watch" and "clean" sub-tasks. I have to come up with naming scheme to represent these in a global namespace, e.g. "clean-a", "clean-b" etc. After only one week as a gulp user, I've actually been doing this in real work. To keep everything straight, I've had to write functions that generate tasks with systematic names.

What if, instead of this string->function registry, the gulp CLI referred to functions in the gulpfile by variable name with some simple way of expressing scope, e.g. gulp a,clean? Now I can use function scope to my advantage in gulpfiles, taking advantage of hiding, and the task names emerge form the structure and can't get out of whack.

It seems to me that would be more convenient and useful. And more in keeping with JavaScript function programming, ideologically speaking. Constructing a way to artificially name functions globally doesn't feel right.

I understand this is a lot different from what people are used to. Quite possibly the idea isn't even coherent. But I'm curious if it could be done.

[yocontra]

@tom-- The function identifier is really only for calling tasks from the CLI. If we figured out a clever way to lose the identifier it would make things quite a bit simpler and we wouldn't need this whole definition/resolution process. gulp.task would cease to exists, and you would just pass functions into the chainable run builders

[OverZealous]

Overall, I really like @phated's solution, especially the way it allows for function reuse and composition. This is very powerful.

@contra If you change the function identifier setup, it should be done in such a way that allows for both naming and describing tasks.

I think there needs to be something equivalent to grunt -h showing a list of tasks and descriptions. Right now, there's no way to easily see a list of available tasks, and this is even more important in gulp, since the gulpfile can easily become hard to read. Edit I'm very wrong on this one, there is a -T command I didn't realize about. Still, it would be nice to have descriptions.

I wonder if there could be a way to use the module pattern to expose public functions to the command line tool, so you can still get a task listing:

function buildStyles() { ... };

var build = gulp.parallel(...);
build.description = 'Builds everything';

// or even module.exports
gulp.exports = {
    'build-styles': buildStyles,
    'build': build
};

You can also easily loop over them and look for a fn.description to render out the description of the command.

I was trying to figure out how to do this without the explicit command name (using fn.name to read the name of the function), but there's a fairly big caveat: You can't set a name for anonymous functions. If there's an easy solution to this, you could change the exports to be:

gulp.exports = [ buildStyles, build ];

If there is a way to get a list of all top-level variables for the gulpfile, then you might be able to do this without the exports at all. Just collect all top-level functions and automatically determine it's name using the variable/function name. However, you might want to only show commands that have an explicit description in the list of available tasks.

If you do it this way, I recommend allowing for hyphen-case to be converted to camelCase automatically from the command line, so that build-styles is treated as buildStyles.

[tom--]

Build scripts are also used by people who don't author or understand them so I think @OverZealous suggestion is very good: only export to the CLI a task that has a non-empty description. If possible, derive task names from function names but don't limit it to the top level.

Otoh, I have no idea how to do it or if it can be done. :(   @tom-- is just a user.

[phated]

That should be something in liftoff probably /cc @tkellen

[OverZealous]

I'm still mulling this over. The thing that has me concerned is that with @phated's suggestion there's no actual task dependency.

Would it still be possible to prevent tasks from running twice if they are requested in the same set of tasks?

For a slightly contrived example, imagine I had:

function clean() { ... };

var build = gulp.series(clean, ... );
var dist = gulp.series(clean, ... );
var buildAndDist = gulp.parallel(build, dist);

I would expect running gulp build-and-dist to run clean, then run the rest of build and dist in parallel. I'm not sure how you would make sure that a dependent task is only run once, before everyone else. I suppose you could add something like fn.$$hasrun to each function object, but then you'd be back at needing something like sequencify to figure out the correct order.

Also, if I ran gulp clean build from the command line I would expect the actual process to be clean, build, not clean, clean, build. More importantly, without something more complicated, I don't see how to set up non-dependent ordered tasks. Meaning, if I had this:

function clean() { ... }
function build() { ... }

Running gulp clean build won't necessarily run in the correct order — the clean may happen before or during the build. You could do this:

function build() { ... }
build.after = [ clean ];

But this is getting a little ugly. However, it does allow for pure functions for most of the gulpfile.

[danielchatfield]

Could you publish it to npm with the beta tag such that it can be installed using npm install gulp --tag beta

[phated]

@danielchatfield that version of orchestrator most likely won't be releasing in gulp, so it doesn't make sense to publish it.

[witoldsz]

@OverZealous

The only odd thing about it — which I haven't been able to figure out a solution — is the description coming after the declaration.

Another option could be like this:

js.description = 'This task compiles the JS files';
function js() {
    return gulp.src()...
});

It does not help with functions being created like var default = gulp.series(clean, build); though...

So, maybe we could add some sugar to gulp:

var js = gulp.describe('This task....', function() {
    ....
});

var watch = gulp.series('Clean, build, then watch files to rebuild on changes', clean, build, function(){
    gulp.watch(paths.js+'**/*', js);
});

[laurelnaiad]

I'm late to comment on this, but I've been working on a fairly complicated gulp process and I've taken a radically different approach.

I'm using lazypipe to create composable pipes. And I'm using some plugins that I've developed to handle conditionals and branching/merging/etc of streams in the processing.

Here's some code that has a very different take on composition. My uber pipe for building is starting to look like this:

// everything on `this` ends up being turned into a task
  this.build = function() { 

// a wrapper for vfs.src that sets base for me, applied to the directories I build into
    return gmlfs.src(targets) 
    .pipe($.clean({force: true}))
// a custom pipe that ensures I clean the pipeline's slate and wait for completion    .pipe(gml.drainAway()) 

// now my context is all files under `src`
    .pipe(gmlfs.src(globs.all)) 

// a plugin that let's me select subsets, handle them 
// individually and in parallel, and then have them 
// merged back into the main pipe
    .pipe(gml.subsets({keepAll: true}, [  

// with the files that match the sass glob,                                                        
      [globs.sass, function(files) { 

        return files
// run them through a lazypipe that does sass stuff
        .pipe(self.sass()) 
// prove we're only working on .scss files that aren't imports
        .pipe($.debug({title: 'sass'})); 

      }],

// for files that match the js glob
      [globs.js, function(files) {  

        return files
// this is a lazypipe that does all things .js-related
        .pipe(self.js()) 
// prove we're only working on .js files
        .pipe($.debug({title: 'js'})); 

      }]

    ]))

// prove that it was all merged back together
    .pipe($.debug({title: 'all'})); 
// write it all out
    .pipe(gmlfs.dest(targets.build);

I guess what I'm getting at is twofold:

• I find composable pipes such as this much more powerful than composable tasks, and lazypipe should, IMHO, be built-in to gulp in some way. I'm planning to release the little stream handling plugins I'm building to support this methodology
• I think it's easier to figure out what's going on when you represent things this way instead of having the dependency resolution process process handle it, because the code is all right in front of you in a form that represents how it will be done, as opposed to the opacity of the task dependency/serialization/parallelization process.

Obviously, switching up the orchestrator implementation and using bach won't preclude me from doing things this way, but it might encourage people to lean on the dependency resolution to the point where it is more difficult to discern what is going on and what depenencies are dependencies within the tasks' internal context in addition to the granularity of tasks as units of execution. Many of us know what kinds of messes can be made when a task runner allows you to or unwittingly encourages you to string together tasks with hidden dependencies...

Please don't go away from streams in favor of composable tasks! Streams must be the de facto way of organizing your code or it's back to impenetrable build files.

Just my 2 cents.

[yocontra]

@stu-salsbury Nowhere did I say I was going away from streams. The two are not mutually exclusive. Also if you look at the proposal @phated is putting forward you'll notice that the new task system has no dependency trees or anything. It's just plain functions.

[laurelnaiad]

I wasn't saying I think they're going away. I meant to be saying that I think it's not necessarily a great idea to promote task-based composition. Pretty soon there will be hidden dependencies between tasks -- where one task relies on another task having put files in a certain place at a certain time, under certain conditions.... and that reminds me painfully of some bad code I wrote to deal with complex interdependencies in another task runner. I think there are ways to manage sequencing that are just as easy and yet more powerful and still stick to streams.... but I also acknowledge that there is a segment of the developer base who may not be comfortable handling everything with composable streams... so I do understand the inclination to compose tasks, even while it make me a little wary for the practices it might encourage.

[yocontra]

@stu-salsbury Where are you getting "hidden dependencies" from? You may be getting our official message confused with other people's recommendations in this thread.

[laurelnaiad]

In gulpfiles — as a result of people coming from other task runners that compose tasks together, where the method of communicating is the drop files all over the place and monkey with tasks being composed of other tasks that depend on each other. I think task-composition of any sort, be it through a dependency resolution system or through a series of parameters that are in plain sight (but still say that one task is a superset of the other) lends itself to people not thinking of each task as a unit of work that the user would reasonably need to run on the command line.

I mean, I don't run gulp jshint if I have a decent gulpfile. I run gulp build or whatever, and if the jshint portion of the build is set to fail the build, it stops there, otherwise I get a warning… or whatever. My point is that in real life people want to do a couple of things.

• build
• test
• publish
• etc.

None of those is named, "sass" or "requirejs" or "mocha" because those are tools, not workflow-level tasks… but if people are led to composing tasks with subtasks, they'll create such miniworkflows as tasks, and the mocha task will depend on the test script having been placed in the right directory by the copy task, and they'll both depend on the requirejs task to have resolved dependenies... or whatever... tasks depending on each other to do things to files… and it's all downhill from there.

[OverZealous]

@stu-salsbury I don't know where you are getting the idea that tasks are called sass or requirejs in this discussion. That's not what task composition is about at all.

It's about composing bigger tasks, like clean -> build -> test -> package -> deploy, or even build-scripts and build-styles, then build-html, which then becomes the build task. Scripts, styles, and html are all processed in separate streams. They probably depend on each other, but the internal streams are independent.

[laurelnaiad]

I'm not saying that you guys are proposing to do things the way I'm describing it, I'm saying people will be more likely to do such things if task composition is a prominent feature. I had previously seen prominent contributors encouraging people not to run tasks from other tasks, and I still believe it's not a great pattern. It's ok, I'm not going to be forced to use it. Note: what is the purpose of a task named "clean"? Do you run that on its own for some reason? Not to be snarky about it, but it doesn't really have any value on its own... nor does "build-styles".... that's part of my point, the other being a slippery slope of under-emphasis on interdependencies.

But oh well. I put in my two cents, that was all I intended to do. I didn't mean to step into an argument about it. If my opinion isn't commonly shared, that's ok. It's all good.

[OverZealous]

@stu-salsbury I'm not trying to argue, but you are nitpicking over a few names, without seeing the bigger picture.

Yes, I run clean on it's own. (Actually, in my code, I have clean-build and clean-dist, with clean being composed of both of them. I can clean the whole project, or just parts of it, as needed.)

Yes, I run or re-run build-styles on it's own, if I was testing something and don't want to re-run the entire build for resolving an issue. (It's a little confrontational to say that someone's task setup "doesn't really have any value", just because you work differently.)

More importantly, even if I never ran these sub-tasks on their own, encapsulating the process to build CSS (or JS, or whatever) in it's own task makes it easier for me to maintain the gulpfile, and makes grouping asynchronous tasks easier. It helps me keep related things together, and unrelated or less-related things separate. For me, your code looks convoluted. Maybe you'd feel the same way about my gulpfile. 😄

Yes, you shouldn't run task B from within task A. That doesn't mean that task C can't be composed of task A followed by task B—currently handled via dependency management, in the future via composition.

Building up bigger tasks from smaller ones is one of the best features of a task runner. Being able to run smaller tasks as you need is a great feature, too. I don't see how this is going to lead to a Grunt-like load-modify-save-repeat setup, since the number one feature of gulp is that it processes files in memory using streams. I don't think that means that every task needs to be a single, self-contained stream for all files, though!

(Note: I don't speak for anyone involved in creating gulp. I'm just an end user myself.)

[laurelnaiad]

More importantly, even if I never ran these sub-tasks on their own, encapsulating the process to build CSS (or JS, or whatever) in it's own task makes it easier for me to maintain the gulpfile, and makes grouping asynchronous tasks easier. It helps me keep related things together, and unrelated or less-related things separate. For me, your code looks convoluted. Maybe you'd feel the same way about my gulpfile.

I guess that's the crux of the matter. The code I posted above is overkill because it's not doing anything yet that deserves worrying about dependencies or interrelationships. Encapsulating the process to do CSS is something we both want to do. I just want to do it by subsetting my source stream and doing a variety of things to my source CSS that have branches... for both my 'build' and my 'dist' directories, I want to convert SASS to CSS... but for the dist directory I also want to minify it. I'd rather not run SASS twice, so the input to building the dist target is output from the step that builds the build target... or I could just run SASS twice and get the same result, if I wasn't worrying about running SASS twice, or preferred to set up a bunch of tasks to use as subtasks so that I can manage pieces of the build that way. It's not a crime to do it one way or the other, I just think it leads toward less performant and less manageable code.

Also, I didn't mean to say your task setup doesn't have value -- no harm intended. I was thinking of a watch-scenario where small changes to test happen when you save them. I see that if you're not running in watch mode you might use a lot more of the smaller focused tasks to get the job done quickly. It wasn't supposed to be personal, one way or the other.

Anyway, let's agree to disagree about whether task composition is feature that we like. I guess we can both agree that gulp is a great task runner and leave it there.

[suedama1756]

Is this still active? When will the new orchestrations be available in gulp, github.com/robrich/gulp#develop was last contributed to a month ago?

[phated]

@suedama1756 orchestrator's develop branch will never be used in gulp.

[suedama1756]

Ok, lets look at the original title "Beta test the next version of gulp and orchestrator", perhaps we can then be a little more helpful. Are the features relating to the post, such as running tasks in series etc. coming to gulp any time soon. Clearly I wouldn't expect the dev branch to be used directly in gulp!!

[robrich]

Look at GitHub.com/orchestrator/bach for more info on the next iteration of
gulp.
On Jun 27, 2014 12:04 PM, "Jason Young" notifications@github.com wrote:

Ok, lets look at the original title "Beta test the next version of gulp
and orchestrator", perhaps we can then be a little more helpful. Are the
features relating to the post, such as running tasks in series etc. coming
to gulp any time soon. Clearly I wouldn't expect the dev branch to be used
directly in gulp!!

—
Reply to this email directly or view it on GitHub
#347 (comment).