Watch task - potentially useful feature request

[zonak]

I wanted to bring up a watch task scenario that might by useful and not that uncommon.

Currently the watch task allows us to monitor a file, a group of files or a folder for changes for certain type of files and upon change fire a task that is already defined. This is great but I think there is space and a need for improvement.

Lets say we want to use the watch task for things like CoffeScript, Less, Stylus compiling instead of using their own watch features for the purpose of centralizing our tasks and also fill in for the absence of a watch feature like in the case of Less.

At this point when a task is initiated upon a file change we can only run an existing task without really knowing which file has changed. If the name of the file that has changed can be passed to a task that knows how to handle it would be very beneficial.

I'll try to explain with an example:

Lets say we are watching a folder for changes in CoffeScript files. If one file changes, we don't necessarily want to re-compile all the files but just the one that has changed.

Given that currently there is no way for tasks to communicate with one another, enabling such mechanism may even lead to some other useful stuff.

Ultimately I'd like to use grunt for everything that it is or potentially might be able to handle and avoid the need of using other build / minify / compile tools.

[webxl]

+1

I tried modifying the watch task to pass the updated filename(s) to the tasks it calls, like this:

...
// Enqueue all specified tasks, followed by this task (so that it loops).
task.run(tasks, changes).run(nameArgs);

But things got a little messy inside lib/util/task.js

...
Task.prototype._taskPlusArgs = function(name) {
    // Task name / argument parts.
    var parts = name.split(':');
    // Start from the end, not the beginning!
    var i = parts.length;
    var task;
    do {
      // Get a task.
      task = this._tasks[parts.slice(0, i).join(':')];
      // If the task doesn't exist, decrement `i`, and if `i` is greater than
      // 0, repeat.
    } while (!task && --i > 0);
    // The task to run and the args to run it with.
    return {nameArgs: name, task: task, args: parts.slice(i), updatedWatchFile: this.updatedWatchFile};
  };

  // Enqueue a task.
  Task.prototype.run = function() {
    // Parse arguments into an array, returning an array of task+args objects.
      if (arguments.length == 2) {
          this.updatedWatchFile = arguments[1];
      } else {
          this.updatedWatchFile = null;
      }
    var things = this.parseArgs([arguments[0]]).map(this._taskPlusArgs.bind(this));
    ...

[cowboy]

I'll probably just create a grunt.state.watch object and store the filenames in there, instead of passing them around. Not sure yet.

[zonak]

Are you planing also on storing an indicator of the tasks that are related to the changed files?

This would be useful so that the task that needs to react on a change has a mechanism of filtering out changes to unrelated files when performing their job.

[cowboy]

I'm not sure I understand. There is no relationship between the changed files that watch detects and any other task.

[zonak]

Let's say we are watching some less files and some coffee files.

A change occurs in a coffee file so the "coffee" task is initiated and the changed files are recorded in the mentioned global. The "coffee" task looks up the changed files and compiles them.

But in the mean time a less file is changed and recorded in the global too.

So couple of questions from this example:

• who and when clears the records of the changed files
• if I want to act only on the "coffee" task related files how would I distinct them from the less files that also changed recently

in the example we can go by extensions but that might not always be the case.

[cowboy]

The watch task doesn't work that way. It simply waits for files to change. When one or more of them change within 250ms of each other, a list of tasks is run. When that list of tasks completes, watch waits for files to change again.

Watch has no idea what files may have changed while the other tasks are running, because while the other tasks are running, it's not running.

[cowboy]

In other words, watch can create a list of files that has been added / deleted / changed while it's running, but it can't know what files (if any) were added / deleted / changed while other tasks were running.

If there's a possibility that a file might be modified while the watch task isn't watching, relying on a watch-generated file list might not be enough for what you want to do.

[sokra]

It is not good to process only the changed files, because of dependencies between files.

consider this scenario:
a.less and b.less.
b.less contains a @include(a.less).
if you change a.less, b.less must be recompiled too.

so i think it's a better idea to make the tasks more clever. tasks should only recompile files which timestamp is newer than output file timestamp. Tasks which supports some kind of including mechanism have to consider it while checking timestamps. This makes compiling also faster.

[zonak]

There are obviously different scenarios and what I wished for is that grunt handles all these tasks that people already do with different approaches. I guess I was aiming more for something like what watchr does.

@sokra less is a bad example here - lessc does compile dependencies, so what you would do in most cases is compile only your main less files and their dependencies are taken care by the less compiler.

A better example for my point would be the case where we have 150 coffee files for example - we don't want to compile all of them each time a single file changes but we should be watching all 150 of them for changes.

The list of tasks that run upon file change is determined by the file pattern defined so the logical step would be to somehow pass along the information which files exactly were changed - I don't think that such scenario would be in contradiction with the way grunt watches for file changes. With placing the list of changed files in a global all we would need is some sort of flagging that would indicate who is addressing these changes and let the task clear them out once they are done with what they are doing.

I'll try with an example:

We are watching the mentioned 150 coffee files. Upon a single file change grunt saves the file name in the global and calls the "coffee" task. The coffee task flags the files it is going to deal with (so that another task doesn't do the same) and once done it clears them from the global object. If more than one task is defined for the set of files, the last one executed clears the files from the global object.

In the mean time if there is a subsequent file change it is taken care of in the same manner but by filtering out the files that are marked as being processed if there is a time overlap.

Again these are ideas that I think would be useful but not necessarily realistic. Thank you for looking into them.

[sokra]

couldn't comparing of timestamps solve it? (because there is a one-to-one relation between coffee and js files)

[zonak]

I was thinking of scenarios where we have multiple tasks defines for the same set of files - something that grunt has as a feature.

Just for example: running a compile and a minify task against the same set of files.

[sokra]

hmm.. i do not get it :(

did you mean the compile task tells the minify task which files were changed?

[zonak]

Here's on of grunt's own examples:

https://github.com/cowboy/grunt-jquery-example/blob/master/grunt.js

lint: {
  files: ['grunt.js', 'src/**/*.js', 'test/**/*.js']
},
watch: {
  files: '<config:lint.files>',
  tasks: 'lint qunit'
}

Hope this explains it better.

[cowboy]

@zonak, all that does is reference the list of files used in lint.files to avoid having to maintain the list in two placed. The watch task is completely unaware of the relationship between watch.files and lint.files.

[zonak]

My point was that both the lint task and the qunit task run against the same set of files and initiated by a change in that list of files.

[cowboy]

The watch task is really very simple. It just waits for files to change, then executes some arbitrary tasks. It's not at all smart.

A robust, generalized approach to this problem might be to write two methods such as these:

• file.filesModified - return a list of files and their last modified times, optionally handling wildcards (like file.expand does).
• file.compareFilesModified - compare two of these lists of files, outputting the differences as an object where each relevant file shows as added, deleted, or changed.

If used in a once-executed task, because there will be no cached list of modified files, it will just get all files. If used in a repeated task (via watch), the most recent list of modified files can be stored in a variable so that each time the task is run, it only handles the files modified since the last time.

[zonak]

That's an approach that could work.

Calling just the list of defined tasks could resolve my concern of running a sass task for example against "coffee" files. I do have couple of points of concern:

• a "coffee" file changes - grunt runs a compile task, then a lint task and ultimately a qunit task. If the execution of the first 2 tasks takes more than 250ms and in the mean time a "sass" file changes. By the time the qunit task runs we have the "sass" file change as the freshest and possibly have the qunit task executed against the "sass" files
• how would we keep the length of the list of modified files under control - if we are watching files for a longer period of time this list can become huge

[Takazudo]

+1 to this feature.

I also got a compiling time problem about coffeescript and sass, too.
It seemed that I needed to make sass compiling grunt and coffescript compiling grunt to solve it.

[cowboy]

@Takazudo I don't understand what you mean by this:

It seemed that I needed to make sass compiling grunt and coffescript compiling grunt to solve it.

Could you elaborate?

[Takazudo]

@cowboy

Oh, I'm sorry to explain too little. Here's whtat I mean.
https://github.com/Takazudo/test

When I code coffeescripts, I don't want to run sass compiles.
When I code sass, I don't want to run coffee compiles.
So, I created two grunt files and ran each individually.

[paulirish]

+1

I wonder if http://brunch.io/ could be used by the task for the watching and recompilation...

In my perfect world, brunch and grunt are compatible. cc @paulmillr

[cowboy]

I'm not sure what people are +1 ing. Per one of my previous comments:

Watch can create a list of files that has been added / deleted / changed while it's running, but it can't know what files (if any) were added / deleted / changed while other tasks were running.

If there's a possibility that a file might be modified while the watch task isn't watching, relying on a watch-generated file list might not be enough.

And given that, I proposed an alternative approach:

The watch task is really very simple. It just waits for files to change, then executes some arbitrary tasks. It's not at all smart.

A robust, generalized approach to this problem might be to write two methods such as these:

• file.filesModified - return a list of files and their last modified times, optionally handling wildcards (like file.expand does).
• file.compareFilesModified - compare two of these lists of files, outputting the differences as an object where each relevant file shows as added, deleted, or changed.

If used in a once-executed task, because there will be no cached list of modified files, it will just get all files. If used in a repeated task (via watch), the most recent list of modified files can be stored in a variable so that each time the task is run, it only handles the files modified since the last time.

But I haven't received much in the way of feedback. So, after all that, does anyone have any? Consider how you'd have to write tasks. Obviously you don't want writing tasks to be painful or awkward.. but the watch task only does so much.

[zonak]

There were couple of concerns expressed about how would your suggestion work and that is where the discussion kind of went stale.

I understand that the current implementation doesn't have all the features people are asking for but filling in for the watch features provided by 3-rd party tools like LiveReload and DevKit or for running the watch tasks provided by cs, sass, less and the likes in a bunch of terminals is what grunt would be great for.

In an ideal case grunt would be taking care of building, compiling, minifying, watching and even deploying code. That would be one of the biggest appeals or grunt - a one stop shop for all these tasks which is why people are eager to see these things implemented.

[paulmillr]

Oh hai.

Yep, @brunch can be freely used for watching, it already supports many plugins like jade, sass, styl, less, iced, hogan etc. And it has node.js api.

[Takazudo]

@cowboy

I tried to create the task "parallelwatch" by myself.
Though there may be a better way to do this, could you take a look at my code?

https://github.com/Takazudo/test/tree/gruntWatch
https://github.com/Takazudo/test/blob/gruntWatch/tasks/parallelwatch.js

This task was what I wanted to do.

[cowboy]

So, check 0cf795c out. It shows an updated watch task config for grunt v0.3.8 (now in npm). You can specify just files and tasks as before, but if you specify multiple targets with those properties, running grunt watch will watch all specified wildcards, running the appropriate tasks for any changed files. You can, of course, just watch only one target by specifying grunt watch:targetname.

It doesn't update watch to monitor file changes in-between watchings, and doesn't provide a list of changed files, but it should be helpful.

[paulmillr]

Still not sure why reinvent the wheel counting that I made beautiful 100% async non-blocking api for these things and already resolved the problems in brunch.

@cowboy why? I could help you on integrating the thing.

[cowboy]

@paulmillr patches welcome!

[paulirish]

@paulmillr yeah i'd love to see a nice way of tackling that integration. could we get your help?

[paulmillr]

@paulirish sure. I'll start working on this.

[Takazudo]

Multiple watch worked! Thanks so much!

[cowboy]

So, let's say that I'm watching two sets of files and when any files change, the appropriate task or tasks will be run. For example, given the watch configuration in grunt v0.3.8's grunt.js gruntfile which is basically this:

watch: {
  scripts: {
    files: ['grunt.js', 'lib/**/*.js', 'tasks/*.js', 'tasks/*/*.js', 'test/**/*.js'],
    tasks: 'lint test'
  },
  docs: {
    files: ['README.md', 'docs/*.md'],
    tasks: 'docs'
  }
}

Right now, the watch task will intelligently be able to determine which file has changed and run the correct tasks. For example, if lib/grunt.js is changed, the lint and test tasks will be run. If docs/api.md is changed, the docs task will be run. And if both of those files change within a few hundred ms of each other, all three tasks will be run. This behavior was easy to implement, and is already in the live grunt.

The problem arises when trying to provide a list of changed files that tasks can use. I could separate all files matching a set of wildcard patterns into target name-based buckets, eg. scripts and docs, so that if lib/grunt.js, test/grunt_test.js and docs/api.md were all changed together, an object would be created, maybe as grunt.task.watch, that might look something like this:

{
  scripts: ['lib/grunt.js', 'test/grunt_test.js'],
  docs: ['docs/api.md']
}

The problem here is that the lint and test tasks don't know that they were being run via a watch target named scripts and the docs task wouldn't necessarily know that the watch target it ran from was named docs, because watch target names are completely arbitrary. This means tasks wouldn't be able to easily figure out where their list of "changed files" existed.

So that approach doesn't work.

I could go a step further, and parse the tasks property for each target, creating an array of relevant modified files for each task-named property, like so (given the same three modified files):

{
  lint: ['lib/grunt.js', 'test/grunt_test.js'],
  test: ['lib/grunt.js', 'test/grunt_test.js'],
  docs: ['docs/api.md']
}

This could be a little more sane, if it wasn't for the fact that any task can internally run any other task. Or be an alias task, so what would happen if the tasks property referenced a default task which was an alias to lint test? It would get very complicated, very fast, and I'm not managing graphs of aliases and dependencies.

So that approach doesn't work either.

The only solution I can see would be to make available a single list of changed files, that might look something like this:

['lib/grunt.js', 'test/grunt_test.js', 'docs/api.md']

Which isn't great, because then all the changed files have been jumbled up together, into a single list. But it's definitely possible. Then it would be the responsibility for a task to check that changed files list, if it existed, to see what files it cared about (which I would provide an API method for).

Does anyone have any other suggestions as to how a task run via watch might be told how the files it cares about have changed, in a generic way? If not, do you have any suggestions on how I should consider approaching this problem?

How do you feel about my "single list of changed files" idea?

[zonak]

I like the last option.

Whoever needs to define a task that processes only changed files can easily get the relevant files by using:

grunt.utils._.intersection(files, changed_files)

where:

• files is the set of files defined for the task
• changed_files is whatever we receive from the watch task

Then we can decide what to do with the results - for example if the resulting array is empty do nothing or process all files defined for the task.

One thing that I wanted to ask is how are the processed files going to be cleared out - would that be the job of the task or some other mechanism would take care of this?

[cowboy]

@zonak every time the watch task runs again, it will reset the files list. Tasks should leave the list as-is so that other tasks can use the files list.

[cowboy]

Ok, so after talking to @zonak in IRC for a while we came up with this.

Inside a task, you've got a wildcard pattern or array of wildcard patterns. For this example, let's pretend we're in a multitask and have this.file.src to work with.

This is the basic "always handle all files" example. Whether the task with this code was run via the watch task or standalone, it would always process all files matching the specified patterns.

// Get all relevant files.
var files = grunt.file.expandFiles(this.file.src);

This would be an alternate (and new) signature for the same thing:

// Get all relevant files.
var files = grunt.file.expandFiles(this.file.src, {filter: 'current'});

If you're using watch, concerned about performance and want to handle only files changed since the last watch execution, you could do something like this (new signatures):

// Check for any files deleted since the last execution of the `watch` task.
// This will *only* match files when the task is run via the `watch` task.
var deletedFiles = grunt.file.expandFiles(this.file.src, {filter: 'deleted'});
// If there are any deleted files, delete their compiled counterparts.
if (deletedFiles.length > 0) {
  deleteCompiledFiles(deletedFiles);
}

// Check for any files changed or added since the last execution of the `watch`
// task. This will *only* match files when the task is run via the `watch` task.
var files = grunt.file.expandFiles(this.file.src, {filter: 'changed'});
// If there are any changed files (including new files) compile them.
if (files.length > 0) {
  compileChangedFiles(files);
} else if (deletedFiles.length === 0) {
  // Since there are no changed or deleted files, this task wasn't run via the
  // watch task. So just handle files normally.
  files = grunt.file.expandFiles(this.file.src, {filter: 'current'});
  compileChangedFiles(files);
}

Adding a changedOrCurrent filter could be a convenient time-saver for the last example. For example, this could be useful inside the lint task. If any relevant files have changed (watch), process just those files. Otherwise, if no relevant files have been deleted (not watch), process all files normally.

// Check for any files changed or added since the last execution of the `watch`
// task. If no files are matched, and no deleted files are found for the given
// wildcard patterns, match all files that would've been matched with 'current'.
var files = grunt.file.expandFiles(this.file.src, {filter: 'changedOrCurrent'});
// If there are any changed files (including new files) process them.
processFiles(files);

Are there ever going to be situations where the changed behavior will be preferred to the changedOrCurrent behavior? I can't think of any right now. It seems like I'd always want deleted, current or changedOrCurrent and not changed. Thoughts?

[cowboy]

Ok, ignore all the crazy stuff I said before.

I've added two things: a grunt.file.watchFiles object, and a grunt.file.match method. Those links point to the wip branch docs, so click them and read the docs.

Those two things should allow you do everything you want, without over-complicating the grunt.file.expandFiles method.

@zonak, can you review this in the wip branch and tell me what you think? I'd love to be able to close this issue as fixed.

[zonak]

Tried the mentioned additions in the wip branch and did some test tasks.

Everything I tried worked great. Did some tests with:

• creating new files
• renaming existing files
• deleting existing files
• changing existing files

All cases were detected and recorded in the file.watchFiles properties.

The grunt.file.match method also does what it is documented - a great helper necessary to figure out what changes should be handled by what task.

I think that these additions do provide all the necessary prerequisites for developing watch tasks answering to the most common scenarios or at least to the ones I had imagined.

Thank you Ben for all the hard work.

[cowboy]

Great. I'm closing this ticket.

@paulmillr I'd love to see how you integrate brunch to improve grunt's watch task. When you have something for me to look at, please open a new issue and we'll discuss it there. Thanks!