Event Processing Pipeline (WIP) #65

nathany · 2013-09-11T03:57:53Z

This introduces a pipeline for processing events before forwarding them to the Event channel.

The trigger pipeline step is used to filter create, modify, delete, rename events. It's logic is an extraction of the code that was in purgeEvents().
The hidden pipeline step filters hidden files and directories
The pattern pipeline step filters based on shell file pattern, eg. ".go,.c"
The throttle pipeline step filters trailing events on the same file
The logging step logs all new events (and the verbose flag is used elsewhere)

Initially I avoided changing the exported API, but since I began adding additional pipeline steps, I've started to implement changes as originally proposed in #64.

The tests pass on OS X, Linux, BSD and Windows and I have been testing it from Looper as well. I have seen some intermittent failures in TestFsnotifySubDir on OS X (#67).

Update usage information in README and example_test (GoDoc).
how to handle fallbacks when pipeline (was fsnFlags) was not found in the Watcher structure on Linux/BSD

Event

add IsDir() to Event interface/FileEvent implementation (do a separate PR) NEXT

Recursive

The auto watch pipeline step needs to watch a folder with the same pipeline/options
Need to handle errors from the auto watch (put on the Error channel?)
Remove watch recursively (the externalWatches addition to kqueue may help here?)
Tests

Hidden

does Windows need special handling for filtering hidden files/folders
handle hidden edge case, request a watch on a hidden folder with Hidden:false
make hidden option more clear (Keep/IgnoreHidden?)

Throttle

note behaviour and configuration may change to match FSEvents adapter in the future [Feature Request] OSX / FSEvents #54

Verbose

determine more specifically what it should/shouldn't log
should it work with other logging packages? can we add that later easily enough? (just more Options)

API

should the notifier interface be public and what should it be called? Either way the Event channel must remain a type of FileEvent to avoid breaking changes.
continue discussion of public API Proposed API Changes #64 before merging to master? come back to channels vs. iterators and other API changes as part of the move to os/fsnotify.

Tests

Keep current integration tests for previous API and write up new ones to test against new APIs.

nathany · 2013-09-11T04:00:23Z

fsnotify_bsd.go

 	wmut          sync.Mutex          // Protects access to watches.
-	fsnFlags      map[string]uint32   // Map of watched files to flags used for filter
-	fsnmut        sync.Mutex          // Protects access to fsnFlags.
+	pipelines     map[string]pipeline // Map of watched files to pipelines used for filtering


I was learning towards map[string]*pipeline here to save memory when inheriting from the parent directory, but I never figured out the locking/copying that would be necessary for processEvent(). Any suggestions or is this good enough for now?

I'm really not that well versed in Mutexes and pointers.

keep integration tests as-is to prove the deprecated APIs still function

nathany · 2013-09-21T18:54:54Z

fsnotify_bsd.go

 		} else {
-			w.fsnFlags[filePath] = FSN_ALL
+			w.pipelines[filePath] = pipeline{}


not entirely sure what to do with these fallbacks in BSD and Linux, especially as more steps are added to the pipeline

@remogatto

based on Andrea Fazzi's prettyautotest @remogatto https://github.com/remogatto/prettytest/blob/master/pta/prettyautotest.go

might want some Windows-specific tests

use notes for deprecations http://godoc.org/go/doc#Note

finding i need to do "vagrant up --provision" instead of just vagrant up

would like tags to match. semantic versions might be nice :-)

should not have been exported #64 (comment)

nathany · 2013-10-24T16:57:36Z

@howeyc Have you had a chance to review this?

I'll plan on implementing a recursive watcher and autowatch pipeline step this weekend.

howeyc · 2013-10-25T15:57:34Z

Yes I have and I like it so far. What do you think about making your "notifier" interface public adding some kind of NextEvent() function?

Something like this: https://gist.github.com/howeyc/7156865

nathany · 2013-10-25T16:08:45Z

Looks good to me.

I think my big challenge for the recursive watcher will be ensuring all the options for the pipeline steps are available to all the folders.

I'd really like to tease apart the code for fsnotify and the inotify/kqueue/etc. adapters, but I'd rather do a big internal refactoring as a separate pull request.

Tracks external watches Conflicts: fsnotify_bsd.go

tried Notifier/notification, but it's just so long... and mostly inconsistent. surprisingly this doesn't conflict with the Event channel in a Watcher

* autowatch step needs to add watches * remove watches recursively

passing the pipeline instead of options

Conflicts: CHANGELOG.md

Conflicts: AUTHORS CHANGELOG.md

nathany · 2013-11-30T08:51:08Z

Sorry this has been taking me so long. I'm going to try to get recursive watching wrapped up ~~this weekend~~ and replace out the custom one I built for Looper to get some "real world" use.

If anyone else wants to try out this code from a branch, just add a remote for git@github.com:gophertown/fsnotify.git to your local copy of howeyc/fsnotify and fetch the pipeline branch. See "for other team members" in contributing for details.

Obviously it's going to take a bit longer to complete... with the number of unchecked boxes and my current pace.

It would be great to have some others try out the code and API changes on a branch before this gets merged to master. /cc @robfig

failing test for autoWatch

(failing)

nathany · 2014-01-07T10:03:51Z

integration_test.go

+		t.Fatalf("failed to create subdir in test directory: %s", err)
+	}
+
+	time.Sleep(50 * time.Millisecond) // give system time to detect folder before creating file


Really this delay shouldn't be necessary. I wonder if this is an example of a race condition that minux mentioned (events out of order or the file event goes missing?).

https://codereview.appspot.com/48310043/ # 16

nathany · 2014-01-11T09:33:44Z

minux suggested a user-defined function to replace Pattern and Hidden. It's essentially a func filter(path string) bool. It's more flexible. Want to exclude files based on a pattern? Sure thing. Obviously it moves a fair bit of code (with lovely tests) into the user code, but it should be easy enough to write a small wrapper with the same API.

What about adding multiple filters? These filters are slightly different than the Event pipeline:

Run when adding a watch to skip folders/files that don't need to be watched (subdirectories, sendDirectoryChangeEvents)
Run as part of the pipeline on the Event.Path().

Also, heard the suggestion of making the Event interface smaller.

nathany · 2014-01-16T05:02:49Z

It looks like these features won't be in os/fsnotify, but development could be moved to a wrapper library.

https://github.com/libgo/fsnotify_ext (not up-to-date with the changes here)

nathany · 2014-05-17T19:58:16Z

I'm not sure if these ideas will become part of os/fsnotify, but perhaps I'll build a library that does encompass these ideas/code.

sending done would close w.kq before Remove had a chance to remove the watches with EV_DELETE, resulting in a file handle leak. ref howeyc#59 also make Close() report the first error returned by Remove and continue. closes howeyc#65

nathany added 2 commits September 10, 2013 20:10

skip integration tests in -short mode

1b0fe62

should fetch from fork

95f4bc1

nathany reviewed Sep 11, 2013
View reviewed changes

trigger processing pipeline

f88b2a8

nathany mentioned this pull request Sep 21, 2013

[Feature Request] OSX / FSEvents #54

Open

nathany added 2 commits September 21, 2013 10:21

newPipeline selectively enables pipeline steps

eea7078

introduce Triggers type, a newPipeline option

ac4b907

keep integration tests as-is to prove the deprecated APIs still function

nathany reviewed Sep 21, 2013
View reviewed changes

nathany added 3 commits September 21, 2013 13:49

add WatchPath, Options and a Hidden file filter

03b497f

file name pattern pipeline step

158eb58

a simple throttling filter

fe9c957

based on Andrea Fazzi's prettyautotest @remogatto https://github.com/remogatto/prettytest/blob/master/pta/prettyautotest.go

This was referenced Sep 21, 2013

Double modify events on Mac OS X #62

Closed

Tests run twice nathany/looper#6

Open

nathany added 9 commits September 21, 2013 17:35

Verbose option logs all new events

edb9d9d

handle relative file paths

150925e

hidden file checking for Windows

374279f

might want some Windows-specific tests

godoc

fc1722a

use notes for deprecations http://godoc.org/go/doc#Note

use 64-bit vagrant boxes for the race detector (at least for linux)

dd592a9

finding i need to do "vagrant up --provision" instead of just vagrant up

update AUTHORS

84f1118

start a CHANGELOG #66

cd4a1f2

would like tags to match. semantic versions might be nice :-)

handy project file for sublime text users

bba2306

remove FD_SET and friends

1766670

should not have been exported #64 (comment)

nathany added 3 commits October 27, 2013 12:56

Merge remote-tracking branch 'origin/master' into pipeline

14aaf6a

Tracks external watches Conflicts: fsnotify_bsd.go

documentation updates

72cb051

Event as an exported interface

7524685

tried Notifier/notification, but it's just so long... and mostly inconsistent. surprisingly this doesn't conflict with the Event channel in a Watcher

nathany added 3 commits October 27, 2013 17:07

unfinished recursive watcher

2ebc386

* autowatch step needs to add watches * remove watches recursively

a step closer to autoWatch

05aeb7f

passing the pipeline instead of options

Merge branch 'master' into pipeline

0b9aa69

Conflicts: CHANGELOG.md

nathany mentioned this pull request Nov 30, 2013

Proposed API Changes #64

Closed

Merge branch 'master' into pipeline

bcbf494

Conflicts: AUTHORS CHANGELOG.md

nathany mentioned this pull request Nov 30, 2013

Milestones #78

Closed

9 tasks

nathany added 6 commits January 3, 2014 19:35

copyright headers

171ba26

spelling mistakes

5d97943

gofmt

142233c

TestFsnotifyRecursive

463d45e

failing test for autoWatch

autowatch pipeline mock/test

bfc2020

(failing)

IsDir() for BSD and pipeline tests passing

f2614ea

nathany reviewed Jan 7, 2014
View reviewed changes

nathany closed this Jan 16, 2014

nathany reopened this May 17, 2014

nathany mentioned this pull request Jun 29, 2014

Additional filters fsnotify/fsnotify#10

Open

nathany closed this Jun 29, 2014

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Event Processing Pipeline (WIP) #65

Event Processing Pipeline (WIP) #65

nathany commented Sep 11, 2013

nathany Sep 11, 2013

nathany Sep 22, 2013

nathany Sep 21, 2013

nathany commented Oct 24, 2013

howeyc commented Oct 25, 2013

nathany commented Oct 25, 2013

nathany commented Nov 30, 2013

nathany Jan 7, 2014

nathany commented Jan 11, 2014

nathany commented Jan 16, 2014

nathany commented May 17, 2014

Event Processing Pipeline (WIP) #65

Event Processing Pipeline (WIP) #65

Conversation

nathany commented Sep 11, 2013

Event

Recursive

Hidden

Throttle

Verbose

API

Tests

nathany Sep 11, 2013

Choose a reason for hiding this comment

nathany Sep 22, 2013

Choose a reason for hiding this comment

nathany Sep 21, 2013

Choose a reason for hiding this comment

nathany commented Oct 24, 2013

howeyc commented Oct 25, 2013

nathany commented Oct 25, 2013

nathany commented Nov 30, 2013

nathany Jan 7, 2014

Choose a reason for hiding this comment

nathany commented Jan 11, 2014

nathany commented Jan 16, 2014

nathany commented May 17, 2014