New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Inotify for windows #1451

Closed
gopherbot opened this Issue Jan 28, 2011 · 22 comments

Comments

Projects
None yet
5 participants
@gopherbot

gopherbot commented Jan 28, 2011

by xavier.mehaut:

Hello,
I saw with interest the sub-package os.inotify ; unfortunatly, it is only for linux up
to now... Do you intend to make a windows port of this library... If so, you can check
the jnotify (http://jnotify.sourceforge.net/) project which offers the same capabilities
for linux, windows and macos...
best regards
Xavier
@robpike

This comment has been minimized.

Contributor

robpike commented Jan 29, 2011

Comment 1:

Owner changed to @alexbrainman.

Status changed to Accepted.

@alexbrainman

This comment has been minimized.

Member

alexbrainman commented Jan 29, 2011

Comment 2:

I have never used anything like that myself, and I don't know how it could work. But
quick search suggests ReadDirectoryChangesW could be a good fit here. This is very low
priority for me, but if you're interested to try and implement it yourself
http://golang.org/doc/contribute.html, everyone will help you along.
Alex

Labels changed: added os-windows.

Status changed to HelpWanted.

@gopherbot

This comment has been minimized.

gopherbot commented Jan 30, 2011

Comment 3 by xavier.mehaut:

i use personaly jnotify which provides also a wrapper for windows. All the source code
is available for calling the dll. Maybe it could help. 
I ll try to see next week if i ve time.
@alexbrainman

This comment has been minimized.

Member

alexbrainman commented Jan 31, 2011

Comment 4:

It uses ReadDirectoryChangesW alright. It also uses "completion ports" to monitor for
changes, but lucky us, we have goroutines, we could just block <g>.
@gopherbot

This comment has been minimized.

gopherbot commented Feb 14, 2011

Comment 5 by hectorchu:

My attempt at this can be found at http://golang.org/cl/4188047/.
If anybody can review and/or test it that would be great!
@gopherbot

This comment has been minimized.

gopherbot commented Feb 14, 2011

Comment 6 by xavier.mehaut:

Thanks a lot for this contribution ; I'll try to take time to test it soon ;
@gopherbot

This comment has been minimized.

gopherbot commented Jun 18, 2011

Comment 7 by crazy1be:

I've started on an API that should work with the subset of functionality provided by
both windows and linux, callback-based rather than channel-based. Currently only
supports linux, but making watchers for other operating systems should be trivial; they
need only satisfy the Watcher interface (see the README). Supports Modify, Delete, and
Create events currently. A Move event is planned, but not implemented for InotifyWatcher.
Let me know if you have any feedback or concerns about this API. Repository is at
https://github.com/crazy2be/fsmon.
@alexbrainman

This comment has been minimized.

Member

alexbrainman commented Jun 20, 2011

Comment 8:

I don't use this functionality myself, so it is hard for me to comment. But you're
welcome to start a discussion on golang-nuts, if you have some proposal in mind.
FYI, there is this change http://golang.org/cl/4188047/ that has not come to
anything yet.
Alex
@ancientlore

This comment has been minimized.

ancientlore commented Jun 23, 2011

Comment 9:

I am keen to try the changes in http://golang.org/cl/4188047/ if I can get my
Windows build working. What's the best way to move this toward release?
@alexbrainman

This comment has been minimized.

Member

alexbrainman commented Jun 23, 2011

Comment 10:

> I am keen to try the changes in http://golang.org/cl/4188047/ if I can get my
Windows build working. 
If you're asking how to build go for Windows, then you need mingw tools. You could
download them yourself: http://code.google.com/p/go-wiki/wiki/WindowsBuild, or you could
get pre-build zip file that Joe Poirier prepares
https://groups.google.com/d/topic/golang-nuts/u9h4TSs5mRk/discussion. Then you need to
fetch go repo with mercurial and build it.
Alternatively, I use linux/386 to develop and just cross-compile to windows/386.
> What's the best way to move this toward release?
As far as I remember, CL didn't go anywhere, because we couldn't decide on what the
interface should look like, so it is simple, reliable and works across of different OS
and OS-apis. Russ suggested we discuss that on golang-nuts. And that is where it
stopped. You could start discussion again if you have good suggestion on how to
implement it.
Alex
@gopherbot

This comment has been minimized.

gopherbot commented Jun 23, 2011

Comment 11 by crazy1be:

The difficulty here is that the interfaces provided by different operating systems
differ so much that making a single, coherent, interface is going to potentially result
in loosing a significant amount of functionality. In particular, we loose the advantages
each system provides, making implementing something that makes full use of the APIs
(e.g. a search indexing engine) quite difficult. However, that being said, we can easily
create an API that supports the most useful subset of functionality: watching arbitrary
directories for create, delete, and modify events.
The situation on Linux is probably the rosiest, which is part of the reason os/inotify
already exists. The inotify api provides events of almost every variety, for any
directory, in a reasonably coherent API. The go wrapper on this is pretty nice,
abstracting the low-level API surprisingly nicely.
On Windows, things are still pretty nice, if the function calls are a bit longer and
uglier :P. ReadDirectoryChangesW() appears to support the subset of functionality I
described above almost perfectly, and even has the additional feature of being able to
watch sub-directories natively, no additional directory-walking code required. I wasn't
able to figure out how you register to receive and then receive events, but It seems
possible that ReadDirectoryChangesW() returns all changes since the folder was opened or
it was last called.
On Mac OSX, the situation is UGLY. The API is completely and totally different from the
ones on Windows and Linux, and, it seems, designed for a completely different usage
case. It was implemented in order to allow the search indexer to keep up-to-date on file
changes, and was initially private to only that program. Since then, the FSEvents API
has been added, which allows external applications access to the same information that
the search indexer gets, filtered by top-level directory. This is recursive in nature,
and it is impossible to make it non-recursive (although you can obviously ignore events
from subfolders). Because you can ignore information, that is not a big issue. The
bigger issue is that this API does not provide you with any information on *how* a file
changed, telling you only that it changed. The jnotify library gets around this by
checking for changes itself, but this has obvious disadvantages in that you can only see
changes within the directory, and you're liable to miss events if there are multiple
events within a short period. Moves seem especially difficult to guess, i'm not sure how
jnotify gets around this.
All in all, I think that we should provide the low-level (syscall) and medium-level
(os/inotify) interfaces to each library, in case application developers want to take
advantage of this additional functionality on a particular platform. The wrapper library
should provide only basic functionality, like detecting modifications, moves, creations,
and deletions. I'm undecided whether we should support recursive watching, it certainly
has quite a few uses I can think of, and is supported directly on all OSes except Linux.
The interface for this library is tricky to decide on. I started on mine
(http://github.com/crazy2be/fsmon) with Callbacks, simply because that seemed like it
would work best for the way I was using the library. However, channels are also an
option, perhaps divided like this:
watcher := fsmon.NewWatcher(true)
watcher.AddWatch("foo")
str, str := <-watcher.Moves
err := <-watcher.Errors
str := <-watcher.Deletes
str := <-watcher.Creations
str := <-watcher.Modifications
Let me know of your ideas and thoughts on this, @michael.
@alexbrainman

This comment has been minimized.

Member

alexbrainman commented Jun 23, 2011

Comment 12:

This is all very interesting, but there are 7 people (who monitor this issue) that can
see your post. That's not enough. 
Please, post your suggestion on dolang-dev or golang-nuts. You should receive better
response there. I know nearly nothing about the technology. Might be able to help with
implementation. But we need to decide on api first.
Thank you.
Alex
@gopherbot

This comment has been minimized.

gopherbot commented Jun 23, 2011

Comment 13 by xavier.mehaut:

Hi
As, when I started an internal project, inotify for windows didn't existed,
I used instead JNotify which is portable and I compiled natively  the
project with excelsior jet. The result is efficient and simple to implement.
Maybe you could follow a same scheme...
@gopherbot

This comment has been minimized.

gopherbot commented Jun 23, 2011

Comment 14 by xavier.mehaut:

I personnaly think that recursivity is mandatory
@ancientlore

This comment has been minimized.

ancientlore commented Jun 23, 2011

Comment 15:

Thanks for all the feedback. We do need to move this to golang-nuts.
Alex, thanks for the links - I do have all that installed, just having some errors in
some tests. Hopefully will figure it out soon.
crazy2be, to receive events that a directory or subdirectory was modified in Windows you
use FindFirstChangeNotification/FindNextChangeNotification. It returns a handle that you
wait on using WaitForSingleObject or WaitForMultipleObjects. You still have to use
ReadDirectoryChanges to get the specific changes - these other functions just give your
thread a way to wake up when changes happen.
I personally like the initial inotify interface using one channel. Using a channel seems
more idiomatic for Go. Because you can filter the events, it seems odd to break them
into multiple channels - you'd have channels that never have messages. Plus, multiple
events could happen on the same file, and I am not 100% sure but I think the events can
be combined.
I'll start a topic on golang-nuts.
@gopherbot

This comment has been minimized.

gopherbot commented Aug 10, 2011

Comment 16 by scottmg@chromium.org:

I don't think ReadDirectoryChangesW is correct API to be using for this sort of general
facility. NTFS Change Journals are the preferred method and are used by the OS for other
services (eg. indexing).
@gopherbot

This comment has been minimized.

gopherbot commented Aug 10, 2011

Comment 17 by groby@chromium.org:

To amend what scot said above - ReadDirectoryChangesW is broken in that it misses
changes on large change volumes. NTFS change journals are the only reliable method on
Windows.
@gopherbot

This comment has been minimized.

gopherbot commented Aug 11, 2011

Comment 18 by crazy1be:

Interestingly enough, change journals (see
http://msdn.microsoft.com/en-us/library/aa363798(v=vs.85).aspx) are also quite alike
what is offered on MacOSX. Not sure if a similar alternative exists for Linux.
Ultimately, before deciding what the API will look like, we need to decide what
capabilities will be most important for people using the API, and what some of the
common usage scenarios are.
In light of this, I think it might help if interested parties make a brief statement of
their intended use of the library, and what features it would enable in their
application.
What do you want to use such a file change notification library for?
@gopherbot

This comment has been minimized.

gopherbot commented Aug 11, 2011

Comment 19 by crazy1be:

Link got mangled... Go here:
http://msdn.microsoft.com/en-us/library/aa363798(v=vs.85).aspx
@rsc

This comment has been minimized.

Contributor

rsc commented Oct 6, 2011

Comment 20:

Labels changed: added priority-low, removed priority-medium.

@alexbrainman

This comment has been minimized.

Member

alexbrainman commented Oct 15, 2011

Comment 21:

Just a note. There is some new activity on http://golang.org/cl/4188047/ and
http://golang.org/cl/5248062/. Please, comment if you are interested.
Alex
@gopherbot

This comment has been minimized.

gopherbot commented Oct 20, 2011

Comment 22 by hectorchu:

This issue was closed by
https://code.google.com/p/go/source/detail?r=04345e5969cf00ae17e0fa3d35a4116d086286a9.

Status changed to Fixed.

@golang golang locked and limited conversation to collaborators Jun 24, 2016

This issue was closed.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.