Watching directories #68

Closed
japj opened this Issue Jun 26, 2011 · 10 comments

Comments

Projects
None yet
4 participants
@japj
Contributor

japj commented Jun 26, 2011

On Windows, it seems ReadDirectoryChangesW could be used to watch directory changes in IOCP style
http://msdn.microsoft.com/en-us/library/aa365198(v=vs.85).aspx

As a result, libuv might be a logical place to support for fs.watchFile in a crossplatform way.

This relates to joyent/node#1171

@ry

This comment has been minimized.

Show comment Hide comment
@ry

ry Jun 26, 2011

Contributor

+1

I've been unimpressed with libev's ev_stat - which Node currently uses. It supports only inotify - and doesn't do directory watching. We should probably not use that on the Unixes either. @rmustacc also expressed intrest in this task.

Linux: inotify http://www.kernel.org/doc/man-pages/online/pages/man7/inotify.7.html

What's the optimal facility on Solaris, OSX, and FreeBSD?

Can someone suggest an API?

Contributor

ry commented Jun 26, 2011

+1

I've been unimpressed with libev's ev_stat - which Node currently uses. It supports only inotify - and doesn't do directory watching. We should probably not use that on the Unixes either. @rmustacc also expressed intrest in this task.

Linux: inotify http://www.kernel.org/doc/man-pages/online/pages/man7/inotify.7.html

What's the optimal facility on Solaris, OSX, and FreeBSD?

Can someone suggest an API?

@japj

This comment has been minimized.

Show comment Hide comment
@japj

japj Jun 26, 2011

Contributor

I am confused now, what exactly do you mean with "doesn't do directory watching"?

inotify supports "IN_CREATE - a file in a watched directory is created" and some other related events, so if you do fs.watchFile(directory) that should work I think (although maybe not they way you intended/expected)?

(and I think that is what @vincentcr is doing in the node issue 1171)

Contributor

japj commented Jun 26, 2011

I am confused now, what exactly do you mean with "doesn't do directory watching"?

inotify supports "IN_CREATE - a file in a watched directory is created" and some other related events, so if you do fs.watchFile(directory) that should work I think (although maybe not they way you intended/expected)?

(and I think that is what @vincentcr is doing in the node issue 1171)

@ry

This comment has been minimized.

Show comment Hide comment
@ry

ry Jun 26, 2011

Contributor

I meant ev_stat doesn't support recursive directory watching

Contributor

ry commented Jun 26, 2011

I meant ev_stat doesn't support recursive directory watching

@rmustacc

This comment has been minimized.

Show comment Hide comment
@rmustacc

rmustacc Jun 26, 2011

Owner

FreeBSD, Mac OS X, NetBSD, and OpenBSD use something called kqueue. SunOS uses Event Ports. While they are similar to inotify, they end up working differently. inotify ends up caching a lot of stuff in the kernel and thus can drop events. However, because of that it will tell you what dirent has changed. All the other APIs can tell you what event occurred in a directory, but they won't tell you what changed (i.e. you don't get the dirent name of what's changed). The benefit of this though is that you won't ever drop events and it minimizes races.

Nothing supports recursive directory watching; however, it wouldn't be too bad for someone to implement on top of this API. However, I would not propose that being something we natively support.

Owner

rmustacc commented Jun 26, 2011

FreeBSD, Mac OS X, NetBSD, and OpenBSD use something called kqueue. SunOS uses Event Ports. While they are similar to inotify, they end up working differently. inotify ends up caching a lot of stuff in the kernel and thus can drop events. However, because of that it will tell you what dirent has changed. All the other APIs can tell you what event occurred in a directory, but they won't tell you what changed (i.e. you don't get the dirent name of what's changed). The benefit of this though is that you won't ever drop events and it minimizes races.

Nothing supports recursive directory watching; however, it wouldn't be too bad for someone to implement on top of this API. However, I would not propose that being something we natively support.

@ry

This comment has been minimized.

Show comment Hide comment
@ry

ry Jun 26, 2011

Contributor

I've heard OSX has a different file notification mechanism which is preferred to kqueue. I can't find a link atm.

Contributor

ry commented Jun 26, 2011

I've heard OSX has a different file notification mechanism which is preferred to kqueue. I can't find a link atm.

@ry

This comment has been minimized.

Show comment Hide comment
@ry

ry Jun 26, 2011

Contributor

yes, thank you

Contributor

ry commented Jun 26, 2011

yes, thank you

@ry

This comment has been minimized.

Show comment Hide comment
@ry

ry Jun 26, 2011

Contributor

From Apple:

Choosing an Event Mechanism

File system events are intended to provide notification of changes with directory-level granularity. For most purposes, this is sufficient. In some cases, however, you may need to receive notifications with finer granularity. For example, you might need to monitor only changes made to a single file. For that purpose, the kernel queue (kqueue) notification system is more appropriate.

If you are monitoring a large hierarchy of content, you should use file system events instead, however, because kernel queues are somewhat more complex than kernel events, and can be more resource intensive because of the additional user-kernel communication involved.

Contributor

ry commented Jun 26, 2011

From Apple:

Choosing an Event Mechanism

File system events are intended to provide notification of changes with directory-level granularity. For most purposes, this is sufficient. In some cases, however, you may need to receive notifications with finer granularity. For example, you might need to monitor only changes made to a single file. For that purpose, the kernel queue (kqueue) notification system is more appropriate.

If you are monitoring a large hierarchy of content, you should use file system events instead, however, because kernel queues are somewhat more complex than kernel events, and can be more resource intensive because of the additional user-kernel communication involved.

@rmustacc

This comment has been minimized.

Show comment Hide comment
@rmustacc

rmustacc Jun 26, 2011

Owner

I have a lot of notes and API comparisons on the Unix side of thing and started sketching some implementations. Let me play around and solidify that there and then see what overlap there is with what Windows gives you. From taking a brief look at the mentioned function, it seems like Windows is doing what Linux is doing with inotify and gives you the name of the dirent that changed.

The kind of information that the consumer wants at a minimum is:

  • Which directory changed? (pathname)
  • What changes occurred? (flag of events)

The trick is going to be that certain backends like kqueue don't give you the name, so you have to keep track of it initially from the request. Those pieces are doable. Uniformly getting what file inside a directory was changed (if that was the event) is not worth the effort nor race conditions.

Owner

rmustacc commented Jun 26, 2011

I have a lot of notes and API comparisons on the Unix side of thing and started sketching some implementations. Let me play around and solidify that there and then see what overlap there is with what Windows gives you. From taking a brief look at the mentioned function, it seems like Windows is doing what Linux is doing with inotify and gives you the name of the dirent that changed.

The kind of information that the consumer wants at a minimum is:

  • Which directory changed? (pathname)
  • What changes occurred? (flag of events)

The trick is going to be that certain backends like kqueue don't give you the name, so you have to keep track of it initially from the request. Those pieces are doable. Uniformly getting what file inside a directory was changed (if that was the event) is not worth the effort nor race conditions.

@bnoordhuis

This comment has been minimized.

Show comment Hide comment
@bnoordhuis

bnoordhuis Mar 15, 2012

Contributor

Closing, we have a file watcher API now.

Contributor

bnoordhuis commented Mar 15, 2012

Closing, we have a file watcher API now.

@bnoordhuis bnoordhuis closed this Mar 15, 2012

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment