Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Polling fallback #9

Open
nathany opened this issue Jun 29, 2014 · 25 comments

Comments

Projects
None yet
10 participants
@nathany
Copy link
Member

commented Jun 29, 2014

Whether or not fsnotify implements polling support itself, some thought should be given into how polling could work as an alternative to native OS events.

GoConvey uses polling exclusively to avoid the "too many files" error #8.

"we walk the file system every quarter second and use the sum of the last mod time stamp and size of each go file as a quick comparison. Along the way we make note of new and deleted packages, all the while skipping 'ignored' packages. It's actually quite speedy." - @mdwhatcott, Gopher Slack

If you use an approach like what I've done in GoConvey, make sure to count empty directories as well... - @mdwhatcott

https://github.com/smartystreets/goconvey/blob/master/web/server/watcher/scanner.go

@pifantastic reports that node's gaze (https://github.com/shama/gaze#errors) library detects EMFILE errors and falls back to polling.

Polling could be opt-in for network file systems (NFS), Vagrant or Plan 9 -- where OS events don't work or are unavailable.

@mdwhatcott

This comment has been minimized.

Copy link

commented Oct 9, 2014

FYI, the link (above) to the goconvey scanner is no longer valid (I've recently rewritten that package. My approach, however, is unchanged:

https://github.com/smartystreets/goconvey/tree/master/web/server/watch

@omeid

This comment has been minimized.

Copy link

commented Jan 31, 2015

👍 for polling.

It is not ideal, but it is better than none and instead of everyone developing it on their own, it makes sense to have one that is made better by many people.

In terms of the opt-in option, fsnotify could have an option to "register" different "watch providers", in the spirit of sql package. This would also allow people to develop their own providers for other services, say Dropbox, s3, and so forth, but maybe that is jumping ahead of ourselves.

@nathany

This comment has been minimized.

Copy link
Member Author

commented Jan 31, 2015

I've never even thought of extending it to webhook notifications like Dropbox and Google Drive provide. Interesting idea. Not sure if it belongs in this package, but having a common interface in top could be cool.

The sql driver model is one I've been thinking about, particularly if there are multiple options for a given OS.

@omeid

This comment has been minimized.

Copy link

commented Jan 31, 2015

Yeah, it really depends how you look at it, I wouldn't consider them as general webhooks as you can think of dropbox or s3 just another type of filesystem.

Regardless of that, having a watch-driver interface and different driver is a better design IMHO.

@nathany

This comment has been minimized.

Copy link
Member Author

commented Jan 31, 2015

Indeed. It also fits with another desire I have, which is to make it easier to contribute to without being knowledgable of every single platform. Separate drivers for inotify, kqueue, etc. would be one way to achieve that.

@szechyjs

This comment has been minimized.

Copy link

commented Jun 24, 2015

👍 polling would be great. I'm trying to migrate a python application from watchdog to fsnotify. Unfortunately I think this is the roadblock, as the filesystem that needs monitored is on NFS and polling is the only option.

@nathany

This comment has been minimized.

Copy link
Member Author

commented Jun 26, 2015

Yah, polling is the only option for NFS as far as I know. You can also check https://github.com/rjeczalik/notify but I don't think it has polling yet either.

@calavera

This comment has been minimized.

Copy link

commented Dec 11, 2015

You might be interested in checking how docker support fsnotify and polling at the same time. We have a compatible interface and a fallback initialization for when fsnotify is not supported:

https://github.com/docker/docker/blob/master/pkg/filenotify/filenotify.go

maybe @cpuguy83 would be interested in moving the polling here so we can maintain only one package.

@cpuguy83

This comment has been minimized.

Copy link
Contributor

commented Dec 11, 2015

That would be great.
The reason I did not submit here is it seemed rather unknown if polling support was desired and we really needed to implement it.

@nathany

This comment has been minimized.

Copy link
Member Author

commented Dec 11, 2015

Please do submit a pull request.

@omeid

This comment has been minimized.

Copy link

commented Dec 13, 2015

@nathany Any plans for the v2 with driver interfaces? I will be happy to help out.

@nathany

This comment has been minimized.

Copy link
Member Author

commented Dec 14, 2015

@omeid I don't have any plans for what a driver interface would look like yet (and to be honest, I haven't done much work on fsnotify lately).

Personally, I'd prefer to see the current code base cleaned up before doing a big API change. The Windows internals are particularity crufty.
https://github.com/go-fsnotify/fsnotify/milestones/v2%20Internals

Maybe we could start a new issue to figure out the details of transitioning to a driver model?

@cpuguy83

This comment has been minimized.

Copy link
Contributor

commented Dec 14, 2015

@nathany Can you clarify what you'd like to see?

@nathany

This comment has been minimized.

Copy link
Member Author

commented Dec 16, 2015

@cpuguy83 Do you think you would be able to add polling without changing the API? Maybe just for operating systems that fsnotify doesn't currently support?

There are other situations where polling would be desirable, but I'm not sure how to detect them, or if it should be done more manually (which is why a driver-style API is relevant to this discussion).

@nathany

This comment has been minimized.

Copy link
Member Author

commented Dec 16, 2015

In terms of API, here are related issues #104 and #75.

Polling should also help with #45 for Windows users.

@nathany nathany added this to the v2 Internals milestone Dec 16, 2015

@nathany

This comment has been minimized.

Copy link
Member Author

commented Oct 2, 2016

@mdwhatcott @cpuguy83 Would you be willing to build a stand-alone fsnotify/polling package that could be incorporated into fsnotify as a secondary step?

If so, I'll create a repo called polling or poller or whatever you prefer.

For me the key considerations are:

  • providing a speedy implementation as outlined in the original post
  • a simple and idiomatic Go API specifically for polling
  • solid unit and/or integration tests (preferably without any external testing dependencies to keep it simple)
  • well documented both in the README, examples, and API docs
@mdwhatcott

This comment has been minimized.

Copy link

commented Oct 2, 2016

@nathany - Sounds like a fun project but I can't commit to it at this time.

@nathany

This comment has been minimized.

Copy link
Member Author

commented Oct 3, 2016

ok.

Here is a polling watcher by @radovskyb https://github.com/radovskyb/watcher

@nathany

This comment has been minimized.

Copy link
Member Author

commented Oct 18, 2016

@radovskyb Hey Benjamin, Would you be interested in transferring watcher into the fsnotify organization and working on it here? Still as a stand-alone repository for the time being.

The API already looks pretty close to fsnotify.

Once some of the low-level bits are extracted from fsnotify (e.g. #173) I'd like to incorporate polling into fsnotify as a fallback, while still allowing people to use the poller directly if that's all they want.

@radovskyb

This comment has been minimized.

Copy link

commented Oct 18, 2016

Sounds like a good idea. We can talk about it on the fsnotify Slack channel :)

@syntaqx

This comment has been minimized.

Copy link

commented Feb 3, 2019

+1 for having the ability to both fallback to, and explicitly require polling.

Docker for Windows running inside of Hyper-V host mount shared volumes using Samba/CIFS. Unfortunately, the CIFS implementation in the linux kernel doesn't support inotify events. There are some workarounds, but allowing polling would be preferred.

@cpuguy83

This comment has been minimized.

Copy link
Contributor

commented Feb 4, 2019

@syntaqx See how we handle fallback here and here

Basically just wrapped fsnotify to make it comply with this interface:

type FileWatcher interface {
	Events() <-chan fsnotify.Event
	Errors() <-chan error
	Add(name string) error
	Remove(name string) error
	Close() error
}

And then built a polling implementation.

@syntaqx

This comment has been minimized.

Copy link

commented Feb 4, 2019

@cpuguy83 Thanks for the information! I actually brought that exact package in as a dependency last night, but I'd hope we could still eventually get the functionality built into the fsnotify/fsnotify package. I lost far too much of my life trying to understand why things weren't working :P

@dtaniwaki

This comment has been minimized.

Copy link

commented Feb 25, 2019

I really wanted to use this feature, but since I can't find any stable one in public, I implemented an extensible polling logic for HDFS which implements Walk method in the HDFS gateway of argo-events by myself.

@congliu01

This comment has been minimized.

Copy link

commented Mar 28, 2019

Is someone working on this now or is there any plan? If not, I am interested in contributing to this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.