Added feature to poll files for help with virtual machines/docker

[carsongee]

We do a lot of our development in docker, and the events for changed files are generated in the host OS instead of inside docker, so ptw doesn't see that files have changed. Do you have any interest in adding a polling option, that will just scan the folder for modified file times and trigger tests? I may be able to write the code if you are.

[joeyespo]

Hi, @carsongee. Perhaps.

Actually, do you think this is a watchdog bug? If the files are indeed changing, watchdog should pick it up regardless if it's running in a container or not. Or am I misunderstanding?

[carsongee]

I just checked their repo, and found a very similar issue: gorakhargosh/watchdog#283. So it isn't really designed to handle events on remote file systems like we have with mounted container volumes since it is just using kernel events, so if you run watchdog in the container, since the kernel in the container isn't firing the event, the one in the host OS is, the watchdog can't react to it.

[joeyespo]

Good find!

Looks like watchdog provides a decent workaround with PollingObserver.

It'd be great it watchdog was able to detect whether inotify was reliable or not, so you wouldn't need an application-level concern. This is, in my opinion, a leaky abstraction that watchdog could potentially plug up. (Interested in opening a new issue asking for them to look into this? Docker is indeed growing in popularity.)

We could look at other libraries to see if they work inside of Docker out-of-the-box so pytest-watch users don't have to waste any time with this surprise. In the short-term, adding an option to use polling should be ok. I'd happily pull in a PR if you wanted to add a --poll option.

[carsongee]

I added the PR. As to reporting it upstream, are you thinking the issue would be to have the standard observer figure out if it needs to use polling? That would be pretty cool. I don't think there is any other way to address it in watchdog since the nature of remote filesystems is not to generate kernel events, that would have to be a fix in Mac and Linux kernels I think, but I'm a little out of my depth on this.

[joeyespo]

are you thinking the issue would be to have the standard observer figure out if it needs to use polling

Exactly. Then all tools that use watchdog would just work under this scenario instead of having to implement their own custom solutions.

I believe it's possible. For instance, one (bad) way would be to have watchdog create a .watchdog file and delete it right away in the directory it's observing. If no kernel events happen, turn on polling.

[carsongee]

gorakhargosh/watchdog#316

[joeyespo]

@carsongee Thank you!

[joeyespo]

FYI Released pytest-watch 3.0.0 today, which includes --poll.

Be sure to pip install --upgrade pytest-watch 😃