Watchdog on windows is not working properly.

[caetanus]

In windows, watchdog is leaving events in the wrong form. sometimes a directory is deleted, and watchdog tells a file file was deleted instead.
many events are not catch by watchdog.

i just made a filewatcher for windows using ctypes that watchdog could use as backend on windows:

https://github.com/caetanus/windows-file-changes-notify

if you appreciate and tells me how, i can put myself it as backend on watchdog, but feel free if you want.

thankyou

[Ormod]

I took a brief look at it and saw that the solution you tried for the "knowing if it was a dir to be deleted" was you keep the dir tree in memory. Though since those look like to be just paths that doesn't sound like it eats up that much memory on average. I wonder if there isn't any way to coax that info out of Windows itself?

As for missing events, could you outline which ones are not there, as in why is your approach guaranteed to have fewer of those? One way to get this into tree might be to fix whatever is causing missing files in the current approach, and maybe have the "is this event a dir delete or file delete" as an optional thing you can turn on for extra accuracy.

As for how it could be used from watchdog as a backend from a src code POV you'd need to implement the Observer class.

[caetanus]

Knowing if the object was a file or directory:

About the information about the Directory was deleted, I just spent a couple of weeks on msdn and viewing other implementations of micrsosoft in .NET, and windows will never tell you if the file object was a directory.
keeping the directory tree into memory will have some colateral effects: it may cause a race condition if you writing a bunch of directories in the moment you start looking at it, it may lost the internal tree state, but it will occur only in very rare cases.
maintaining the tree in memory just for directories in a dir that have 1 thousand subdirectories in a average of 32 unicode chars each, you will spent just 80kb of memory including the paths itself and the tree of sets.

behavior differences

the differences of watching directories in windows and other systems are:

in windows when you move the file between directories, windows will act as the file was deleted then recreated into another place (there is a workaround by checking the mactimes of the deleted file and the new created file.)

when you move a file from outside the repository to the repository, the file will act as a created.
when you move a file from the repository to outside of it, the windows will act as deleted.

about of the internal tree, there is another way using CreateShadowCopy, in this way, you can create instantly a shadow backup that DON`T spend more space in disk neither will fail because exclusive access to files, to check the type of files, but it is very a complicated thing to do.

in the tests, i take a several stress test to make sure that no event will be lost.

Observer Class

there is any documentation how to implement an observer class?

[tamland]

Rather than implementing another observer class the current one should be fixed. It's all the same windows api right.

The low level win api code is in winapi.py. The mapping of raw windows events into the cross platform ones happens in read_directory_changes.py. Specifically in queue_events, so everything related to fixing the behavior differences is in there (As you can see there are already numerous hacks).

Like you say there are several known issues with the windows code so patches are much appreciated.

[caetanus]

Ok, but in this case, at the time I wrote the backend, i was very
needing this working because of my job. and I got no time to learn the
watchdog architecture in that time and now the base of the backend is
done in my repository, what should we do?

Em 07-05-2013 10:21, takoi escreveu:

Rather than implementing another observer class the current one should
be fixed. It's all the same windows api right.

The low level win api code is in winapi.py
https://github.com/gorakhargosh/watchdog/blob/master/src/watchdog/observers/winapi.py.
The mapping of raw windows events into the cross platform ones happens
in read_directory_changes.py
https://github.com/gorakhargosh/watchdog/blob/master/src/watchdog/observers/read_directory_changes.py.
Specifically in |queue_events|, so everything related to fixing the
behavior differences is in there (As you can see there are already
numerous hacks).

Like you say there are several known issues
https://github.com/gorakhargosh/watchdog/issues?labels=windows&sort=updated&state=open
with the windows code so patches are much appreciated.

—
Reply to this email directly or view it on GitHub
#158 (comment).

[tamland]

What we should do? That will be up to you or whoever wants to. I don’t have any windows machine available and can't help with that. There's nobody currently responsible for the windows part if that's what you're asking. The original developer of it is long gone.

[caetanus]

I can mantain the windows backend, i have no machine with windows too, but i have VMs and I've been using win32api for more than 15 years.

[Ri0n]

looks like my problem is the same: just renamed subdirectory in watched directory and got FileMovedEvent with event.is_directory = False.

[thecyman]

I think we should make this two specific issues:
"WatchDog on Windows cannot determine if file is a folder" and possibly another for "WatchDog on Windows detects moves differently"
Otherwise, it makes it sound like WatchDog for Windows is "broken" in general, but it is not.

Because so far as I can tell, for both of these, this is more or less just the design of Windows:
It detects moves as "delete+created" for files that are outside of the folder. And it simply doesn't say whether it's a file or folder by design.
(In some cases you can use os.path.is_dir() but only if the file exists.)
I think the workaround mentioned by @caetanus for file moves is really smart! (To check the modification times of the deleted and created file to see if they match, although technically there could be a different file deleted at that time, in most use cases it will be sufficient.) Thanks! That is much more efficient then keeping a hash of every single file.

[Ri0n]

hm probably reading USN may be helpful. it seems usn records have some additional identification info

[romix]

Were some or all of the problems described in this issue fixed by #195?

[tamland]

No, this is about read_directory_changes Detecting moved have always worked. The only thing that doesn't is that moving from/to a watched directory produce create/delete. Actually, I would argue that this behavior is fine. It's more consistent with the other platforms, e.g. with polling it will never be possible to distinguished move to and create without scanning the entire file system.

[tamland]

#92