CREATE event fired only on delete. Detecting whether copying file has finished.

[mtvv]

Hi,

We seem to have a problem detecting large files being moved from a network share mounted on the host system to the shared folder. First we tried using the ATTRIB event to detect a new file arriving in the folder. The file is then moved to a different place for further processing.

Moving small files or moving large files from a folder on the same share as the shared folder seems to work fine, however when a large file is moved from a different share, it is actually copied and since that takes time, our script moves the file before it has finished writing.

Trying to investigate the problem, I noticed that the CREATE event is not fired at all, and strangely enough the CREATE event is actually fired only when deleting the file. The strange thing is that I see that the moved files are being detected as "added" in the output of vagrant fsnotify.

==> default: fsnotify: Added: File_from_host.MOV
==> default: fsnotify: Added: File_from_sshfs.MOV

I am testing on a system with MacOS as host and Ubuntu as guest.
The shared folder is mounted in guest using sshfs:

config.vm.synced_folder "shared", "/home/vagrant/shared", 
    type: "sshfs", 
    sshfs_opts_append: "-o nonempty -o cache=no -o uid=1000 -o gid=1000", 
    fsnotify: true 

The network share which serves as the origin of moved files is also mounted using sshfs.

In vagrant I tried monitoring for changes using inotifywait:

inotifywait --monitor --timefmt '%H:%M:%S' --format '%T %w %e %f' ./shared

Moving file from host to shared folder in Finder (same share, file is moved)

09:46:56 ./shared/ OPEN File_from_host.MOV
09:46:56 ./shared/ ATTRIB File_from_host.MOV
09:46:56 ./shared/ CLOSE_WRITE,CLOSE File_from_host.MOV

Moving file from network share (sshfs) in Finder (progress bar for 5 seconds, file is copied)

09:39:50 ./shared/ OPEN File_from_sshfs.MOV
09:39:50 ./shared/ ATTRIB File_from_sshfs.MOV
09:39:50 ./shared/ CLOSE_WRITE,CLOSE File_from_sshfs.MOV
09:39:55 ./shared/ OPEN File_from_sshfs.MOV
09:39:55 ./shared/ ATTRIB File_from_sshfs.MOV
09:39:55 ./shared/ CLOSE_WRITE,CLOSE File_from_sshfs.MOV

Renaming file in Finder:

10:30:28 ./shared/ OPEN Renamed_File.MOV
10:30:28 ./shared/ ATTRIB Renamed_File.MOV
10:30:28 ./shared/ CLOSE_WRITE,CLOSE Renamed_File.MOV
10:30:28 ./shared/ CREATE File_from_host.MOV
10:30:28 ./shared/ OPEN File_from_host.MOV
10:30:28 ./shared/ ATTRIB File_from_host.MOV
10:30:28 ./shared/ CLOSE_WRITE,CLOSE File_from_host.MOV
10:30:28 ./shared/ DELETE File_from_host.MOV

Deleting file from shared folder using Finder:

09:48:36 ./shared/ CREATE Renamed_File.MOV
09:48:36 ./shared/ OPEN Renamed_File.MOV
09:48:36 ./shared/ ATTRIB Renamed_File.MOV
09:48:36 ./shared/ CLOSE_WRITE,CLOSE Renamed_File.MOV
09:48:37 ./shared/ DELETE Renamed_File.MOV

Any other ideas on how to reliably detect a completed movement/copying process of a file would be most welcome!
Thanks in advance!

[adrienkohlbecker]

Hey @mtvv,

Thanks for opening an issue and sorry for the late reply.

This plugin works by listening to filesystem events on the host (which is why it correctly detects the file as added), and forwarding them to the VM using a combination of touch and rm over SSH.

In the creation case, it is only doing a touch on the VM, which is why you don't see the CREATE event forwarded. That's because the file already exists when the plugin executes. The deletion case is a bit weird, in order to forward the DELETE event, the plugin firsts re-create the file using touch, then immediately deletes it again using rm. You can see that in motion in your Finder renaming example, it touch-es the new name, then touch-es the old name, then removes the old name again.

You can see how forwarding an actual CREATE event wouldn't work: the plugin would need to have a way to first delete the file, then re-create it with the same contents in the VM.

Hope this helps! Let me know if you have further questions. I'm not sure how I can help with your use case without a way to map events 100% accurately.