use polling for cache if available

[remusb]

Some remotes support polling for changes on them. As far as I can tell, only drive does this now.
cache can subscribe to those events and automatically expire its entries too.

Some issues I found already:

• moving a file doesn't seem to expire both parent folders, only the destination. At the moment this makes things a bit tricky as cache would best work if it doesn't expire its entries at all but it seems some cases aren't included in the notifications
• I noticed that the notification is sent to entries which are included in the global dircache that the drive remote uses to translate between a drive ID and a path in rclone. I haven't seen an issue yet but I'm wondering if there might a case where DirCache will not have an entry while it does exist in cache.

[zenjabba]

Any update for google drive on this?

[remusb]

@zenjabba not yet, I've mostly been working on the reliable upload enhancement which is tidying up just nicely to be released in the spare time I had.
This is next though so that should be good :)

Thanks for asking

[zenjabba]

Waiting on tender hooks for this update.

[remusb]

I got it running, just testing to make sure nothing breaks :)

[remusb]

Forgot to link the commit: b33e3f7

Though I do find that the first point still happens: moving a file doesn't seem to expire both parent folders, only the destination
I need to understand why though. It's either the drive remote filtering the notifications out or simply Google doesn't send in this way.

But that seems to be the only quirk. Apart from it, both cache and the vfs cache are told to expire their entries.

[remusb]

Link to beta: https://beta.rclone.org/v1.39-128-gb33e3f77/

[danielloader]

Is it enabled by default or do you need an experimental flag to enable it?

[remusb]

@Daniel-Loader it's enabled as long as the wrapped remote supports it too. In gdrive's case, it will.

[remusb]

To confirm, the message poll-interval is not supported by this remote shouldn't appear anymore at startup.

[zenjabba]

started testing today, will report back after a few days.

[danielloader]

Yeah ditto, testing alongside plexdrive for comparison, so far so good!

[zenjabba]

I am also calling this a win so far!

[remusb]

New beta for a related issue: https://beta.rclone.org/v1.39-130-g54724a13/
#2051

It's really about temporary uploading but the fix involves this new feature.

[zenjabba]

I'm assuming it's not a "critical" update, and I'll update at next quiet time... in the year 2051

[danielloader]

Quick question though, is this evicting a directory at a time if something changes, or the files?

What happens if a file is added to/removed from a directory of 20,000 files?

[remusb]

If you see a lot of unexpected eof during readings then it's likely the same cause as in the linked issue. That version should fix it. But I think it's a bit of an edge case when it happens so you might not encounter it at all.

@zenjabba

[remusb]

@jmoriau @Daniel-Loader

You are both correct. If there's a large folder then it's going to be a pain to evict each file. Sadly, the entire watch feature I'm using seems to be watching only for folders. There are several ways to improve on this:

• move the expiry in the background so that it doesn't block rclone completely. Problem is that notifications will be slower to pick up in this way for large folders
• maybe enhance the API in rclone to provide a different notification channel for remotes. By adding support for individual files, it should remove the need to expire all the folder when only 1 file is changed

I think both of them need to be implemented

[danielloader]

So with API driven cache eviction is there a need for the info age? If so should I remount it with a value much larger than 3 hours?

[remusb]

@Daniel-Loader Yep, I'm doing 24h for my personal drive. But I was also using that before, I don't write anything outside the mount.

Go higher, see how it works. If you encounter a scenario where something was changed on drive and it's not reflected after at least 1 minute in cache, let me know so that I can test it.

[ncw]

@remusb wrote:

maybe enhance the API in rclone to provide a different notification channel for remotes. By adding support for individual files, it should remove the need to expire all the folder when only 1 file is changed

That sounds like a good idea. The API was developed for the VFS cache so could do with expanding!

[edsimpsons83]

Experiencing the issue mentioned above (moving a file doesn't seem to expire both parent folders, only the destination) but other than that this is working great!

[jusher00]

I'm still having problems with the new polling feature for cache...

rclone v1.39-160-gc929de9dβ
- os/arch: linux/amd64
- go version: go1.9.3

/usr/bin/rclone-beta mount cache: /mnt/media/gdrive/ --cache-db-purge --allow-other --read-only --log-file ~/rclone.log

I use drive -> crypt -> cache.

I write to the remote from another machine, on a different network, with rclone move --delete-empty-src-dirs --drive-chunk-size 256M. The weird thing is, that some files show up on the cache remote and others don't. Oddly enough, it seems to matter which directory I move the files to. For some directories the polling works flawlessly, for others I haven't seen a single update. Specifically:

/mnt/media/gdrive/top_level1/New Dir (2018)/New File (2018).txt
New Dir (2018) and New File (2018).txt are uploaded with rclone move, but both not picked up by polling
I also experience the same problem in a top_level2 directory, with exactly the same dir/file structure as above

/mnt/media/gdrive/top_level3/Old Dir/New Dir/New File [2018].txt
New Dir and New File [2018].txt are uploaded with rclone move, both picked up by polling just fine

Maybe expiring a top level directory doesn't work correctly? Or maybe the parenthesis in a dir name cause trouble?

I'm running out of ideas otherwise.. It's a single rclone move command I issue for uploading, so all the files/directories should behave the same, shouldn't they?

On my other machine, where I just rclone mount the encrypted remote, without the cache, polling works fine for all uploaded files, however that is not --read-only.

Maybe somebody got an idea? Would be more than happy to do some troubleshooting but really don't know where to start :/

[neckaros]

@jusher00 i don't think it is the issue here but be careful as said in the doc having:
drive -> crypt -> cache
can lead to ban
drive -> cache -> crypt
is the way to go

[jusher00]

Yeah now I got fooled by my own naming convention haha, I wish drive -> crypt -> cache would work, but I actually have it set up just like you suggested, drive -> cache -> crypt.

[remusb]

With the recent ChangeNotify changes, there shouldn't be cases where we don't get notifications. I'm stopping further enhancements on this feature for 1.4.

If you have issues with the notification system feel free to open a new issue.