New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

use polling for cache if available #1937

Closed
remusb opened this Issue Dec 22, 2017 · 25 comments

Comments

Projects
None yet
7 participants
@remusb
Collaborator

remusb commented Dec 22, 2017

Some remotes support polling for changes on them. As far as I can tell, only drive does this now.
cache can subscribe to those events and automatically expire its entries too.

Some issues I found already:

  • moving a file doesn't seem to expire both parent folders, only the destination. At the moment this makes things a bit tricky as cache would best work if it doesn't expire its entries at all but it seems some cases aren't included in the notifications
  • I noticed that the notification is sent to entries which are included in the global dircache that the drive remote uses to translate between a drive ID and a path in rclone. I haven't seen an issue yet but I'm wondering if there might a case where DirCache will not have an entry while it does exist in cache.
@zenjabba

This comment has been minimized.

zenjabba commented Jan 7, 2018

Any update for google drive on this?

@ncw ncw changed the title from use pooling for cache if available to use polling for cache if available Jan 11, 2018

@remusb

This comment has been minimized.

Collaborator

remusb commented Jan 23, 2018

@zenjabba not yet, I've mostly been working on the reliable upload enhancement which is tidying up just nicely to be released in the spare time I had.
This is next though so that should be good :)

Thanks for asking

@zenjabba

This comment has been minimized.

zenjabba commented Feb 9, 2018

Waiting on tender hooks for this update.

@remusb

This comment has been minimized.

Collaborator

remusb commented Feb 9, 2018

I got it running, just testing to make sure nothing breaks :)

@remusb

This comment has been minimized.

Collaborator

remusb commented Feb 10, 2018

Forgot to link the commit: b33e3f7

Though I do find that the first point still happens: moving a file doesn't seem to expire both parent folders, only the destination
I need to understand why though. It's either the drive remote filtering the notifications out or simply Google doesn't send in this way.

But that seems to be the only quirk. Apart from it, both cache and the vfs cache are told to expire their entries.

@remusb

This comment has been minimized.

Collaborator

remusb commented Feb 11, 2018

@danielloader

This comment has been minimized.

Contributor

danielloader commented Feb 11, 2018

Is it enabled by default or do you need an experimental flag to enable it?

@remusb

This comment has been minimized.

Collaborator

remusb commented Feb 11, 2018

@daniel-loader it's enabled as long as the wrapped remote supports it too. In gdrive's case, it will.

@remusb

This comment has been minimized.

Collaborator

remusb commented Feb 11, 2018

To confirm, the message poll-interval is not supported by this remote shouldn't appear anymore at startup.

@zenjabba

This comment has been minimized.

zenjabba commented Feb 11, 2018

started testing today, will report back after a few days.

@danielloader

This comment has been minimized.

Contributor

danielloader commented Feb 11, 2018

Yeah ditto, testing alongside plexdrive for comparison, so far so good!

@zenjabba

This comment has been minimized.

zenjabba commented Feb 11, 2018

I am also calling this a win so far!

@remusb

This comment has been minimized.

Collaborator

remusb commented Feb 11, 2018

New beta for a related issue: https://beta.rclone.org/v1.39-130-g54724a13/
#2051

It's really about temporary uploading but the fix involves this new feature.

@zenjabba

This comment has been minimized.

zenjabba commented Feb 11, 2018

I'm assuming it's not a "critical" update, and I'll update at next quiet time... in the year 2051

@danielloader

This comment has been minimized.

Contributor

danielloader commented Feb 11, 2018

Quick question though, is this evicting a directory at a time if something changes, or the files?

What happens if a file is added to/removed from a directory of 20,000 files?

@remusb

This comment has been minimized.

Collaborator

remusb commented Feb 12, 2018

If you see a lot of unexpected eof during readings then it's likely the same cause as in the linked issue. That version should fix it. But I think it's a bit of an edge case when it happens so you might not encounter it at all.

@zenjabba

@remusb

This comment has been minimized.

Collaborator

remusb commented Feb 12, 2018

@jmoriau @daniel-loader

You are both correct. If there's a large folder then it's going to be a pain to evict each file. Sadly, the entire watch feature I'm using seems to be watching only for folders. There are several ways to improve on this:

  • move the expiry in the background so that it doesn't block rclone completely. Problem is that notifications will be slower to pick up in this way for large folders
  • maybe enhance the API in rclone to provide a different notification channel for remotes. By adding support for individual files, it should remove the need to expire all the folder when only 1 file is changed

I think both of them need to be implemented

@danielloader

This comment has been minimized.

Contributor

danielloader commented Feb 12, 2018

So with API driven cache eviction is there a need for the info age? If so should I remount it with a value much larger than 3 hours?

@remusb

This comment has been minimized.

Collaborator

remusb commented Feb 12, 2018

@daniel-loader Yep, I'm doing 24h for my personal drive. But I was also using that before, I don't write anything outside the mount.

Go higher, see how it works. If you encounter a scenario where something was changed on drive and it's not reflected after at least 1 minute in cache, let me know so that I can test it.

@ncw

This comment has been minimized.

Owner

ncw commented Feb 13, 2018

@remusb wrote:

maybe enhance the API in rclone to provide a different notification channel for remotes. By adding support for individual files, it should remove the need to expire all the folder when only 1 file is changed

That sounds like a good idea. The API was developed for the VFS cache so could do with expanding!

@edsimpsons83

This comment has been minimized.

edsimpsons83 commented Feb 19, 2018

Experiencing the issue mentioned above (moving a file doesn't seem to expire both parent folders, only the destination) but other than that this is working great!

@jusher00

This comment has been minimized.

jusher00 commented Feb 22, 2018

I'm still having problems with the new polling feature for cache...

rclone v1.39-160-gc929de9dβ
- os/arch: linux/amd64
- go version: go1.9.3
/usr/bin/rclone-beta mount cache: /mnt/media/gdrive/ --cache-db-purge --allow-other --read-only --log-file ~/rclone.log

I use drive -> crypt -> cache.

I write to the remote from another machine, on a different network, with rclone move --delete-empty-src-dirs --drive-chunk-size 256M. The weird thing is, that some files show up on the cache remote and others don't. Oddly enough, it seems to matter which directory I move the files to. For some directories the polling works flawlessly, for others I haven't seen a single update. Specifically:

/mnt/media/gdrive/top_level1/New Dir (2018)/New File (2018).txt
New Dir (2018) and New File (2018).txt are uploaded with rclone move, but both not picked up by polling
I also experience the same problem in a top_level2 directory, with exactly the same dir/file structure as above

/mnt/media/gdrive/top_level3/Old Dir/New Dir/New File [2018].txt
New Dir and New File [2018].txt are uploaded with rclone move, both picked up by polling just fine

Maybe expiring a top level directory doesn't work correctly? Or maybe the parenthesis in a dir name cause trouble?

I'm running out of ideas otherwise.. It's a single rclone move command I issue for uploading, so all the files/directories should behave the same, shouldn't they?

On my other machine, where I just rclone mount the encrypted remote, without the cache, polling works fine for all uploaded files, however that is not --read-only.

Maybe somebody got an idea? Would be more than happy to do some troubleshooting but really don't know where to start :/

@neckaros

This comment has been minimized.

neckaros commented Feb 22, 2018

@jusher00 i don't think it is the issue here but be careful as said in the doc having:
drive -> crypt -> cache
can lead to ban
drive -> cache -> crypt
is the way to go

@jusher00

This comment has been minimized.

jusher00 commented Feb 22, 2018

Yeah now I got fooled by my own naming convention haha, I wish drive -> crypt -> cache would work, but I actually have it set up just like you suggested, drive -> cache -> crypt.

@remusb remusb added this to the v1.40 milestone Mar 15, 2018

@remusb

This comment has been minimized.

Collaborator

remusb commented Mar 15, 2018

With the recent ChangeNotify changes, there shouldn't be cases where we don't get notifications. I'm stopping further enhancements on this feature for 1.4.

If you have issues with the notification system feel free to open a new issue.

@remusb remusb closed this Mar 15, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment