Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature request: Listen filesystem events and sync #249

Open
dbcm opened this issue Dec 11, 2015 · 28 comments

Comments

Projects
None yet
@dbcm
Copy link

commented Dec 11, 2015

It would be nice to have rclone in daemon mode, listening some paths and when something change, rclone sync it.

@ncw ncw added the enhancement label Dec 11, 2015

@ncw

This comment has been minimized.

Copy link
Owner

commented Dec 11, 2015

A nice idea. Could use this library: https://github.com/fsnotify/fsnotify

Maybe a user interface I'd choose is something like this

rclone fsnotify sync/copy/move /source/path remote:test

What this would do is run the sync/copy/move then wait for an fsnotify event and sync/copy/move the files that changed.

Maybe users would not want the initial sync/copy/move - not sure.

@dbcm

This comment has been minimized.

Copy link
Author

commented Dec 11, 2015

\o/

@dibu28

This comment has been minimized.

Copy link
Collaborator

commented Dec 11, 2015

+1

@zeshanb

This comment has been minimized.

Copy link

commented Dec 15, 2015

Having doubts this CloudSyncHelper is what it's claiming to be:
https://github.com/XElementSoftware/CloudSyncHelper/blob/master/deployment/CloudSyncHelper.xml

On Tue, Dec 15, 2015 at 8:13 AM, timofonic notifications@github.com wrote:

I have an issue with using Google Drive over Ext4 under Windows. I use
dual and hate it, but I have no option because of homework.

Here's the relevant issue over Ext4 by using ext2fsd

:

https://sourceforge.net/p/ext2fsd/discussion/143329/thread/37c888a1/?limit=25
[image: googledrivebigeyecantbelied]
https://cloud.githubusercontent.com/assets/49494/11811379/7c17e3aa-a335-11e5-8925-7408418abe44.PNG
[image: googledrivehardlinktoext4directory]
https://cloud.githubusercontent.com/assets/49494/11811380/7c1d0a06-a335-11e5-8784-e3284d5fe3ef.PNG
[image: google-drive-wants-ntfs]
https://cloud.githubusercontent.com/assets/49494/11811381/7c25b8a4-a335-11e5-8923-cadd1785dd6a.PNG

I found CloudSyncHelper might be useful, but I'm unable to understand how
to install it. I submitted an issue:
XElementDev/CSH#6
XElementDev/CSH#6


Reply to this email directly or view it on GitHub
#249 (comment).

@danzig666

This comment has been minimized.

Copy link

commented Dec 16, 2015

You can use odrive for that: https://www.odrive.com/

@dibu28

This comment has been minimized.

Copy link
Collaborator

commented Dec 25, 2015

+1

@ncw

This comment has been minimized.

Copy link
Owner

commented Jan 2, 2016

@dibu28 are you interested in working on this?

@mlanner

This comment has been minimized.

Copy link

commented Jan 2, 2016

I don't know much about Go, but I did find a Go wrapper for inotify here:

https://godoc.org/golang.org/x/exp/inotify

Only for Linux it seems but might be able to be extended.

Inotify is very simple and tends to work well. Maybe inotify's features would be somewhat easy to integrate?

@danzig666

This comment has been minimized.

Copy link

commented Jan 2, 2016

I'd rather use this one, it's cross platform:
https://github.com/go-fsnotify/fsnotify
We probably need to keep a local database of remote files too.

@mlanner

This comment has been minimized.

Copy link

commented Jan 2, 2016

Yes, it seems better. Kind of where this thread started anyway. :)

@ncw ncw added this to the Unplanned milestone Feb 10, 2016

@klauspost

This comment has been minimized.

Copy link
Collaborator

commented Feb 18, 2016

@timofonic: Please stop adding "bump" messages. This is a new feature, and there are a lot of those.

This is not as simple as adding two lines of code, this is complex and will require a lot of work to function properly. I have done file monitoring, and there are a lot of cases you need to consider. This is also why a lot of the current applications offering this feature break all the time.

This is a free time project, and @ncw (and the rest of us) are free to chose what we work on. In my personal opinion there are a lot of more important features to work on. Look at the amount of current "support" issues - if we add another "fragile" feature that will not be helped.

@protonmesh

This comment has been minimized.

Copy link

commented Sep 26, 2016

Until this is available, thinking of running an rclone cron job. I am using rclone to backup files to a remote server, so running once daily is sufficient for me.

My only concern is for the rare scenario whether the previous rclone job has not finished while a new one begins. Tested this scenario and the remote had a few duplicate files uploaded from both jobs. Need to enforce mutual exclusion on the jobs, either by detecting running rclone job (through pgrep) and then do a delayed retry of the new job, or by having the jobs use a lock file for mutual exclusion.

Anyone else handle this a different way?

@resipsa2917

This comment has been minimized.

Copy link

commented Oct 15, 2016

How about a Bash daemon which uses inotify and then spawns rclone? The Bash scripts can be found at https://github.com/resipsa2917/rcloned

@bobbintb

This comment has been minimized.

Copy link

commented Nov 3, 2016

@protonmesh
Use flock.

@rhummelmose

This comment has been minimized.

Copy link

commented Jan 10, 2017

I wrote this small Bash service that may be of interest: https://github.com/rhummelmose/rclonesyncservice

@ncw

This comment has been minimized.

Copy link
Owner

commented Jan 11, 2017

@rhummelmose very nice. It is about time I made a third party tools page which I could link that from... I made an issue to remind myself to do it #1019

@bobbintb

This comment has been minimized.

Copy link

commented Jan 12, 2017

There is also this, if anyone is interested:
https://forum.rclone.org/t/sync-daemon-for-rclone/252

@zeshanb

This comment has been minimized.

Copy link

commented Jul 14, 2017

@ncw

This comment has been minimized.

Copy link
Owner

commented Jul 14, 2017

I think it is a nice idea. I haven't had time to work on it though, so If anyone else has, I'd be happy to guide them.

@Redrield

This comment has been minimized.

Copy link

commented Sep 12, 2017

I can share a temporary solution for this I've rolled using incron, and a small shell script, if anyone wants it.

@Vamoss

This comment has been minimized.

Copy link

commented Oct 25, 2017

Hi @Redrield , could you share you shell script? I want to have a try.

Cheers!

@Redrield

This comment has been minimized.

Copy link

commented Oct 25, 2017

@Vamoss I've actually converted it to a python script, but here you go https://gist.github.com/Redrield/81dd7c0cebd5760c65d1127495e487f8
You're gonna need to install pyinotify (and inotify for the actual OS)

@vrossum

This comment has been minimized.

Copy link

commented Mar 7, 2018

This is what I do on linux.
The script runs rclone whenever a file changes, but also every 300secs (timeout)
to make sure to update when remote files have changed.
The sleep helped against too frequent updates.

#!/bin/bash
while [[ true ]] ; do
rclone sync ....
while inotifywait --recursive --timeout 300 -e modify,delete,create,move ~/XXX ; do
rclone sync ....
echo sleeping
sleep 5
done
done

@untoreh

This comment has been minimized.

Copy link

commented Jun 4, 2018

rclone serve syncthing would be something interesting although as already said probably out of scope.

Of course you can always have a local copy of the folder with syncthing on it and rclone mount + another syncthing instance in a different path or remote, but at that point it's better to just use a more robust fs implementation since rclone mount it still experimental

@Saren-Arterius

This comment has been minimized.

Copy link

commented Aug 11, 2018

Hi, Dropbox will be only supporting unencrypted ext4 on Nov 7, the syncing will be artificially made not to work for other filesystems, including btrfs which is the one I am using.

If rclone supports daemon like syncing mode, I will be very happy that I can easily escape from Dropbox.

@Hoeze

This comment has been minimized.

Copy link

commented Aug 19, 2018

+1 for rclone serve syncthing
It would be damn awesome if rclone would integrate with syncthing!

@SysLabIT

This comment has been minimized.

Copy link

commented Sep 28, 2018

#!/bin/bash

while [[ true ]] ; do

	# performs synchronizations / copy
	rclone sync ....

	# waiting for something to change or it will pass 300 seconds
	inotifywait --recursive --timeout 300 -e modify,delete,create,move   SRC_DIR

	# going back to the beginning
done
@backamblock

This comment has been minimized.

Copy link

commented Jan 13, 2019

#!/bin/bash

while [[ true ]] ; do

	# performs synchronizations / copy
	rclone sync ....

	# waiting for something to change or it will pass 300 seconds
	inotifywait --recursive --timeout 300 -e modify,delete,create,move   SRC_DIR

	# going back to the beginning
done

can you tell me how to use it so it automatically runs from boot until shutdown in the background?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.