Two-way (bidirectional) synchronization

[alexander-yakushev]

I'm sorry if this is answered elsewhere but I couldn't find it in that case.

I want to replace my current Owncloud+Owncloud client (Linux)+FolderSync(Android) setup with Drive+Rclone+FolderSync. But there is one thing I can't figure out how to do with rclone — smart two-way deletion synchronization. Which means: if a file was present on both server (Drive) and local machine, and then was deleted on either of them, the file will be eventually removed on both regardless of which direction you run sync first. Same, if a file was added on either server or client, it will be uploaded to the other one.

Can rclone do that, and if doesn't is there a chance of such functionality in future?

How to use GitHub

• Please use the 👍 reaction to show that you are affected by the same issue.
• Please don't comment if you have no relevant information to add. It's just extra noise for everyone subscribed to this issue.
• Subscribe to receive notifications on status change and new comments.

[ncw]

No rclone can't do that yet.

However it could be done I think. Rclone would need to store a little bit of metadata - time of last sync. It could then use that for working out what to do in the delete files case.

[alexander-yakushev]

Thank you for the prompt response!

[ncw]

I've made a note about this for the FAQ - I'll leave this ticket open for future developments!

[ncw]

None at the moment. All the primitives needed are there in rclone, it just needs a 2-way sync expert to fill in the gaps!

[bmlong137]

I think a basic "2-way-sync" could be achieved with two commands. Execute "rclone sync -u " followed by "rclone sync -u ". Obviously doesn't work if both sides are changed, but what generic solution will?

[nu111]

It would be great to have 2-way bi-directional sync!
This way rclone would be a great open-soruce alternative to odrive!
Maybe we could get inspired by the MEGAsync client https://github.com/meganz/MEGAsync

[wilbowma]

FWIW, I'm currently using Unison for two way sync via ssh (https://github.com/bcpierce00/unison). I was considering trying to use rclone, since I use rclone for other syncing. Unison is written in OCaml, so you probably can't reuse the code directly, but it might provide a starting place for developing a good two-way syncing algorithm. It's an academic project with several publications about Unison, http://www.cis.upenn.edu/%7Ebcpierce/papers/index.shtml#File%20Synchronization, which might also be of help.

[hildogjr]

There is also the python @cjnaz work around (I tryed to did some improvements https://github.com/hildogjr/RCloneSync/). But still just a work around.

[cjnaz]

See https://github.com/ncw/rclone/wiki/Third-Party-Integrations-with-rclone for RCloneSync which points to https://github.com/cjnaz/RCloneSync. This is my original version that I've been running for since July 2017. The README.md has an in-depth discussion of the nuances of bidir sync. @hildogjr has updated this for Python 3 support, Python style standards, standardized Linux command line arguments, and added a few features. I have yet to merge his pull requests, but they look good. I would recommend his version (link above) but the core functionality is unchanged, I believe.

[cjnaz]

Hello folks. I'm about to release a good sized update. A primary addition are test cases for the various switches. Also changed the switches to Linux standard. Some Python style guide edits as well. There is only 1 pull request on my repo for an edit for Python 3 support, which I need to install Py3 to validate. The nodelete switch may be a good edition but my brief look at the change is that it actually only disables the local empty rmdir, not deletes locally of deleted files on the Remote side. I don't agree with the make cron addition. That would better be in a separate script, and what the cron should be may be very user specific. I added a suggestion to the new TROUBLESHOOTING.md There have been many forked edits which would need to be considered. I don't plan to change names from mixedCase to lower_case_with_underscores. I'll consider this to be legacy code and not that important to touch globally. Cjn

…

On Wed, Jun 27, 2018, 10:28 AM Hildo Guillardi Júnior < ***@***.***> wrote: Since Rclone browser still stoped on development. This could be used on future by wxPython interface (just a long long shot). — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#118 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AOKq4Txh_lge36_pvYJgObfwwGk-GzFfks5uA8C9gaJpZM4F16Yl> .

[cjnaz]

@deeuu, I'll certainly look through the commits list <https://github.com/deeuu/RCloneSync/commits/master> and see what's new. I'll likely need to manually incorporate the edits since I've made substantial changes. The new version is functionally equivalent, with changes to the command line interface, and a couple bug fixes. Also added is a hash of the `--filters-file` (was `--ExcludesListFile`) to effectively force a `--first-sync` if the filters are changed.

…

On Wed, Jun 27, 2018 at 11:43 AM Dominic Ward ***@***.***> wrote: @hildogjr <https://github.com/hildogjr> Are you planning to merge into the original repo of @cjnaz <https://github.com/cjnaz> ? @cjnaz <https://github.com/cjnaz> Would you consider the package (+ some styling) work I've been working on <https://github.com/deeuu/RCloneSync>? It would be great to fuse our efforts (Thanks again to @cjnaz <https://github.com/cjnaz> for your original work). — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#118 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AOKq4eyFJinOg3gttvVHrKInDvU0JhBlks5uA9JkgaJpZM4F16Yl> .

[ivandeex]

Merged on master.
Beta released on https://beta.rclone.org

[typxxi]

I guess it will become a present for chrismas or eastern cause I wanna avoid the risks of betas.
thx

[Harvie]

Merged on master. Beta released on https://beta.rclone.org

Nice :-) Will this work in star topology? eg.: multiple devices doing bidirectional sync with shared directory on the server?
I am looking for dropbox/googledrive replacement. So my data is available when offline and will resync later to ordinary SFTP server with no server side software.

[ivandeex]

@Harvie
This ticket is about BASIC 2-way sync.
It is closed. It will not handle advanced use cases.
We predicted further questions about applicability to other topologies.
Please post your question to the closest related ticket #5689
Thank you.

[cjnaz]

If you are careful about scheduling, yes you can sync more than two targets. With rclonesync I used to sync both my home storage to Dropbox and also/separately my work computer to the same Dropbox root location. This allowed me to pass personal files back and forth. Multi-target sync is not a native function of rclone bisync or rclonesync. It is two separate operations. It will likely break if both syncs are running at the same time. Schedule carefully. Probably best to set up a wrapper script with some sort of lock file. Early on, rclonesync's lock file did not track the path names so it worked nicely for blocking one sync while the other was still running.

…

On Tue, Nov 2, 2021, 2:41 AM Ivan Andreev ***@***.***> wrote: @Harvie <https://github.com/Harvie> This ticket is about BASIC 2-way sync. It is closed. It will not handle advanced use cases. We predicted further questions about applicability to other topologies. Please post your question to the closest related ticket #5689 <#5689> Thank you. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#118 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADRKVYIOCS5RGFLY7CR2ZMDUJ6W3BANCNFSM4BOXUYSQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[gerardbosch]

Hi there, what are the known backends that "support file modification time"? I was trying with mega.nz as it offers good amount of free storage and strong security. But after trying bisync I got the error:

Failed to bisync: modification time support is missing on path2

I'm wondering about free and good alternatives I can use. I liked also mega.nz as their application is very lightweight compared to something like Dropbox app that eats a lot of RAM. But on an old low spec Windows server I wanted to use rclone as the mega app does not work (requires extra stuff to be installed).

I guess that nothing can be done with mega, at least right now. Any ideas or recommended free backend compatible with bisync? Thanks!

[Jwink3101]

Hi there, what are the known backends that "support file modification time"? I was trying with mega.nz as it offers good amount of free storage and strong security. But after trying bisync I got the error:

Failed to bisync: modification time support is missing on path2

I'm wondering about free and good alternatives I can use. I liked also mega.nz as their application is very lightweight compared to something like Dropbox app that eats a lot of RAM. But on an old low spec Windows server I wanted to use rclone as the mega app does not work (requires extra stuff to be installed).

I guess that nothing can be done with mega, at least right now. Any ideas or recommended free backend compatible with bisync? Thanks!

This really is a better question for the forum. But see the remote overview

Also, I am the author of syncrclone which can support remotes that don’t support modtime. It has pros and cons. See my comparison

[gerardbosch]

Thanks @Jwink3101 for your answer, and sorry for posting here the question that belongs to the forum –just landed here looking for the bisync error :)
Now that I see the link to the remote overview, I think that Google Drive will perfectly meet my requirements (supports ModTime R/W), just in case it helps someone else. Not 20GB but 15GB of free storage, still enough =)