New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Two-way (bidirectional) synchronization #118

Open
alexander-yakushev opened this Issue Sep 1, 2015 · 42 comments

Comments

Projects
None yet
@alexander-yakushev

alexander-yakushev commented Sep 1, 2015

I'm sorry if this is answered elsewhere but I couldn't find it in that case.

I want to replace my current Owncloud+Owncloud client (Linux)+FolderSync(Android) setup with Drive+Rclone+FolderSync. But there is one thing I can't figure out how to do with rclone — smart two-way deletion synchronization. Which means: if a file was present on both server (Drive) and local machine, and then was deleted on either of them, the file will be eventually removed on both regardless of which direction you run sync first. Same, if a file was added on either server or client, it will be uploaded to the other one.

Can rclone do that, and if doesn't is there a chance of such functionality in future?

@ncw ncw added question enhancement and removed question labels Sep 1, 2015

@ncw

This comment has been minimized.

Show comment
Hide comment
@ncw

ncw Sep 1, 2015

Owner

No rclone can't do that yet.

However it could be done I think. Rclone would need to store a little bit of metadata - time of last sync. It could then use that for working out what to do in the delete files case.

Owner

ncw commented Sep 1, 2015

No rclone can't do that yet.

However it could be done I think. Rclone would need to store a little bit of metadata - time of last sync. It could then use that for working out what to do in the delete files case.

@alexander-yakushev

This comment has been minimized.

Show comment
Hide comment
@alexander-yakushev

alexander-yakushev Sep 1, 2015

Thank you for the prompt response!

alexander-yakushev commented Sep 1, 2015

Thank you for the prompt response!

@ncw

This comment has been minimized.

Show comment
Hide comment
@ncw

ncw Sep 2, 2015

Owner

I've made a note about this for the FAQ - I'll leave this ticket open for future developments!

Owner

ncw commented Sep 2, 2015

I've made a note about this for the FAQ - I'll leave this ticket open for future developments!

@ncw ncw added this to the Unplanned milestone Feb 10, 2016

@nigelhorne

This comment has been minimized.

Show comment
Hide comment
@nigelhorne

nigelhorne Sep 14, 2016

It's been a year, what progress has been made?

nigelhorne commented Sep 14, 2016

It's been a year, what progress has been made?

@ncw

This comment has been minimized.

Show comment
Hide comment
@ncw

ncw Sep 14, 2016

Owner

None at the moment. All the primitives needed are there in rclone, it just needs a 2-way sync expert to fill in the gaps!

Owner

ncw commented Sep 14, 2016

None at the moment. All the primitives needed are there in rclone, it just needs a 2-way sync expert to fill in the gaps!

@Technifocal

This comment has been minimized.

Show comment
Hide comment
@Technifocal

Technifocal Sep 14, 2016

@nigelhorne Kinda creepy, I was just Googling for this exact issue... four hours after you, a year after the original post date.

Technifocal commented Sep 14, 2016

@nigelhorne Kinda creepy, I was just Googling for this exact issue... four hours after you, a year after the original post date.

@ncw ncw changed the title from Correct two-way synchronization to Two-way (bidirectional) synchronization Sep 19, 2016

@bmlong137

This comment has been minimized.

Show comment
Hide comment
@bmlong137

bmlong137 Oct 3, 2016

I think a basic "2-way-sync" could be achieved with two commands. Execute "rclone sync -u " followed by "rclone sync -u ". Obviously doesn't work if both sides are changed, but what generic solution will?

bmlong137 commented Oct 3, 2016

I think a basic "2-way-sync" could be achieved with two commands. Execute "rclone sync -u " followed by "rclone sync -u ". Obviously doesn't work if both sides are changed, but what generic solution will?

@allanlaal

This comment has been minimized.

Show comment
Hide comment
@allanlaal

allanlaal Oct 8, 2016

2-way bi-directional sync is really needed considering that the official Dropbox Linux client has not worked for years

allanlaal commented Oct 8, 2016

2-way bi-directional sync is really needed considering that the official Dropbox Linux client has not worked for years

@nu111

This comment has been minimized.

Show comment
Hide comment
@nu111

nu111 Mar 25, 2017

It would be great to have 2-way bi-directional sync!
This way rclone would be a great open-soruce alternative to odrive!
Maybe we could get inspired by the MEGAsync client https://github.com/meganz/MEGAsync

nu111 commented Mar 25, 2017

It would be great to have 2-way bi-directional sync!
This way rclone would be a great open-soruce alternative to odrive!
Maybe we could get inspired by the MEGAsync client https://github.com/meganz/MEGAsync

@supagu

This comment has been minimized.

Show comment
Hide comment
@supagu

supagu Apr 8, 2017

yes would love to see a 2 way sync

supagu commented Apr 8, 2017

yes would love to see a 2 way sync

@benji791

This comment has been minimized.

Show comment
Hide comment
@benji791

benji791 Sep 12, 2017

Would be great to see this. It would allow to keep in sync OneDrive and Freenas.

benji791 commented Sep 12, 2017

Would be great to see this. It would allow to keep in sync OneDrive and Freenas.

@wilbowma

This comment has been minimized.

Show comment
Hide comment
@wilbowma

wilbowma Sep 30, 2017

FWIW, I'm currently using Unison for two way sync via ssh (https://github.com/bcpierce00/unison). I was considering trying to use rclone, since I use rclone for other syncing. Unison is written in OCaml, so you probably can't reuse the code directly, but it might provide a starting place for developing a good two-way syncing algorithm. It's an academic project with several publications about Unison, http://www.cis.upenn.edu/%7Ebcpierce/papers/index.shtml#File%20Synchronization, which might also be of help.

wilbowma commented Sep 30, 2017

FWIW, I'm currently using Unison for two way sync via ssh (https://github.com/bcpierce00/unison). I was considering trying to use rclone, since I use rclone for other syncing. Unison is written in OCaml, so you probably can't reuse the code directly, but it might provide a starting place for developing a good two-way syncing algorithm. It's an academic project with several publications about Unison, http://www.cis.upenn.edu/%7Ebcpierce/papers/index.shtml#File%20Synchronization, which might also be of help.

@hildogjr

This comment has been minimized.

Show comment
Hide comment
@hildogjr

hildogjr Apr 27, 2018

There is also the python @cjnaz work around (I tryed to did some improvements https://github.com/hildogjr/RCloneSync/). But still just a work around.

hildogjr commented Apr 27, 2018

There is also the python @cjnaz work around (I tryed to did some improvements https://github.com/hildogjr/RCloneSync/). But still just a work around.

@cjnaz

This comment has been minimized.

Show comment
Hide comment
@cjnaz

cjnaz Apr 27, 2018

See https://github.com/ncw/rclone/wiki/Third-Party-Integrations-with-rclone for RCloneSync which points to https://github.com/cjnaz/RCloneSync. This is my original version that I've been running for since July 2017. The README.md has an in-depth discussion of the nuances of bidir sync. @hildogjr has updated this for Python 3 support, Python style standards, standardized Linux command line arguments, and added a few features. I have yet to merge his pull requests, but they look good. I would recommend his version (link above) but the core functionality is unchanged, I believe.

cjnaz commented Apr 27, 2018

See https://github.com/ncw/rclone/wiki/Third-Party-Integrations-with-rclone for RCloneSync which points to https://github.com/cjnaz/RCloneSync. This is my original version that I've been running for since July 2017. The README.md has an in-depth discussion of the nuances of bidir sync. @hildogjr has updated this for Python 3 support, Python style standards, standardized Linux command line arguments, and added a few features. I have yet to merge his pull requests, but they look good. I would recommend his version (link above) but the core functionality is unchanged, I believe.

@yonjah

This comment has been minimized.

Show comment
Hide comment
@yonjah

yonjah Jun 27, 2018

Contributor

@cjnaz are you planing to merge @hildogjr changes ?
It seem like both of you are actively working on each fork without merging in quite some time.

Contributor

yonjah commented Jun 27, 2018

@cjnaz are you planing to merge @hildogjr changes ?
It seem like both of you are actively working on each fork without merging in quite some time.

@hildogjr

This comment has been minimized.

Show comment
Hide comment
@hildogjr

hildogjr Jun 27, 2018

There is also a branch of mine by @deeuu that appear to trying to make a formal package.

hildogjr commented Jun 27, 2018

There is also a branch of mine by @deeuu that appear to trying to make a formal package.

@deeuu

This comment has been minimized.

Show comment
Hide comment
@deeuu

deeuu Jun 27, 2018

Hey, yeah I want the ability to pip install. @hildogjr I'll send you a pull request once I'm done, and then it would be good to get it into the original repo

deeuu commented Jun 27, 2018

Hey, yeah I want the ability to pip install. @hildogjr I'll send you a pull request once I'm done, and then it would be good to get it into the original repo

@hildogjr

This comment has been minimized.

Show comment
Hide comment
@hildogjr

hildogjr Jun 27, 2018

Since Rclone browser still stoped on development. This could be used on future by wxPython interface (just a long long shot).

hildogjr commented Jun 27, 2018

Since Rclone browser still stoped on development. This could be used on future by wxPython interface (just a long long shot).

@cjnaz

This comment has been minimized.

Show comment
Hide comment
@cjnaz

cjnaz Jun 27, 2018

cjnaz commented Jun 27, 2018

@deeuu

This comment has been minimized.

Show comment
Hide comment
@deeuu

deeuu Jun 27, 2018

@hildogjr Are you planning to merge into the original repo of @cjnaz ?
@cjnaz Would you consider the package (+ some styling) work I've been working on?

It would be great to fuse our efforts (Thanks again to @cjnaz for your original work).

deeuu commented Jun 27, 2018

@hildogjr Are you planning to merge into the original repo of @cjnaz ?
@cjnaz Would you consider the package (+ some styling) work I've been working on?

It would be great to fuse our efforts (Thanks again to @cjnaz for your original work).

@cjnaz

This comment has been minimized.

Show comment
Hide comment
@cjnaz

cjnaz Jun 27, 2018

cjnaz commented Jun 27, 2018

@cjnaz

This comment has been minimized.

Show comment
Hide comment
@cjnaz

cjnaz Jun 27, 2018

cjnaz commented Jun 27, 2018

@deeuu

This comment has been minimized.

Show comment
Hide comment
@deeuu

deeuu Jun 27, 2018

There are quite a few here but perhaps it is best if I start again from your latest version; after all, I'm the new guy. Do take a look anyway and let me know what you think.

I haven't really looked into the python 3 support mods of @hildogjr, but it might make sense to get those merged before you comit your big revision? Thoughts?

deeuu commented Jun 27, 2018

There are quite a few here but perhaps it is best if I start again from your latest version; after all, I'm the new guy. Do take a look anyway and let me know what you think.

I haven't really looked into the python 3 support mods of @hildogjr, but it might make sense to get those merged before you comit your big revision? Thoughts?

@cjnaz

This comment has been minimized.

Show comment
Hide comment
@cjnaz

cjnaz Jun 27, 2018

cjnaz commented Jun 27, 2018

@hildogjr

This comment has been minimized.

Show comment
Hide comment
@hildogjr

hildogjr Jun 27, 2018

Could be easily installed (and dependences) and configured by a simple command for this cold be pip install rclonesync in the command terminal (Windows/Linux/Mac-OS). Also the user can easy updated and tracked version.

Even, in this configuration procedure could be added the rclone executable installation.

You will need to create an account in https://pypi.org/ for this. I am just a collaborator and administrator of one package there until now, but this really help in the propagation of the software.

hildogjr commented Jun 27, 2018

Could be easily installed (and dependences) and configured by a simple command for this cold be pip install rclonesync in the command terminal (Windows/Linux/Mac-OS). Also the user can easy updated and tracked version.

Even, in this configuration procedure could be added the rclone executable installation.

You will need to create an account in https://pypi.org/ for this. I am just a collaborator and administrator of one package there until now, but this really help in the propagation of the software.

@deeuu

This comment has been minimized.

Show comment
Hide comment
@deeuu

deeuu Jun 27, 2018

@cjnaz I'm happy to develop the package, but it would be great if @hildogjr submits a pull request to merge his python 3 work first, and then @cjnaz makes the big update. I can then establish the package.

deeuu commented Jun 27, 2018

@cjnaz I'm happy to develop the package, but it would be great if @hildogjr submits a pull request to merge his python 3 work first, and then @cjnaz makes the big update. I can then establish the package.

@hildogjr

This comment has been minimized.

Show comment
Hide comment
@hildogjr

hildogjr Jun 27, 2018

Agree.
I would help @cjnaz, after with needed some tips about PyPI.

hildogjr commented Jun 27, 2018

Agree.
I would help @cjnaz, after with needed some tips about PyPI.

@cjnaz

This comment has been minimized.

Show comment
Hide comment
@cjnaz

cjnaz Jun 27, 2018

cjnaz commented Jun 27, 2018

@cjnaz

This comment has been minimized.

Show comment
Hide comment
@cjnaz

cjnaz Jul 2, 2018

cjnaz commented Jul 2, 2018

@cjnaz

This comment has been minimized.

Show comment
Hide comment
@cjnaz

cjnaz Jul 4, 2018

cjnaz commented Jul 4, 2018

@who-am-1

This comment has been minimized.

Show comment
Hide comment

who-am-1 commented Jul 18, 2018

@deeuu

This comment has been minimized.

Show comment
Hide comment
@deeuu

deeuu Jul 18, 2018

Looks like there is another repo:
https://github.com/cjnaz/rclonesync-V2
@cjnaz how come?
btw, I'm planning on looking over this very soon!

deeuu commented Jul 18, 2018

Looks like there is another repo:
https://github.com/cjnaz/rclonesync-V2
@cjnaz how come?
btw, I'm planning on looking over this very soon!

@brenthuisman

This comment has been minimized.

Show comment
Hide comment
@brenthuisman

brenthuisman Jul 25, 2018

Isn't this something that should be part of rclone, perhaps even its default operation when syncing? It is what I initially expected the sync command to do.

brenthuisman commented Jul 25, 2018

Isn't this something that should be part of rclone, perhaps even its default operation when syncing? It is what I initially expected the sync command to do.

@hildogjr

This comment has been minimized.

Show comment
Hide comment
@hildogjr

hildogjr Jul 25, 2018

I think is o what most of "not expert user" expected. Something to replace or provide some desktop bidirectional synchronization.

hildogjr commented Jul 25, 2018

I think is o what most of "not expert user" expected. Something to replace or provide some desktop bidirectional synchronization.

@ncw

This comment has been minimized.

Show comment
Hide comment
@ncw

ncw Jul 30, 2018

Owner

What do you think @cjnaz - would you like to integrate the functionality of rclonesync as a new rclone command?

Owner

ncw commented Jul 30, 2018

What do you think @cjnaz - would you like to integrate the functionality of rclonesync as a new rclone command?

@hildogjr

This comment has been minimized.

Show comment
Hide comment
@hildogjr

hildogjr Jul 30, 2018

For me, would be very nice. Something such as rclone bisync (bidirectional synchronization).
I could even join to effort.

hildogjr commented Jul 30, 2018

For me, would be very nice. Something such as rclone bisync (bidirectional synchronization).
I could even join to effort.

@cjnaz

This comment has been minimized.

Show comment
Hide comment
@cjnaz

cjnaz Jul 31, 2018

Having rclone support bisync (thanks @hildogjr that's a better name) would be great. A few thoughts...

  • rclone bisync would wrap the capability into a well supported tool, rather than a side script that will chase rclone bugs and new features.
  • I don't know about the behavior and corner cases of all the other remotes. We'd probably want to phase in wider support over time.
  • My scheme of saving prior bisync status in a local working directory has no precedent in rlcone, and the error cases and their handling may not be widely agreed to. We should have a discussion on these topics before proceeding.
  • What is the real problem we are solving? Having this as a separate rclonesync.py script is inconvenient, or unfortunate, or seriously broken for what reasons? Just wanting to understand the true value proposition.

cjn

cjnaz commented Jul 31, 2018

Having rclone support bisync (thanks @hildogjr that's a better name) would be great. A few thoughts...

  • rclone bisync would wrap the capability into a well supported tool, rather than a side script that will chase rclone bugs and new features.
  • I don't know about the behavior and corner cases of all the other remotes. We'd probably want to phase in wider support over time.
  • My scheme of saving prior bisync status in a local working directory has no precedent in rlcone, and the error cases and their handling may not be widely agreed to. We should have a discussion on these topics before proceeding.
  • What is the real problem we are solving? Having this as a separate rclonesync.py script is inconvenient, or unfortunate, or seriously broken for what reasons? Just wanting to understand the true value proposition.

cjn

@hildogjr

This comment has been minimized.

Show comment
Hide comment
@hildogjr

hildogjr Aug 1, 2018

  1. Better user experience and integration;
  2. Neither I, but after an alpha release of this command a lot of user will improve this discussion and may be finish and stable to all clouds in few releases;
  3. Total agree, this is the critical point of this implementation and I appreciate opinions, mainly of the programmers of rclone. Even opinion about the implementation in Go-language.
  4. This may be a step forward to create a fully and unique integrated tool to the most common cloud services.

hildogjr commented Aug 1, 2018

  1. Better user experience and integration;
  2. Neither I, but after an alpha release of this command a lot of user will improve this discussion and may be finish and stable to all clouds in few releases;
  3. Total agree, this is the critical point of this implementation and I appreciate opinions, mainly of the programmers of rclone. Even opinion about the implementation in Go-language.
  4. This may be a step forward to create a fully and unique integrated tool to the most common cloud services.
@brenthuisman

This comment has been minimized.

Show comment
Hide comment
@brenthuisman

brenthuisman Aug 1, 2018

  1. It also signals its a feature that's not going away, and will be supported like any other command.
  2. FreeFileSync (afaik the only other tool doing bi-directional sync) also generates local and remote index-files. Seems like a good idea, and it could remove the very bandwidth and time intensive remote indexing.

brenthuisman commented Aug 1, 2018

  1. It also signals its a feature that's not going away, and will be supported like any other command.
  2. FreeFileSync (afaik the only other tool doing bi-directional sync) also generates local and remote index-files. Seems like a good idea, and it could remove the very bandwidth and time intensive remote indexing.
@ncw

This comment has been minimized.

Show comment
Hide comment
@ncw

ncw Aug 1, 2018

Owner

Having rclone support bisync (thanks @hildogjr that's a better name) would be great. A few thoughts...

rclone bisync would wrap the capability into a well supported tool, rather than a side script that will chase rclone bugs and new features.

Yes. It would get maintained as part of rclone and would have a test suite run as part of rclone's testing infrastructure.

I don't know about the behavior and corner cases of all the other remotes. We'd probably want to phase in wider support over time.

The remotes are very consistent - the integration tests ensure that. I'm always amazed at when I add a new remote, get the integration tests to pass, then everything else just works!

My scheme of saving prior bisync status in a local working directory has no precedent in rlcone, and the error cases and their handling may not be widely agreed to. We should have a discussion on these topics before proceeding.

Yes, rclone has steered clear of making a local database. However the cache backend does now so there is precedent.

What is the real problem we are solving? Having this as a separate rclonesync.py script is inconvenient, or unfortunate, or seriously broken for what reasons? Just wanting to understand the true value proposition.

I think having it as an rclone subcommand would be of benefit to users. It then only one thing for the user to install. Making it part of rclone itself will make consistent with the rest of rclone.

It would mean re-writing it in go, and reworking the internals to call rclone's internal APIs rather than the command line interface so would be quite a lot of work. I'd be willing to help with some of that.

Owner

ncw commented Aug 1, 2018

Having rclone support bisync (thanks @hildogjr that's a better name) would be great. A few thoughts...

rclone bisync would wrap the capability into a well supported tool, rather than a side script that will chase rclone bugs and new features.

Yes. It would get maintained as part of rclone and would have a test suite run as part of rclone's testing infrastructure.

I don't know about the behavior and corner cases of all the other remotes. We'd probably want to phase in wider support over time.

The remotes are very consistent - the integration tests ensure that. I'm always amazed at when I add a new remote, get the integration tests to pass, then everything else just works!

My scheme of saving prior bisync status in a local working directory has no precedent in rlcone, and the error cases and their handling may not be widely agreed to. We should have a discussion on these topics before proceeding.

Yes, rclone has steered clear of making a local database. However the cache backend does now so there is precedent.

What is the real problem we are solving? Having this as a separate rclonesync.py script is inconvenient, or unfortunate, or seriously broken for what reasons? Just wanting to understand the true value proposition.

I think having it as an rclone subcommand would be of benefit to users. It then only one thing for the user to install. Making it part of rclone itself will make consistent with the rest of rclone.

It would mean re-writing it in go, and reworking the internals to call rclone's internal APIs rather than the command line interface so would be quite a lot of work. I'd be willing to help with some of that.

@RasmusKallas

This comment has been minimized.

Show comment
Hide comment
@RasmusKallas

RasmusKallas Sep 29, 2018

SyncThing (also in Go) might be useful.

RasmusKallas commented Sep 29, 2018

SyncThing (also in Go) might be useful.

@cjnaz

This comment has been minimized.

Show comment
Hide comment
@cjnaz

cjnaz Oct 3, 2018

V2.3 adds Windows support, support for UNC paths, and adds a --rclone switch for specifying an alternate version of rclone from what's found in the path environment var. I also explored an rclone Crypt setup that I find quite useful for securing passing/synching files between various endpoints. I need to look into getting rclone and rclonesync up on Android so that I have a clear path for syncing secure files with my phone.
I hope to start working on incorporating bisync into rclone proper in a few weeks.

cjnaz commented Oct 3, 2018

V2.3 adds Windows support, support for UNC paths, and adds a --rclone switch for specifying an alternate version of rclone from what's found in the path environment var. I also explored an rclone Crypt setup that I find quite useful for securing passing/synching files between various endpoints. I need to look into getting rclone and rclonesync up on Android so that I have a clear path for syncing secure files with my phone.
I hope to start working on incorporating bisync into rclone proper in a few weeks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment