Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sync only the file change, not entire file [$1,755] #179

Closed
wesleyhuang opened this issue Dec 14, 2012 · 131 comments
Closed

Sync only the file change, not entire file [$1,755] #179

wesleyhuang opened this issue Dec 14, 2012 · 131 comments
Assignees
Milestone

Comments

@wesleyhuang
Copy link

@wesleyhuang wesleyhuang commented Dec 14, 2012

In Dropbox, there notes that only the file changes are sync-ed, not the entire file. https://www.dropbox.com/help/8/en. It is great if ownCloud can do the same. Especially for large files, sync-ing the entire files make a lot of bandwidth and time unnessararily wasted.

In my testing with latest 4.5.4 and sync client in Ubuntu 12.04, I prepared a 1GB text file, append a few characters to the end, and monitor the traffic and the file in the server, I see the entire 1GB file is transffered to the server and the server is actually creating a new file.

A reply from the forum indicates that librsync has this feature http://librsync.sourceforge.net/, maybe the csync can be switched to the libsync.

@allo-
Copy link

@allo- allo- commented Jan 26, 2013

i think this is ESSENTIAL for a sync client. And detecting of moved files. Maybe combining this like rsync -y (trying to find a similiar base file on the remote side to speedup upload), but note that rsync -y can slow down (lib)rsync when working in large directories.

@rarspace01
Copy link

@rarspace01 rarspace01 commented Mar 14, 2013

any update on this?

@LoZio
Copy link

@LoZio LoZio commented Jul 11, 2013

+1 for this. There's a wealth of files that may slightly change each time and are big to sync. Outlook PST, truecrypt, databases, phone backups, ...
Any update on this?

@jmstacey
Copy link

@jmstacey jmstacey commented Jul 14, 2013

I'd like to see this as well.

@fbeauvais
Copy link

@fbeauvais fbeauvais commented Jul 22, 2013

Waiting on this to install my first owncloud server.

@notDavid
Copy link

@notDavid notDavid commented Aug 3, 2013

I hope this will be implemented, seems essential..

  • for some reason my ownCloud client (1.3.0, OS X) has trouble syncing big files with small changes, and always re-uploads entire files...
@tityrus
Copy link

@tityrus tityrus commented Aug 4, 2013

I also see this as essential. Please.

@xeloader
Copy link

@xeloader xeloader commented Aug 14, 2013

I agree on this one as well, we need to be able to have some kind of incremental sync possibility before this is 100% usable.

@dragotin
Copy link
Contributor

@dragotin dragotin commented Aug 14, 2013

The problem is known, and we will get to it. At one day.

Until then, please try to retain from +1 comments ✌️

@Poelziminator
Copy link

@Poelziminator Poelziminator commented Sep 28, 2013

Can you give an estimate when will see the “one day”?
This functionality would open up whole new possibilities like syncing Truecrypt-volume files or Lightroom catalogs.

@danimo
Copy link
Contributor

@danimo danimo commented Sep 28, 2013

@Poelziminator Not in any release this year. Note that most work for this will be in the server and the general design.

@zwo-bot
Copy link

@zwo-bot zwo-bot commented Oct 29, 2013

This is the key-feature for everone works with trucrypt or bigger dbfiles. Please make it happen.

@jancborchardt
Copy link
Member

@jancborchardt jancborchardt commented Jan 5, 2014

This as well seems related to the 1.6 »Sync Performance« milestone. @MTRichards @dragotin @danimo?

@MTRichards
Copy link

@MTRichards MTRichards commented Jan 6, 2014

This is delta file syncing, and likely too complicated to get into 1.6 because it requires major server work too. The idea is to first improve performance on file level sync to get it more efficient, and then increase the granularity of the file comparisons (to file chunks), but going right to file chunks without first getting the file level sync comparisons would hurt performance more than help at this point because of the sheer volume of comparisons required.

@curtisz
Copy link

@curtisz curtisz commented Jan 27, 2014

I was about to choose ownCloud (with a very high probability of buying Enterprise) for use at our company but decided against it because of this particular issue. I'm a bit shocked that this isn't supported, considering how important this is for (potential) customers using TrueCrypt et al. This isn't so much of a +1 as an "at least one enterprise customer lost to the other guys".

@huksley
Copy link

@huksley huksley commented Mar 5, 2014

This is very important for virtual machine disk image files. Single byte change causes upload of multi-gigabyte files to remote server.

@beniroquai
Copy link

@beniroquai beniroquai commented Mar 11, 2014

Yes! Great idea!

@powerpaul17
Copy link

@powerpaul17 powerpaul17 commented Mar 11, 2014

As this is really the most important feature of a cloud service and it seems that nobody is interested in or working on it, I would like to offer some help with this issue. Is there already some information about what needs to be done, where to start, etc.?

@sagar-srivastava
Copy link

@sagar-srivastava sagar-srivastava commented Mar 24, 2014

I agree this is a very important feature. Owncloud is such a wonderful piece of software. I tested it today and found its quality up to the mark. Just Delta file sync addition will make it complete. Web interface/WebDav/Desktopsync/file sharing/ has worked out great on my VPS, works with ISPConfig3 implementation.

please initiate this effort and I am willing to buy the enterprise edition for my company.

@sodetemplin
Copy link

@sodetemplin sodetemplin commented Mar 31, 2014

This point should be an ownCloud priority, for sure ! Without this functionnality, the EE version is "just" the community one with more support ?

@kevincox
Copy link

@kevincox kevincox commented Mar 31, 2014

I remember hearing that owncloud wanted to keep the files stored on disk in their entirety. Is this (still) true? Because if so you could just generate a zsync signature file and a custom receiver that generated the entire file.

If the files are allowed to be broken up (maybe in a future version) then they can be chunked and a very efficient sync endpoint could be made.

What are the thoughts on this. I may consider working on this is my spare time.

@jancborchardt
Copy link
Member

@jancborchardt jancborchardt commented Apr 11, 2014

@dragotin @danimo @MTRichards what’s the plan on this one? I know it’s a big one, but it’s requested very often and seems to be important to improve performance.

@danimo
Copy link
Contributor

@danimo danimo commented Apr 12, 2014

@jancborchardt As long as the server doesn't offer any delta syncs, we can't implement it.

@L0j1k
Copy link

@L0j1k L0j1k commented Apr 12, 2014

@danimo Can you explain what you mean in a little more detail, please?

@danimo
Copy link
Contributor

@danimo danimo commented Apr 12, 2014

@LoZio Currently, we use WebDAV as the communication protocol with the server. Additionally, we can upload in chunks, but only if we transfer the entire file. Delta-sync requires another protocol extension. Also, we have no gurantee that the server is holding the hash-wise same file, since the server does not store file hashes.

@jancborchardt
Copy link
Member

@jancborchardt jancborchardt commented Apr 12, 2014

So what’s the plan with the server-side regarding this one? @karlitschek @DeepDiver1975 @PVince81

@karlitschek
Copy link

@karlitschek karlitschek commented Apr 12, 2014

We should do this in the future. But it is more a long term feature

@DeepDiver1975
Copy link
Member

@DeepDiver1975 DeepDiver1975 commented Feb 7, 2018

Technically zsync could detect moves, but my code doesn’t support that, all bytes moved will be resent, this was to make it easier to get something working quickly. I mentioned this as a future work area.

in which area are the adoptions neccessary to make this work? I'm trying to find out the impact - if bigger refactorings are necessary I question the current in depth review ...

@PVince81
Copy link
Member

@PVince81 PVince81 commented Feb 7, 2018

does the current impl at least help if appending stuff at the end of files ? In this case I expect that since the beginning's offset did not change it would only sync the appended block.

@PVince81
Copy link
Member

@PVince81 PVince81 commented Feb 7, 2018

did anyone here already test the feature ? if you did, please report where you saw improvements (file types, use case, etc)

@ahmedammar
Copy link
Contributor

@ahmedammar ahmedammar commented Feb 7, 2018

Appending will only send appended bytes. Regarding where the work needs to be done:

  1. zsync - code wasn’t designed with upload path in mind, I implemented that and use the path of least resistance, meaning dropping moved block support.
  2. oC - again to support moved blocks we’d need a more complex upload path where moved chunks are processed somehow, this can be as simple as sending a file with moved block to:from and modifying assembly code to handle appropriately.

I recommend we don’t try to do this now but after current simple approach is well tested.

@ahmedammar
Copy link
Contributor

@ahmedammar ahmedammar commented Feb 24, 2018

Just wanted to clarify something, the above only applies to the uploader, the downloader will not redownload moved chunks.

@gpothier
Copy link

@gpothier gpothier commented Apr 8, 2018

@PVince81 Regarding the file types that would benefit from delta sync: we are a publishing house and we work with rather big (up to 500MB) Adobe InDesign files. Small changes to these files are very good candidates for delta sync (Dropbox syncs small changes to these files almost instantly).

@ckamm ckamm modified the milestones: 2.5.0, 2.6.0 Apr 25, 2018
@ckamm
Copy link
Member

@ckamm ckamm commented Apr 25, 2018

The feature is stabilizing in the delta-sync branch and test builds will become available around the time 2.5.0 is released. If things go smoothly it'll be in 2.6.0.

@KevinLeigh
Copy link

@KevinLeigh KevinLeigh commented Jun 10, 2018

Is this issue currently being resolved? Would like to start resolving this issue but do not want to if someone else is.

@ahmedammar
Copy link
Contributor

@ahmedammar ahmedammar commented Jun 10, 2018

Yeah would be nice if this gets closed so I can claim the bounty?

@ahmedammar
Copy link
Contributor

@ahmedammar ahmedammar commented Jul 22, 2018

Is this really still not merged? What's the hold-up guys? Can we close this ticket??

@guruz
Copy link
Collaborator

@guruz guruz commented Jul 22, 2018

@ahmedammar It will be in client 2.6.0
On Friday we released 2.5.0 beta1. So it's coming :)
I don't know how the bounty thing works, but i'll try to get that info.

@DeepDiver1975
Copy link
Member

@DeepDiver1975 DeepDiver1975 commented Jul 22, 2018

@guru was the PR finally merged? Yes? Close Ticket and we Trigger payment on Bountysource

@guruz guruz closed this Jul 23, 2018
@ahmedammar
Copy link
Contributor

@ahmedammar ahmedammar commented Jul 23, 2018

Thanks! Have submitted the bounty claim too ...

@curtiszimmerman
Copy link

@curtiszimmerman curtiszimmerman commented Jul 23, 2018

@ahmedammar I don't use Owncloud anymore, but I was one of the contributors to the Bountysource bounty, and I just wanted to thank you for all this work. It was very interesting (and impressive) to read your status updates and commits. I hope you buy at least one beer from the bounty (or a nice dinner)!

@hodyroff
Copy link

@hodyroff hodyroff commented Jul 24, 2018

Thanks @ahmedammar really appreciated your contribution and looking forward to your continued participation - as time fits. Delta sync will for sure help to bring ownCloud into new places and motivate people to contribute and use it! Privacy and security and a full concentration on file sync and share are so important in todays world. Again Thank You!

@trampi
Copy link

@trampi trampi commented Jul 24, 2018

I know that one should not post off-topic comments in issues. However, I want to thank @ahmedammar and all other contributors in this issue. This is a really nice and long awaited feature. Well done, everyone!

@LoZio
Copy link

@LoZio LoZio commented Jul 24, 2018

I also want to thank you @ahmedammar for you work. My first post above is from 5 years ago and I'll be happy to go back to OC if this works!

@warnerbryce
Copy link

@warnerbryce warnerbryce commented Jul 24, 2018

@LoZio it’s gonna be merged to Nextcloud too.

@ahmedammar
Copy link
Contributor

@ahmedammar ahmedammar commented Jul 24, 2018

Hi all, thanks for all the kind words, much appreciated, it was a fun project and I had fun doing it, hope it stands the test of time. Would like to extend the thanks to all the oC devs who helped throughout the process and especially @ckamm who has so aptly continued the development where I left off, very nice! Thanks again all!

@guruz
Copy link
Collaborator

@guruz guruz commented Mar 20, 2019

This feature will be in the upcoming 2.6 alpha release.
Meanwhile you can try it inside the daily builds 2.6.x https://download.owncloud.com/desktop/daily/

@bolandross
Copy link

@bolandross bolandross commented Jun 21, 2019

So, let's say I wanted to experiment with this new feature. I just downloaded the 2.6.0alpha1 Desktop Client and enabled delta synchronisation. Unfortunately, without noticing any effect. So my question is: Which ownCloud server version is necessary for this to work as intended?

Thanks in advance and keep up the good work.

@michaelstingl
Copy link
Collaborator

@michaelstingl michaelstingl commented Jun 21, 2019

@bolandross The server part is only in the master branch of the ownCloud server. I hasn't been released in a production version yet.

@chrisgraham
Copy link

@chrisgraham chrisgraham commented Nov 1, 2019

Somebody needs to update the FAQ: https://owncloud.org/faq/#partialsyncing

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

You can’t perform that action at this time.