New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Requesting delta-sync in longterm #417

Open
exokkk opened this Issue Jul 15, 2016 · 28 comments

Comments

Projects
None yet
@exokkk

exokkk commented Jul 15, 2016

Hi all,

Delta Sync would be great for my truecrypt/veracrypt huge files (~30-100gb). Without delta sync I must stay absent from this product.

Delta Sync would also provide you a feature to distinguish from owncloud

Couldn't there be an optional (maybe extension / folder / file -based) mechanism to perform Delta Sync ("optionally" as I agree that Delta Sync does not make sense for all kind of files/folders)? Maybe even using something existing like rsync?

@rullzer

This comment has been minimized.

Member

rullzer commented Jul 15, 2016

I have looked into zsync (client rsync). If we would ever implement such a feature it will most likely will be using that. Since you have to offload all the computation to the client side else you will kill the server.

I don't know about truecrypt/veracrypt. But actually most container formats (and encrypted is even worse). don't lend themself particually well for delta sync. Since often a small change results in a a lot of changed bytes.

@exokkk

This comment has been minimized.

exokkk commented Jul 15, 2016

I can only speak for truecrypt (but suppose that veracrypt does the same): if you change 1MB within the container the whole file changes only a bit more than 1MB as well (I am very sure about this). Also not even password changes would change the whole file, only little parts, refer https://news.ycombinator.com/item?id=6523286 and http://crypto.stackexchange.com/questions/18479/how-does-truecrypt-change-password-without-the-need-for-a-complete-re-encryption
So Delta Sync + Truecrypt (veracrypt) is a really perfect combination.
Although I can see that this feature will not be desired from many people there are some cases, like mine, where it would be great. Maybe there are other cases that I/we cannot think of but exist. Not sure, but VM images might profit from delta sync as well for example. Also, for some files you might uncompress -> compare -> delta_sync -> compress_server_side_again [ok this might be a too costly action, I do not know. This would work for e.g. *.pptx etc as well]
Clientside computation seems rational to me.

Thanks for considering the feature in any future release.

@tflidd

This comment has been minimized.

Contributor

tflidd commented Jul 16, 2016

That is an interesting feature for virtual images. Are there any experiences with encrypted containers and diff sync tools?

One annoying thing about the sync-process is that you must transfer the files through the client. You can't just place them from a hard disk or use a faster transfer mechanism. Therefore many ask for a diff-sync feature but the ability to compare files (based on a hash-sum) would already help a lot of these people and it's much easier to implement.

@maverick74

This comment has been minimized.

maverick74 commented Dec 13, 2016

I don't mind at all to offload all the computation to the client side, as long as such feature is made available!!! We, actually, consider this a very important feature for business!

To have an idea in a 0-10 scale what would be the cost (not monetary) of developing this?

Tks

@wudimenghuan

This comment has been minimized.

wudimenghuan commented Jan 19, 2017

Dropbox and Onedrive have delta sync.
Seafile have delta sync, but it cause files broken.
I hope you see the rsync. I do need delta sync

@cowai

This comment has been minimized.

cowai commented Feb 16, 2017

@rullzer How would the client have the previous file for calculating the delta?

@Bigpet

This comment has been minimized.

Bigpet commented Mar 23, 2017

@cowai you either need to keep a copy of your last sync around (using file-system specific things like shadow copy seems out of the question for the broad range of platforms with sync clients) or you have to do block-level syncing instead, like "syncthing" does it.

@stratacast

This comment has been minimized.

stratacast commented Apr 13, 2017

I think block-level syncing like syncthing is probably the easiest implementation in code, and perhaps the cheapest to write. I'm seriously interested in this, and I know some companies that are too (Quickbooks files man...ugly stuff). Like @Bigpet said, you'd need a copy of the file before changes onhand, or put some hooks into writes that go into that specific directory, but the latter sounds very messy and dangerous. I wish I knew how to write code better because I would 100% do this..I'm definitely a Kindergarten koder compared to a lot of people that put stuff on github. Thought I'd voice that there's interest on my end, and on the end of local companies I know.

@eglipeter

This comment has been minimized.

eglipeter commented Jun 5, 2017

Are there concrete plans when delta syncing will be available? May I hope to see this implemented in Nextcloud 13 already?

@gschenck

This comment has been minimized.

gschenck commented Oct 18, 2017

There is some progress on owncloud:

owncloud/core#16162

@ahmedammar

This comment has been minimized.

ahmedammar commented Oct 26, 2017

@gschenck Please feel free to try out the latest code, the core implementation should be complete now.

@jkaberg

This comment has been minimized.

jkaberg commented Oct 26, 2017

@ahmedammar any plans on submitting the PR against NC as well further down the road?

@ahmedammar

This comment has been minimized.

ahmedammar commented Oct 26, 2017

@jkaberg once the work is complete and merged in oC I can have a look, assuming the code-base isn't too different at the core ...

@maverick74

This comment has been minimized.

maverick74 commented Nov 28, 2017

@ahmedammar can you give us an update about the feature? (If possible a probable ETA?)

@ahmedammar

This comment has been minimized.

ahmedammar commented Nov 28, 2017

@maverick74 no ETA for nextcloud, if someone is willing to open a bounty for it I could look into it more urgently, otherwise, for reference:
owncloud/client#6131
owncloud/core#29404

@L00maca

This comment has been minimized.

L00maca commented Dec 24, 2017

It's not much and I'm not even sure I did this right since I never did this before, but I don't mind chipping in to help this get done.
Bountysource

@jospoortvliet

This comment has been minimized.

Member

jospoortvliet commented Jan 2, 2018

The bounty is already at 115 dollar now. It should not be terribly hard to get this merged in Nc client and server, I think, but it won't make it for 13 😄

@ahmedammar

This comment has been minimized.

ahmedammar commented Jan 2, 2018

I won’t be looking into this until oC actually merge first, since that saves me any duplicated effort. Unless this bounty gets so big that I can ignore oC all together :)

@maverick74

This comment has been minimized.

maverick74 commented Mar 12, 2018

FWIW i guess there are some news at owncloud/core#29404

@wudimenghuan

This comment has been minimized.

wudimenghuan commented Mar 17, 2018

@maverick74 So It can be merged... @rullzer @jospoortvliet

@tflidd

This comment has been minimized.

Contributor

tflidd commented Mar 21, 2018

FWIW i guess there are some news at owncloud/core#29404

That's the server side. Client-side is still on a development branch and subject to testing (https://github.com/owncloud/client/labels/Delta-sync). Unless this is not finished, it doesn't make a lot of sense to merge anything at the moment, so you can only help testing it.

@petrk94

This comment has been minimized.

petrk94 commented Apr 9, 2018

I think nextcloud should hurry up, delta sync will be released in the next owncloud update:
https://owncloud.com/owncloud-implements-delta-sync-technology/

@jospoortvliet

This comment has been minimized.

Member

jospoortvliet commented Apr 12, 2018

@petrk94 yeah, it could in theory be merged - but ownCloud notes it'll be in testing until 2019, let's see. @ahmedammar can make a PR for the server - the client will get it as we sync upstream actively still.

@petrk94

This comment has been minimized.

petrk94 commented Apr 12, 2018

Im wondering why I get so much thump down, just want to keep the thread updated :/

@nextcloud-bot nextcloud-bot added the stale label Jun 20, 2018

@jcklpe

This comment has been minimized.

jcklpe commented Aug 9, 2018

If I'm understanding stuff correctly it sounds like NextCloud won't be having this feature any time soon, correct?

@nextcloud-bot nextcloud-bot removed the stale label Aug 9, 2018

@tflidd

This comment has been minimized.

Contributor

tflidd commented Aug 9, 2018

ownCloud currently uses client version 2.4. Version 2.5 is in beta tests now (owncloud/client#6483) and the delta sync feature was announced for version 2.6. Now with a bit of guessing, between major releases there are often 6 months or more, so I wouldn't expect a working client before the end of this year.

From Nextcloud side, they took over their own development to realize the new client-side encryption which is currently in beta status. This feature is one of the main priorities at the moment, and it will probably take some time to ship this feature and get it really stable. After that, they could implement delta sync but I won't expect it before mid-2019. This is no official statement, priorities can change ...

@maverick74

This comment has been minimized.

maverick74 commented Nov 15, 2018

Apparently the client-side is already merged ( owncloud/client#6297 )

But they're still hunting for bugs until 2.6.0.

It would be nice to have it as an experimental Opt-in feature over here, however :)

@jospoortvliet

This comment has been minimized.

Member

jospoortvliet commented Nov 16, 2018

We might merge it during the course of our 2.6 development, I suppose - but we have a huge amount of things we want to work on, not sure how high the prio is on this one. Help is welcome - if somebody feels like creating a PR for our client that backports this feature, that'd be cool of course!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment