Requesting delta-sync in longterm

[exokkk]

Hi all,

Delta Sync would be great for my truecrypt/veracrypt huge files (~30-100gb). Without delta sync I must stay absent from this product.

Delta Sync would also provide you a feature to distinguish from owncloud

Couldn't there be an optional (maybe extension / folder / file -based) mechanism to perform Delta Sync ("optionally" as I agree that Delta Sync does not make sense for all kind of files/folders)? Maybe even using something existing like rsync?

[rullzer]

I have looked into zsync (client rsync). If we would ever implement such a feature it will most likely will be using that. Since you have to offload all the computation to the client side else you will kill the server.

I don't know about truecrypt/veracrypt. But actually most container formats (and encrypted is even worse). don't lend themself particually well for delta sync. Since often a small change results in a a lot of changed bytes.

[exokkk]

I can only speak for truecrypt (but suppose that veracrypt does the same): if you change 1MB within the container the whole file changes only a bit more than 1MB as well (I am very sure about this). Also not even password changes would change the whole file, only little parts, refer https://news.ycombinator.com/item?id=6523286 and http://crypto.stackexchange.com/questions/18479/how-does-truecrypt-change-password-without-the-need-for-a-complete-re-encryption
So Delta Sync + Truecrypt (veracrypt) is a really perfect combination.
Although I can see that this feature will not be desired from many people there are some cases, like mine, where it would be great. Maybe there are other cases that I/we cannot think of but exist. Not sure, but VM images might profit from delta sync as well for example. Also, for some files you might uncompress -> compare -> delta_sync -> compress_server_side_again [ok this might be a too costly action, I do not know. This would work for e.g. *.pptx etc as well]
Clientside computation seems rational to me.

Thanks for considering the feature in any future release.

[tflidd]

That is an interesting feature for virtual images. Are there any experiences with encrypted containers and diff sync tools?

One annoying thing about the sync-process is that you must transfer the files through the client. You can't just place them from a hard disk or use a faster transfer mechanism. Therefore many ask for a diff-sync feature but the ability to compare files (based on a hash-sum) would already help a lot of these people and it's much easier to implement.

[maverick74]

I don't mind at all to offload all the computation to the client side, as long as such feature is made available!!! We, actually, consider this a very important feature for business!

To have an idea in a 0-10 scale what would be the cost (not monetary) of developing this?

Tks

[wudimenghuan]

Dropbox and Onedrive have delta sync.
Seafile have delta sync, but it cause files broken.
I hope you see the rsync. I do need delta sync

[cowai]

@rullzer How would the client have the previous file for calculating the delta?

[Bigpet]

@cowai you either need to keep a copy of your last sync around (using file-system specific things like shadow copy seems out of the question for the broad range of platforms with sync clients) or you have to do block-level syncing instead, like "syncthing" does it.

[stratacast]

I think block-level syncing like syncthing is probably the easiest implementation in code, and perhaps the cheapest to write. I'm seriously interested in this, and I know some companies that are too (Quickbooks files man...ugly stuff). Like @Bigpet said, you'd need a copy of the file before changes onhand, or put some hooks into writes that go into that specific directory, but the latter sounds very messy and dangerous. I wish I knew how to write code better because I would 100% do this..I'm definitely a Kindergarten koder compared to a lot of people that put stuff on github. Thought I'd voice that there's interest on my end, and on the end of local companies I know.

[eglipeter]

Are there concrete plans when delta syncing will be available? May I hope to see this implemented in Nextcloud 13 already?

[gschenck]

There is some progress on owncloud:

owncloud/core#16162

[ahmedammar]

@gschenck Please feel free to try out the latest code, the core implementation should be complete now.

[jkaberg]

@ahmedammar any plans on submitting the PR against NC as well further down the road?

[ahmedammar]

@jkaberg once the work is complete and merged in oC I can have a look, assuming the code-base isn't too different at the core ...

[maverick74]

@ahmedammar can you give us an update about the feature? (If possible a probable ETA?)

[ahmedammar]

@maverick74 no ETA for nextcloud, if someone is willing to open a bounty for it I could look into it more urgently, otherwise, for reference:
owncloud/client#6131
owncloud/core#29404

[L00maca]

It's not much and I'm not even sure I did this right since I never did this before, but I don't mind chipping in to help this get done.

[jospoortvliet]

The bounty is already at 115 dollar now. It should not be terribly hard to get this merged in Nc client and server, I think, but it won't make it for 13 😄

[ahmedammar]

I won’t be looking into this until oC actually merge first, since that saves me any duplicated effort. Unless this bounty gets so big that I can ignore oC all together :)

[maverick74]

FWIW i guess there are some news at owncloud/core#29404

[wudimenghuan]

@maverick74 So It can be merged... @rullzer @jospoortvliet

[tflidd]

FWIW i guess there are some news at owncloud/core#29404

That's the server side. Client-side is still on a development branch and subject to testing (https://github.com/owncloud/client/labels/Delta-sync). Unless this is not finished, it doesn't make a lot of sense to merge anything at the moment, so you can only help testing it.

[petrk94]

I think nextcloud should hurry up, delta sync will be released in the next owncloud update:
https://owncloud.com/owncloud-implements-delta-sync-technology/

[jospoortvliet]

@petrk94 yeah, it could in theory be merged - but ownCloud notes it'll be in testing until 2019, let's see. @ahmedammar can make a PR for the server - the client will get it as we sync upstream actively still.

[petrk94]

Im wondering why I get so much thump down, just want to keep the thread updated :/

[jcklpe]

If I'm understanding stuff correctly it sounds like NextCloud won't be having this feature any time soon, correct?

[tflidd]

ownCloud currently uses client version 2.4. Version 2.5 is in beta tests now (owncloud/client#6483) and the delta sync feature was announced for version 2.6. Now with a bit of guessing, between major releases there are often 6 months or more, so I wouldn't expect a working client before the end of this year.

From Nextcloud side, they took over their own development to realize the new client-side encryption which is currently in beta status. This feature is one of the main priorities at the moment, and it will probably take some time to ship this feature and get it really stable. After that, they could implement delta sync but I won't expect it before mid-2019. This is no official statement, priorities can change ...

[maverick74]

Apparently the client-side is already merged ( owncloud/client#6297 )

But they're still hunting for bugs until 2.6.0.

It would be nice to have it as an experimental Opt-in feature over here, however :)

[jospoortvliet]

We might merge it during the course of our 2.6 development, I suppose - but we have a huge amount of things we want to work on, not sure how high the prio is on this one. Help is welcome - if somebody feels like creating a PR for our client that backports this feature, that'd be cool of course!