zstd rsyncable option

[terrelln]

We're considering adding the --rsyncable option to zstd, but we want to make sure that we're not solving a problem that no one has. We imagine it being useful for multi-gigabyte backups with chunks of a few MB. Does anyone want this features, and if so what is your use case?

[sachk]

I'd like it for getting a bit more bandwidth transferring over GbE with SSDs.

[junkhacker]

another benefit to this would be for compressed files stored on deduplicated file-systems, assuming the same benefit as gzip --rsyncable. (before anyone says anything about deduplicated systems being more efficient with uncompressed files, sometimes files are compressed due to how they're generated/used and should be kept that way)

[Orteko]

One use case to consider is virtual machine backups - we currently use gzip with --rsyncable on a local LVM snapshot to create a local dump and then rsync over slow network to a remote host.

[terrelln]

Thanks for the feedback! It seems like there is definitely a use case for --rsyncable, and I'll make sure it gets in the next release.

[terrelln]

@Orteko @junkhacker @sachk and anyone else who cares.

I've just merged rsyncable mode into the development branch. It is enabled using zstd --rsyncable. We use a larger chunk size than gzip (depending on the compression level). If you want to try it out, I'd love any feedback you have.

[sachk]

Thanks, this should be really useful for me.

[Orteko]

Thank you - awesome work!

Just built from head and gave it a test run - everything working as expected for me.

[cemeyer]

Sorry to hijack the PR thread. Is there any documentation for what this option does, specifically, or the mechanism by which it is intended to work? I am having trouble finding it.

[terrelln]

Normally, when you compress a file, even a 1 byte change can cause the entire compressed file afterwards to be completely different. This isn’t great for rsync, since it will only upload the delta, but the delta will be the whole file even for a 1 byte change.

The rsyncable mode adds “synchronization points” to the compressed file, where the compression state gets reset. So if you add/remove/modify only one part of a file, the compressed file will only differ up to the next synchronization point. This makes the compressed file much more delta-able.

We could probably improve the man page.

[cemeyer]

@terrelln Thanks. Are synchronization points essentially independent frames / blocks (i.e., seekable zstd, minus the index)? And are they sized to line up at some sort of block boundaries? (I don't know how clever rsync's delta detection policy is.)