Zstandard as default compression method, replacing deflate

[benbenik]

#3708 Environment info

• Duplicati version: 2.0.5.1_beta_2020-01-18
• Operating system: Windows 10
• Backend: WebDAV

Description

The default compression method in Duplicati is deflate (practically gzip).
A modern outperformer is Zstandard/ Zstd.
Zstd is able to get comparable compression ratios at much higher speeds, both for compression and decompression.
But Zstd can be used for any use case: either higher compression at lower speeds or vice versa.
Thus, Zstd is a versatile compression method that vastly outperforms the currently used deflate, with flexibility.
My request is to include Zstandard as the default compression method at a setting that delivers a comparable compression ratio as deflate. For example: zstd level 4 gives a 500% speed improvement and even a minor compression improvement, compared to deflate.

Steps to reproduce

https://community.centminmod.com/threads/round-3-compression-comparison-benchmarks-zstd-vs-brotli-vs-pigz-vs-bzip2-vs-xz-etc.17259/

• Actual result:
  Deflate is the default. The bottleneck in backing up is deflate. Or at equal deflate speed, a higher ratio can be achieved with Zstd.
• Expected result:
  Zstandard is the default compression method at a sane default, e.g. level 4. The bottleneck does not lie with deflate anymore, but with transfer speeds of modern Internet connections, 1Gb networks and conventional HDDs. It will also be possible set a higher Zstd level to increase compression ratio, while still being faster than deflate.

Screenshots

Have a look at: https://community.centminmod.com/threads/round-3-compression-comparison-benchmarks-zstd-vs-brotli-vs-pigz-vs-bzip2-vs-xz-etc.17259/

Debug log

None.

[warwickmm]

There is some related discussion in #2575.

[duplicatibot]

This issue has been mentioned on Duplicati. There might be relevant details there:

https://forum.duplicati.com/t/zstd-compression/10241/4

[schtritoff]

There is a c# port of zstd available, which should make it much easier to implement in duplicati. Maybe promising https://github.com/oleg-st/ZstdSharp

There is streaming option available https://github.com/oleg-st/ZstdSharp/blob/master/src/ZstdSharp.Test/ZstdNetSteamingTests.cs but zstd library is not a complete solution since it lacks container format. The usual way of using zstd is to make tar from files and then compress it ".tar.zst". There are examples where 7z or zip can use zstd compression (for example https://github.com/mcmilk/7-Zip-zstd).

One way to go would be to first implement zstd in upstream project https://github.com/adamhathcock/sharpcompress/blob/master/src/SharpCompress/Common/CompressionType.cs and then to change how SharpCompress is used in here.

Other way would be to implement zstd directly in duplicati (interface https://github.com/duplicati/duplicati/blob/master/Duplicati/Library/Interface/ICompression.cs) but I don't know how one would implement container format (eg zip, tar, 7z, etc).

[yburkouski]

Why no any progress on it? Zstd is a modern compression alg used in an industry, it has benefits in any aspects: it compress as good as 7z (if you need) but much faster, it can compress as lz4 with a good speed, it can compress as a zip with a better speed, the decompression is also faster then for zip/7zip. So it’s a universal alg that can work as lz4-zip-7zip but with a better speed or compression ratio.

All modern browsers and sites not using zip/deflate, they using ‘brotli’ instead cause it gives10% better compression for free. So I’m trying to say that we are not living in 2000 and many things in data compression changed since, better to use modern solutions for the app that is working with a data.

[ts678]

Why no any progress on it?

Lack of volunteer developers, I would think. There are lots of issues (including some backup corruption) that rank higher IMO.
Not every developer does compression either. See About --> Libraries. Look at the SharpCompress library that Duplicati uses:

Add support for zstd as a compression algorithm #223 that got declined in 2017 then picked up in 2021. 2022 status is at PR
Zstandard support #627 and you ever notice a release, maybe Duplicati will use it. Meanwhile progress proceeds in the library.

You might note that I'm presenting sort of the path below from post before yours, except we need somebody to implement it:

One way to go would be to first implement zstd in upstream project https://github.com/adamhathcock/sharpcompress/blob/master/src/SharpCompress/Common/CompressionType.cs and then to change how SharpCompress is used in here.