New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use zstd adaptive mode? #4227

Closed
Daniel15 opened this Issue Dec 21, 2018 · 2 comments

Comments

Projects
None yet
2 participants
@Daniel15
Copy link

Daniel15 commented Dec 21, 2018

zstd v1.3.6 added an "adaptive" mode:

From https://code.fb.com/core-data/zstandard/:

The last feature introduced for large data streams is automatic level determination (--adapt). Adaptive mode measures the speed of the input and output files, or pipes, and adjusts the compression level to match the bottleneck. This mode can be combined with multithreading and long range mode and finely controlled through lower and upper bounds. Combining all three is perfect for server backups because the algorithm can opportunistically convert wait time due to network congestion into improved compression ratio. In initial experiments, we measured an improvement of approximately 10 percent in compression ratio for equivalent or better transmission time.

From https://github.com/facebook/zstd/releases/tag/v1.3.6:

A new command --adapt, makes it possible to pipe gigantic amount of data between servers (typically for backup scenarios), and let the compressor automatically adjust compression level based on perceived network conditions. When the network becomes slower, zstd will use available time to compress more, and accelerate again when bandwidth permit. It reduces the need to "pre-calibrate" speed and compression level, and is a good simplification for system administrators. It also results in gains for both dimensions (better compression ratio and better speed) compared to the more traditional "fixed" compression level strategy.

Not sure if Borgbackup uses this yet, but it sounds useful.

@ThomasWaldmann

This comment has been minimized.

Copy link
Member

ThomasWaldmann commented Dec 21, 2018

We only have ~2MiB sized chunks to compress, can be also much less if you have a lot of small files.

The description sounds a bit like it is implemented on the cli tool layer rather than in the core algorithm.

borg does not exec the cli tool, but directly calls into the zstd code via its library, giving a chunk of data (in memory) and receiving back the compressed data (also in memory). so there are no pipes, files that zstd is seeing.

@ThomasWaldmann

This comment has been minimized.

Copy link
Member

ThomasWaldmann commented Dec 27, 2018

I am closing this for now for the given reasons. If somebody finds a good way to use this via the C api of zstd and integrate it into borg, please give some hints and reopen.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment