Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Use zstd adaptive mode? #4227
zstd v1.3.6 added an "adaptive" mode:
Not sure if Borgbackup uses this yet, but it sounds useful.
We only have ~2MiB sized chunks to compress, can be also much less if you have a lot of small files.
The description sounds a bit like it is implemented on the cli tool layer rather than in the core algorithm.
borg does not exec the cli tool, but directly calls into the zstd code via its library, giving a chunk of data (in memory) and receiving back the compressed data (also in memory). so there are no pipes, files that zstd is seeing.