Disable compression

[rkarlsba]

I have:

• searched open and closed issues for duplicates

---

Version info

Duplicati Version: 2.0.2
Operating System: Debian Stretch
Backend: Jotta

Bug description

Duplicati spends most of its time attempting to compress data already compressed, such as pictures, films or other. This is stored in tmp and then sent upstream. This is a total waste of time, since it really is no use to compress things that already are compressed. I would like to disable compression altogether, since the compressable data here accounts for perhaps one percent of the total.

Steps to reproduce

• Start a backup of uncompressable data and watch while duplicati spends hours trying to compress it further

Actual result: Describe here what happens after you run the steps above (i.e. the buggy behaviour)

Waste of time

Expected result: Describe here what should happen after you run the steps above (i.e. what would be the correct behaviour)

Backup the data, don't spend time trying to compress it.

Screenshots

debug log

[kees-z]

AFAIK, Duplicati doesn't do any compression for file types that can't be compressed further. It uses the text file default_compressed_extensions.txt to decide if the blocks for a certain file extension must be compressed, before storing them in the upload volume file.
You can specify a custom Compressed extensions file with advanced option --compression-extension-file.

[rkarlsba]

thanks - how can I just disable the lot? I don't want dedup either

[rkarlsba]

perhaps we should have a common flag for "act dumb, don't be smart", something like --trump

[kees-z]

how can I just disable the lot? I don't want dedup either

Use another product. Duplicati is a block-based backup solution. That's something completely different than file synchronization.
Look here for an explanation how Duplicati works, read the FAQ for the official answer to your question about synchronization.

[rkarlsba]

Well, there aren't any other products that support the amounts of backends Duplicati can use. I'm just trying to find a way to backup my stuff before Crashplan removes my backup. Regardless of being block- or filebased, it shouldn't be hard to disable compression and dedup. Those are, after all, features, not integral desgn related to the backends.

[rkarlsba]

And btw, running dedup on a client is just a very bad idea. Dedup works if you run it on a massive set of data on the server side as a separte job, usually during the night.

[kees-z]

Regardless of being block- or filebased, it shouldn't be hard to disable compression and dedup

It certainly is. Disabling compression is probably not what you ask for. If I understand you correctly, you want to disable breaking the source files into blocks and storing these blocks into container files, but just uploading the sources files directly to the backend.
That would break almost everything Duplicati offers (deduplication, versioning) and would require a complete rewrite of the code.

running dedup on a client is just a very bad idea. Dedup works if you run it on a massive set of data on the server side as a separte job, usually during the night

No, Duplicati does deduplication on the fly. Before a new backup task is started, Duplicati knows what blocks are on the backend. If a block is uploaded earlier, Duplicati will not re-upload that block to the backend, but just reference to that block in an index file. Result is that you have a lot of full backups available, using a fraction of the storage space needed to store the files multiple times.
And last but not least, I assume you would not want to upload 5TB every day, so deduplication isn't such a bad idea.

[rkarlsba]

I guess disabling compression will do. How can I do this? Does the default_compressed_extensions.txt file support regex, so I can just toss in a .* to it to drop compression altogether?

[kees-z]

--zip-compression-method=None will disable compression, --zip-compression-level=0 will just store the data in zip files without compression:

Duplicati.CommandLine.exe help zip
Zip compression (.zip):
This module provides the industry standard Zip compression. Files created with this module can be read by any
standard-compliant zip application.
Supported options:
--zip-compression-level (Enumeration): Sets the Zip compression level
This option controls the compression level used. A setting of zero gives no compression, and a setting of 9 gives maximum compression.
* values: 0, 1, 2, 3, 4, 5, 6, 7, 8, 9
* default value: BestCompression
--compression-level (Enumeration): Sets the Zip compression level
[DEPRECATED]: Please use the zip-compression-level option instead
This option controls the compression level used. A setting of zero gives no compression, and a setting of 9 gives maximum compression.
* values: 0, 1, 2, 3, 4, 5, 6, 7, 8, 9
* default value: BestCompression
--zip-compression-method (Enumeration): Sets the Zip compression method
This option can be used to set an alternative compressor method, such as LZMA. Note that using another value than Deflate will cause the zip-compression-level option to be ignored.
* values: None, GZip, BZip2, PPMd, Deflate, Rar, LZMA, BCJ, BCJ2, LZip, Unknown
* default value: Deflate
--zip-compression-zip64 (Boolean): Toggles Zip64 support
The zip64 format is required for files larger than 4GiB, use this flag to toggle it
* default value: False

[kenkendk]

I am closing this as I think the original question has been answered (thanks @kees-z )