Voodoo zpaq level 1 threads and RAM

Just a bit more infos

Multithread

Blocks are compressed or decompressed in parallel in separate threads. zpaqfranz automatically detects the number of processor cores and uses all of them in the 64 bit version or at most 2 in the 32 bit version (which is limited to 2 GB memory). You can use the -threads option to change the number of threads. The files are readed one by one (single thread) and written out by a single thread too.

Compression

zpaq has 5 compression levels+1 (no compression).
For the real die hard

• method 0. No compression (there's not much to say. Normally it is used with encrypted files, for which you only want to perform deduplication, such as truecrypt/veracrypt containers or similar)
• method 1. The default is the fastest. It is best for backups where you compress often and extract rarely. Suggested in almost every case. Uses LZ77, compressing by replacing duplicate strings with pointers to previous occurrences. Use about (compression) 128 MB and 32 MB (decompression) for thread
• method 2 compresses slower but decompresses as fast as -method 1. It is best for distributing files where you compress once and extract often. The same as -m1 but spends more time looking for better matches (using a suffix array instead of a hash table). Use about (compression) 450 MB and 128 MB (decompression) for thread
• method 3 uses either BWT (context sorting) or LZ77 for long matches and an order 1 context model and arithmetic coding for literals depending on the file type and eat RAM. Use about (compression) 450 MB and 400 MB (decompression) for thread
• method 4 very good,either uses LZ77, BWT or a high order context model, but huge RAM eater. Use about (compression) 550 MB and 550 MB (decompression) for thread
• method 5 uses a complex, high order context mixing model with over 20 bit prediction components, but loooong time Use about (compression) 850 MB and 850 MB (decompression) for thread

All methods except 5 test whether the data appears to be compressible or already compressed (random). Uncompressible data is simply stored.
An E8E9 filter is applied if x86 data (normally found in .exe and .dll files) is detected. The filter replaces x86 CALL and JMP relative addresses with absolute addresses to make the data more compressible.

Deduplication

For the real die hard

The block size is 16 MB for method 1 and 64 MB for higher methods. You can change the block size to trade compression for memory usage. Very small block size means very good deduplication, but higher overhead.
Estimates need to be increased for the RAM needed to maintain the fragment list (it grows as the size of the fragments decreases), the CRC32 list (but it is small) and the file name list (it can be large enough for millions of files)

RAM usage with multithread

Since each thread uses a certain amount of memory, and that many processors have more virtual threads than physical ones (hyperthreads) it can happen to have machines with a large number of threads, for example 32 or 128 or more.
In this case RAM usage can become massive (roughly up to 1GB per thread), causing large use of virtual memory.
This is why, in general, I don't recommend changing the compression level from the default. Obviously no one forbids it, in fact it is very fun to "play" with the low level parameters (of which there are many)