🐞 ZPAQ Compression/Decompression Broken

[pete4abw]

lrzip-next Version

0.7.63+ Maybe earlier

lrzip-next command line

lrzip-next -d | -t file.tar.lrz

What happened?

Decompressing a ZPAQ lrzip-next archive fails with an MD5 mismatch. Stored MD5s are OK. The decompressed file is broken. This came out of the blue and is a new issue.

What was expected behavior?

MD5s should match. Stored MD5s are correct.

Steps to reproduce

Create a ZPAQ lrzip-next archive
Decompress or Test

Relevant log output

Detected lrzip version 0.7 file.
100%    1363.42 /   1363.42 MB  1:100%  2:100%  3:100%  4:100%  5:100%  6:100%  7:100%  8:100%  9:100%  
Average DeCompression Speed: 19.471MB/s

MD5 CHECK FAILED.
Stored:134a2942867678fa1c3d4284c8b738b2
Output file:901042f7a62b3b7493fc3e885c60af47
Fatal error - exiting

Please provide system details

Slackware-x86_64 current
Kernel Version (uname -a): 5.13.2+
System ram (free -h): $ free -h
total used free shared buff/cache available
Mem: 15Gi 895Mi 11Gi 341Mi 3.0Gi 13Gi
Swap: 15Gi 654Mi 15Gi

Additional Context

Trying to pin down when this broke. ZPAQ code has not changed for a while. Made some changes to the temp buffer prior to writing out.

[pete4abw]

It appears that the 7.15 version of zpaq fails when a block of >64MB is sent to the backend on compression. Forcing a limit of 64MB appears to fix the problem. The issue was with compression, but remains unclear.

Taking decompressed data from thread 6
Taking decompressed data from thread 7
Taking decompressed data from thread 8
100%    1363.42 /   1363.42 MB
Closing stream at 156,800,471, want to seek to 156,800,471

Average DeCompression Speed: 20.044MB/s

MD5: 134a2942867678fa1c3d4284c8b738b2
[OK] - 1,429,647,360 bytes                                
Total time: 00:01:08.62

Some additional testing is required.

[pete4abw]

See new branch zpaq_fix to try out the update. After a review period these changes will be merged. Issue will be reported upstream.

[pete4abw]

Additional testing has revealed that this statement:

It appears that the 7.15 version of zpaq fails when a block of >64MB is sent to the backend on compression. Forcing a limit of 64MB appears to fix the problem. The issue was with compression, but remains unclear.

Is not entirely correct. The issue seems to be that if the block sent to the ZPAQ backend is greater than than the block size specified, ZPAQ may fail. This does not happen for smaller files that are less than ZPAQ block size. Thus. if the ZPAQ block size is specified as 9, which is 512MB, any block sent must be less than that. There is some code in the code in libzpaq.cpp which defines block size as 1MB << bs - 4096.:

void compress(Reader* in, Writer* out, const char* method,
              const char* filename, const char* comment, bool dosha1) {

  // Get block size
  int bs=4;
  if (method && method[0] && method[1]>='0' && method[1]<='9') {
    bs=method[1]-'0';
    if (method[2]>='0' && method[2]<='9') bs=bs*10+method[2]-'0';
    if (bs>11) bs=11;
  }
  bs=(0x100000<<bs)-4096;

Limiting the block size sent to the ZPAQ backend to 1MB << bs - 4096 seems to keep compression from getting corrupted. Testing is ongoing. Higher block sizes seems to cause intense swap space use.

[pete4abw]

Looks good. Commits pushed to master and lzma-21.03beta.