Extraction of 5GB (ZIP64) archive created with Go failed

[philip-firstorder]

To reproduce you need to zip a file bigger than 4GB with another smaller file, in my case my resulting archive it 5GB.

ZIP Method: Storage
Archiver: https://golang.org/src/archive/zip/writer.go

Error when unarchiving:

I managed to unarchive from terminal using the unzip command. Also using The Unarchiver (Mac), 7zip (Windows). So the archive is not corrupted.

Here are the headers I printed from terminal.

$ zipinfo -v F3.zip

Archive:  F3.zip
There is no zipfile comment.

End-of-central-directory record:
-------------------------------

  Zip archive file size:                5928918633 (0000000161641E69h)
  Actual end-cent-dir record offset:    5928918532 (0000000161641E04h)
  Expected end-cent-dir record offset:  5928918532 (0000000161641E04h)
  (based on the length of the central directory and its expected offset)

  This zipfile constitutes the sole disk of a single-part archive; its
  central directory contains 2 entries.
  The central directory is 270 (000000000000010Eh) bytes long,
  and its (expected) offset in bytes from the beginning of the zipfile
  is 5928918262 (0000000161641CF6h).


Central directory entry #1:
---------------------------

  Archive 5GB.zip

  offset of local header from start of archive:   0
                                                  (0000000000000000h) bytes
  file system or operating system of origin:      MS-DOS, OS/2 or NT FAT
  version of encoding software:                   2.0
  minimum file system compatibility required:     MS-DOS, OS/2 or NT FAT
  minimum software version required to extract:   4.5
  compression method:                             none (stored)
  file security status:                           not encrypted
  extended local header:                          yes
  file last modified on (DOS date/time):          2019 Jun 19 19:34:28
  file last modified on (UT extra field modtime): 2019 Jun 19 19:34:28 local
  file last modified on (UT extra field modtime): 2019 Jun 19 17:34:28 UTC
  32-bit CRC value (hex):                         06691a94
  compressed size:                                5720868503 bytes
  uncompressed size:                              5720868503 bytes
  length of filename:                             15 characters
  length of extra field:                          37 bytes
  length of file comment:                         40 characters
  disk number on which file begins:               disk 1
  apparent file type:                             binary
  non-MSDOS external file attributes:             000000 hex
  MS-DOS file attributes (00 hex):                none

  The central-directory extra field contains:
  - A subfield with ID 0x5455 (universal time) and 5 data bytes.
    The local extra field has UTC/GMT modification time.
  - A subfield with ID 0x0001 (PKWARE 64-bit sizes) and 24 data bytes:
    97 86 fd 54 01 00 00 00 97 86 fd 54 01 00 00 00 00 00 00 00 00 00 00 00.

------------------------- file comment begins ----------------------------
5d0105dcabf05d83dd8014bb/Archive 5GB.zip
-------------------------- file comment ends -----------------------------

Central directory entry #2:
---------------------------

  There are an extra -4 bytes preceding this file.

  DJI 0044.mov

  offset of local header from start of archive:   5720868581
                                                  (0000000154FD86E5h) bytes
  file system or operating system of origin:      MS-DOS, OS/2 or NT FAT
  version of encoding software:                   2.0
  minimum file system compatibility required:     MS-DOS, OS/2 or NT FAT
  minimum software version required to extract:   2.0
  compression method:                             none (stored)
  file security status:                           not encrypted
  extended local header:                          yes
  file last modified on (DOS date/time):          2019 Jun 24 17:04:38
  file last modified on (UT extra field modtime): 2019 Jun 24 17:04:39 local
  file last modified on (UT extra field modtime): 2019 Jun 24 15:04:39 UTC
  32-bit CRC value (hex):                         68b23c95
  compressed size:                                208049614 bytes
  uncompressed size:                              208049614 bytes
  length of filename:                             12 characters
  length of extra field:                          37 bytes
  length of file comment:                         37 characters
  disk number on which file begins:               disk 1
  apparent file type:                             binary
  non-MSDOS external file attributes:             000000 hex
  MS-DOS file attributes (00 hex):                none

  The central-directory extra field contains:
  - A subfield with ID 0x5455 (universal time) and 5 data bytes.
    The local extra field has UTC/GMT modification time.
  - A subfield with ID 0x0001 (PKWARE 64-bit sizes) and 24 data bytes:
    ce 95 66 0c 00 00 00 00 ce 95 66 0c 00 00 00 00 e5 86 fd 54 01 00 00 00.

------------------------- file comment begins ----------------------------
5d013d45409327f33177b621/DJI 0044.mov
-------------------------- file comment ends -----------------------------

[aonez]

First time using Go. Can you provide the script used for the compression?

[aonez]

@philip-firstorder I just need an example that sets the Store method.

[aonez]

Maybe I'm just missing how to call the bundled zip package in Go?

[aonez]

I'll take a look at #422 once I can create Store packages

[philip-firstorder]

Just use this: https://golangcode.com/create-zip-files-in-go/
And change to from zip.Deflate to zip.Store
header.Method = zip.Store

[aonez]

Did a quick test and it results in a corrupted file. I'm gonna do some more tests.

[philip-firstorder]

If you can manage to unarchive from terminal using the unzip command. Or using The Unarchiver (Mac/7zip (Windows), then the archive is not really corrupted.

[aonez]

My test consist in a big MKV file (~5GB) and another file of indifferent size, similar to the test you've reported.

Both 7-Zip and The Unarchiver warn there's some corruption. The Unarchiver, after dismissing the warning, extracts properly the big file and ignores the little one while 7-Zip extracts both. unzip works and throws no warning/error.

Using Keka in the default configuration throws an error message and the result files are the same as with The Unarchiver, since it uses unar. If I set it to use p7zip instead (defaults write com.aone.keka UnzipWithUNAR false), it also throws an error but the result is the same as in 7-Zip, all files are extracted.

So even if the file can be read and extracted, I think either the script or the Go library is creating bad headers.

[philip-firstorder]

Thank you for your tests, It’s very good you could reproduce the error.

If the go library creates bad headers, then we need to find out what those are. Can you compare their headers with a valid archive to see the differences?

[philip-firstorder]

I did more tests archiving/unarchiving an Archive 5GB.zip file together with a Small image.png here are the unarchiving results:

Go.zip: archived with Go

Archive Utility (10.11): Error 2 - No such file or directory => Nothing extracted
BetterZip (4.2.4): Both files successfully extracted in the temp folder, but at the end I get the error => The archive is either damaged or in an unsupported format.
The transcript window may contain further details => Then after pressing OK on the error dialog the temp folder is deleted
Keka unar (1.1.17): Error code 1 using "kekaunar" Unknown error => Archive 5GB.zip is successfully extracted, but not Small image.png
Keka p7zip (1.1.17): Error code 2 using "p7zip" Fatal error => Both files successfully extracted in the Go folder
Keka unar (1.1.18): Both files successfully extracted in the Go folder
Keka p7zip (1.1.18): Error code 2 using "p7zip" Fatal error => Both files successfully extracted in the Go folder
Stuffit Expander (16.0.6): No errors => Both files successfully extracted
The Unarchiver (4.1.0): There was a problem while reading the contents of "Go.zip": Data is corrupted => clicked Continue => Archive 5GB.zip is extracted, but not Small image.png
Terminal UnZip (6.00) $ unzip Go.zip: No errors => Both files successfully extracted

Keka.zip: archived with Keka (1.1.17)

Archive Utility (10.11): Error 2 - No such file or directory => Nothing extracted
BetterZip (4.2.4): Both files successfully extracted
Keka unar (1.1.17): No errors => Both files successfully extracted
Keka p7zip (1.1.17): No errors => Both files successfully extracted in the Keka folder
Stuffit Expander (16.0.6): Expansion failed. The structure of the archive is damaged => Nothing extracted
The Unarchiver (4.1.0): There was a problem while reading the contents of "Keka.zip": Data is corrupted => clicked Continue => Archive 5GB.zip is extracted, but not Small image.png
Terminal UnZip (6.00) $ unzip Keka.zip: No errors => Both files successfully extracted

Archive Utility.zip: archived with Finder (macOS 10.14.5)

Archive Utility (10.11): No errors => Both files successfully extracted
BetterZip (4.2.4): The archive is either damaged or in an unsupported format. => Archive 5GB has size on disk only 1.43GB, no file can be extracted
Keka unar (1.1.17): Error code 1 using "kekaunar". Unknown error => Archive 5GB has only 1.42GB extracted, Small image.png is fully extracted in the Archive Utility folder
Keka p7zip (1.1.17): Error code 2 using "p7zip" Fatal error => Nothing extracted
Stuffit Expander (16.0.6): Expansion failed. The structure of the archive is damaged => Nothing extracted
The Unarchiver (4.1.0): Could not extract "Archive 5GB.zip" from the "Archive Utility.zip" archive: The archive file is incomplete => clicked Continue => Archive 5GB has only 1.42GB extracted, Small image.png is fully extracted in the Archive Utility folder
Terminal UnZip (6.00) $ unzip "Archive Utility.zip": See errors below => Archive 5GB has only 1.42GB extracted, Small image.png is fully extracted, empty __MACOSX folder extracted next to the other files in the root

warning [Archive Utility.zip]:  4294967296 extra bytes at beginning or within zipfile
  (attempting to process anyway)
file #1:  bad zipfile offset (local header sig):  4294967296
  (attempting to re-compensate)
  inflating: Archive 5GB.zip         
  error:  invalid compressed data to inflate
file #2:  bad zipfile offset (local header sig):  1417650120
  (attempting to re-compensate)
   creating: __MACOSX/
  inflating: __MACOSX/._Archive 5GB.zip  
  inflating: Small image.png         
  inflating: __MACOSX/._Small image.png