archive/zip: fix the size field in the ZIP64 end-of-central-directory record

[joeshaw]

I have an approximately 8 GB ZIP file created using archive/zip. zipdetails -v on the file ends with:

1FD0EB303 000000004 50 4B 01 02 CENTRAL HEADER #127F  02014B50
1FD0EB307 000000001 14          Created Zip Spec      14 '2.0'
1FD0EB308 000000001 03          Created OS            03 'Unix'
1FD0EB309 000000001 14          Extract Zip Spec      14 '2.0'
1FD0EB30A 000000001 00          Extract OS            00 'MS-DOS'
1FD0EB30B 000000002 08 00       General Purpose Flag  0008
                                [Bit  3]              1 'Streamed'
1FD0EB30D 000000002 00 00       Compression Method    0000 'Stored'
1FD0EB30F 000000004 16 A5 4B 46 Last Mod Time         464BA516 'Wed Feb 11 20:40:44 2015'
1FD0EB313 000000004 B0 92 2C B7 CRC                   B72C92B0
1FD0EB317 000000004 FF FF FF FF Compressed Length     FFFFFFFF
1FD0EB31B 000000004 FF FF FF FF Uncompressed Length   FFFFFFFF
1FD0EB31F 000000002 24 00       Filename Length       0024
1FD0EB321 000000002 1C 00       Extra Length          001C
1FD0EB323 000000002 00 00       Comment Length        0000
1FD0EB325 000000002 00 00       Disk Start            0000
1FD0EB327 000000002 00 00       Int File Attributes   0000
                                [Bit 0]               0 'Binary Data'
1FD0EB329 000000004 00 00 A4 81 Ext File Attributes   81A40000
1FD0EB32D 000000004 FF FF FF FF Local Header Offset   FFFFFFFF
1FD0EB331 000000024 38 36 34 20 Filename              '864 Pics
                    50 69 63 73                       1.68gbs/20140807_053444.jpg'
                    20 31 2E 36
                    38 67 62 73
                    2F 32 30 31
                    34 30 38 30
                    37 5F 30 35
                    33 34 34 34
                    2E 6A 70 67
1FD0EB355 000000002 01 00       Extra ID #0001        0001 'ZIP64'
1FD0EB357 000000002 18 00         Length              0018
1FD0EB359 000000008 D4 23 2A 00   Uncompressed Size   00000000002A23D4
                    00 00 00 00
1FD0EB361 000000008 D4 23 2A 00   Compressed Size     00000000002A23D4
                    00 00 00 00
1FD0EB369 000000008 B8 DA DD FC   Offset to Central   00000001FCDDDAB8
                    01 00 00 00 Dir

1FD0EB371 000000004 50 4B 06 06 ZIP64 END CENTRAL DIR 06064B50
                                RECORD
1FD0EB375 000000008 38 00 00 00 Size of record        0000000000000038
                    00 00 00 00
1FD0EB37D 000000001 2D          Created Zip Spec      2D '4.5'
1FD0EB37E 000000001 00          Created OS            00 'MS-DOS'
1FD0EB37F 000000001 2D          Extract Zip Spec      2D '4.5'
1FD0EB380 000000001 00          Extract OS            00 'MS-DOS'
1FD0EB381 000000004 00 00 00 00 Number of this disk   00000000
1FD0EB385 000000004 00 00 00 00 Central Dir Disk no   00000000
1FD0EB389 000000008 7F 12 00 00 Entries in this disk  000000000000127F
                    00 00 00 00
1FD0EB391 000000008 7F 12 00 00 Total Entries         000000000000127F
                    00 00 00 00
1FD0EB399 000000008 93 B4 06 00 Size of Central Dir   000000000006B493
                    00 00 00 00
1FD0EB3A1 000000008 DE FE 07 FD Offset to Central dir 00000001FD07FEDE
                    01 00 00 00
Unsupported Size (56) in Zip64EndCentralHeader

The zip was built using Go 1.4 or 1.4.1 on a 64-bit Linux system.

A Go reader on OS X can reread the file just fine. This is my first foray into the internals of zip, so I am not sure how or why 56 is an unsupported size.

Smaller files (in the 3.7 GB range) are opened fine by unzip and the built-in OS X zip support.

[joeshaw]

Hmm, one possibility: section 4.3.14.1 of https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT says:

The value stored into the "size of zip64 end of central directory record" should be the size of the remaining record and should not include the leading 12 bytes.

By my count the record's total size is 56 bytes, so the "Size of record" field should be 44 (0x2c) rather than 56 (0x38).

[joeshaw]

Ok, opening a hex editor and changing that 0x38 to a 0x2c makes zipdetails happy, but it might be a red herring. zip -FF fails in the same place regardless:

 Local ( 1 7313354790): copying: 864 Pics 1.68gbs/20150114_202208.jpg   (1875410 bytes)
 Local ( 1 7315230282): copying: 864 Pics 1.68gbs/20150114_202212.jpg
zip I/O error: Bad address
zip error: Output file write failure (write error on zip file)

as does unzip -l:

Archive:  RFM-rgriffin.zip
error [RFM-rgriffin.zip]:  missing 49368129 bytes in zipfile
  (attempting to process anyway)
error [RFM-rgriffin.zip]:  start of central directory not found;
  zipfile corrupt.
  (please check that you have transferred or created the zipfile in the
  appropriate BINARY mode and that you have compiled UnZip properly)

[joeshaw]

zipdetails -v output around the point where it dies:

1B4058E4A 000000004 50 4B 03 04 LOCAL HEADER #1029    04034B50
1B4058E4E 000000001 14          Extract Zip Spec      14 '2.0'
1B4058E4F 000000001 00          Extract OS            00 'MS-DOS'
1B4058E50 000000002 08 00       General Purpose Flag  0008
                                [Bit  3]              1 'Streamed'
1B4058E52 000000002 00 00       Compression Method    0000 'Stored'
1B4058E54 000000004 16 A5 4B 46 Last Mod Time         464BA516 'Wed Feb 11 20:40:44 2015'
1B4058E58 000000004 00 00 00 00 CRC                   00000000
1B4058E5C 000000004 00 00 00 00 Compressed Length     00000000
1B4058E60 000000004 00 00 00 00 Uncompressed Length   00000000
1B4058E64 000000002 24 00       Filename Length       0024
1B4058E66 000000002 00 00       Extra Length          0000
1B4058E68 000000024 38 36 34 20 Filename              '864 Pics
                    50 69 63 73                       1.68gbs/20150114_202212.jpg'
                    20 31 2E 36
                    38 67 62 73
                    2F 32 30 31
                    35 30 31 31
                    34 5F 32 30
                    32 32 31 32
                    2E 6A 70 67
1B4058E8C 0001D3FFF ...         PAYLOAD

1B422CE8B 000000004 50 4B 07 08 STREAMING DATA HEADER 08074B50
1B422CE8F 000000004 AF 15 10 F4 CRC                   F41015AF
1B422CE93 000000004 FF 3F 1D 00 Compressed Length     001D3FFF
1B422CE97 000000004 FF 3F 1D 00 Uncompressed Length   001D3FFF

[joeshaw]

I was able to decompress the whole zip without modification on a Linux box. I wonder if it's a Mac OS-only issue. unzip 6.00 on Linux versus 5.52 on OS X. I'll try installing a newer version on OS X and see if that's the issue.

[joeshaw]

Ok, what all of this has taught me is: large ZIP file support in OS X is garbage, and the failure modes are unkind.

The built-in archive utility doesn't support them, the version of unzip shipped with Yosemite doesn't include ZIP64 support, and the newer version that does support ZIP64 still blows up as soon as it hits 4 GB. Meanwhile, Linux works through the files happily.

The only thing that might make sense is to adjust the "size of zip64 end of central directory record" to make zipdetails happy, but it doesn't seem to affect other tools (unzip, zipinfo) on Linux.

[ianlancetaylor]

After your investigation I can't tell if there is anything for us to do here. Should I go ahead and close out this issue.

[joeshaw]

@ianlancetaylor: I think we should fix the size field in the ZIP64 end-of-central-directory record to match the spec. I'll write up a patch for that.

[joeshaw]

https://go-review.googlesource.com/4760 PTAL

[adg]

On 13 February 2015 at 09:09, Joe Shaw notifications@github.com wrote:

@ianlancetaylor https://github.com/ianlancetaylor: I think we should
fix the size field in the ZIP64 end-of-central-directory record to match
the spec. I'll write up a patch for that.

Sounds good. Please "git codereview mail
https://golang.org/doc/contribute.html" it to adg@golang.org when it's
ready.