tl;dr, you can construct a zip file that has different contents depending on which decoder you use. archive/zip currently uses an interpretation inconsistent with the widely-popular libarchive, but consistent with WinZip and PKWARE's Zip Reader. We should align with libarchive.
A zip archive contains "central directory" (henceforth CD) followed by an "end of central directory record" (henceforth EOCDR). The EOCDR contains fields indicating the size and offset of the CD.
Prior to Go 1.19, archive/zip assumed that a zip archive begins at the start of the file that contains it, locating the central directory at the file offset given in the EOCDR.
Some ecosystems make use of files containing some non-zip content concatenated with the zip archive. There was a long-standing feature request to support these files, #10464, which was implemented in Go 1.19 in https://go.dev/cl/387976. However, this change meant that while Go 1.19 could read some archives that earlier versions could not, it also could not read some archives that earlier versions could. https://go.dev/cl/408734 and https://go.dev/cl/461598 further adjusted the behavior of archive/zip to attempt to support both cases. This is the source of the parser differential.
The zip specification (PKWARE's APPNOTE.TXT) does not, so far as I can tell, describe how to handle the case of an archive appended to some non-zip content.
For simplicity, I'm going to name two incompatible approaches to finding the CD of a zip file:
- "Base-zero": The zip archive begins at the start of the file. The CD is located at the file offset given in the EOCDR.
- "Computed-base": The file may contain some data concatenated with a zip archive. The CD is located at the offset of the EOCDR, minus the size of the CD, minus the offset of the CD, where the size and offset are given in the EOCDR.
I surveyed a variety of implementations. The open-source implementations I examined (libarchive, Info-Zip, and Python's ziplib) all use computed-base. WinZip and PKWARE's Zip Reader appear to use base-zero. Go 1.18 and earlier use base-zero. Go 1.19 and later use a hybrid: Base-zero if the file seems valid under that interpretation, computed-base otherwise.
https://go.dev/cl/461598 includes a mention of "a file in the wild" that could only be parsed under the base-zero implementation. Unfortunately, I haven't been able to find what that file was. This does seem to indicate that the current behavior is intentional and was seen as necessary to avoid breaking backwards compatibility.
The parser misalignment was reported to us as a potential vulnerability (thanks to Jakub Ciolek for the report). After some consideration, our current conclusion is that there does not appear to be consensus on the correct behavior. While zip implementations in common use on Unix platforms (libarchive, Info-Zip, ziplib) align in behavior, WinZip remains highly popular on Windows platforms and it's difficult to argue that PKWARE's implementation is anything other than canonical. No matter what interpretation we choose, we will be misaligned with at least one major implementation.
In addition, while we have lost context on exactly what breakage https://go.dev/cl/461598 was intended to avoid, it indicates that treating this as a security fix to be backported to minor releases runs the risk of breaking existing users.
We feel, therefore, that we should treat this as a hardening measure and (probably) align archive/zip with libarchive in the next major Go release.
tl;dr, you can construct a zip file that has different contents depending on which decoder you use. archive/zip currently uses an interpretation inconsistent with the widely-popular libarchive, but consistent with WinZip and PKWARE's Zip Reader. We should align with libarchive.
A zip archive contains "central directory" (henceforth CD) followed by an "end of central directory record" (henceforth EOCDR). The EOCDR contains fields indicating the size and offset of the CD.
Prior to Go 1.19, archive/zip assumed that a zip archive begins at the start of the file that contains it, locating the central directory at the file offset given in the EOCDR.
Some ecosystems make use of files containing some non-zip content concatenated with the zip archive. There was a long-standing feature request to support these files, #10464, which was implemented in Go 1.19 in https://go.dev/cl/387976. However, this change meant that while Go 1.19 could read some archives that earlier versions could not, it also could not read some archives that earlier versions could. https://go.dev/cl/408734 and https://go.dev/cl/461598 further adjusted the behavior of archive/zip to attempt to support both cases. This is the source of the parser differential.
The zip specification (PKWARE's APPNOTE.TXT) does not, so far as I can tell, describe how to handle the case of an archive appended to some non-zip content.
For simplicity, I'm going to name two incompatible approaches to finding the CD of a zip file:
I surveyed a variety of implementations. The open-source implementations I examined (libarchive, Info-Zip, and Python's ziplib) all use computed-base. WinZip and PKWARE's Zip Reader appear to use base-zero. Go 1.18 and earlier use base-zero. Go 1.19 and later use a hybrid: Base-zero if the file seems valid under that interpretation, computed-base otherwise.
https://go.dev/cl/461598 includes a mention of "a file in the wild" that could only be parsed under the base-zero implementation. Unfortunately, I haven't been able to find what that file was. This does seem to indicate that the current behavior is intentional and was seen as necessary to avoid breaking backwards compatibility.
The parser misalignment was reported to us as a potential vulnerability (thanks to Jakub Ciolek for the report). After some consideration, our current conclusion is that there does not appear to be consensus on the correct behavior. While zip implementations in common use on Unix platforms (libarchive, Info-Zip, ziplib) align in behavior, WinZip remains highly popular on Windows platforms and it's difficult to argue that PKWARE's implementation is anything other than canonical. No matter what interpretation we choose, we will be misaligned with at least one major implementation.
In addition, while we have lost context on exactly what breakage https://go.dev/cl/461598 was intended to avoid, it indicates that treating this as a security fix to be backported to minor releases runs the risk of breaking existing users.
We feel, therefore, that we should treat this as a hardening measure and (probably) align archive/zip with libarchive in the next major Go release.