archive/zip: add support for extended timestamps

[bobjalex]

What version of Go are you using (go version)?

go version go1.8beta2 windows/amd64

What operating system and processor architecture are you using (go env)?

set GOARCH=amd64
set GOBIN=
set GOEXE=.exe
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOOS=windows
set GOPATH=C:\GoLib_beta
set GORACE=
set GOROOT=C:\Go_beta
set GOTOOLDIR=C:\Go_beta\pkg\tool\windows_amd64
set GCCGO=gccgo
set CC=gcc
set GOGCCFLAGS=-m64 -mthreads -fmessage-length=0 -fdebug-prefix-map=C:\Users\Bob\AppData\Local\Temp\go-build778111039=/tmp/go-build -gno-record-gcc-switches
set CXX=g++
set CGO_ENABLED=1
set PKG_CONFIG=pkg-config
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2

What did you do?

I use zip archives in contexts that require correct file times, and are required to be compatible with non-Go zip libraries. Go's default archive/zip writing is incompatible with many other zip libraries and programs (e.g. Java's and Python's provided libraries, the info-zip zip program distributed with Linux, ...).

The problem:

• Most zippers store local time into the main (MSDOS-FAT-type) header time fields, and Go stores
  UTC, causing times to be wrong when unzipped by other programs.
• Go 8 stores UTC into the extended time fields, which is universal and a good thing (Go 7 does
  not store an extended time field).
• Before Go 8, I could work around the issue by offsetting the file dates to cause local time to be stored.
• With Go 8, I can't do that because it also offsets the extended time, which is incorrect.
• The typical way that zippers work is to store local regular times and, if extended times are
  supported, UTC extended times.
• But I can't make that happen with archive/zip in Go 8. It unconditionally stores the same time in
  both extended and non-extended fields -- UTC.
• Since some unzippers use extended times and some don't, results are inconsistent,
  making the problem is actually worse in Go 8 than in prior versions.

Some solutions:
(1) Change archive/zip to store local non-extended times.
- But that might violate the Go 1 principle (???)
(2) Provide an option (in a Go 1 compatible way) that specifies whether the main non-extended
time fields should be local or UTC, defaulting to the current way. Like a public boolean field in
the writer and reader.

I favor (2), since it is flexible and would not violate the Go 1 principle. Default operation would be exactly as pre-Go 8 versions.

What did you expect to see?

Extended time stored as UTC and old (main) time fields stored as local time.

What did you see instead?

All file times stored as UTC.

[dsnet]

This was something that was annoying me about the current zip.FileHeader. I think it was an API mistake for there to be a ModifiedTime and ModifiedDatefield storing the time as MS-DOS formatted uint16s.

The path forward is probably to introduce another field ModTime time.Time and deprecate the MS-DOS timestamps in the same way we dealt with the uint32 sized fields.

EDIT: We'll need a different name since ModTime is already taken as the field name.

[dsnet]

Also, to understand your problem more clearly. Is this problem something blocking you for Go 1.8 release? I don't think we can improve time handling in the zip package for the remainder of Go 1.8, but if this is problematic, we can revert the current change and go for a more complete round-trip fix in Go 1.9.

[bradfitz]

Let's partially roll this back for 1.8 and fix it fully in 1.9.

[bobjalex]

This problem is not blocking me -- I have several ways to work around it.

In fiddling with this problem, I created a new zip module derived from the 1.8 one that solves the problem and is compatible with older versions. I added 2 new function that allow specification of whether the MSDOS 16-bit time and date fields should store as UTC (as it does now) or local (as many existing zip programs and libraries do):
func FileInfoHeaderLocal(fi os.FileInfo, localTime bool) (*FileHeader, error) // for writing
func OpenReaderLocal(name string, localTime bool) (*ReadCloser, error) // for reading

That seems to be all that is necessary (with an handful of other hidden code changes to make that work, but not very much, really).

My new version seems to be working OK for my purposes, but has not been exhaustively QA'd.

I attempted to attach my code -- feel free to use these changes if suitable, or point out improvements.

newzip.zip

[dsnet]

In light of #18378, I'm planning on reverting the whole change and just fix it fully for Go 1.9.

[gopherbot]

CL https://golang.org/cl/34651 mentions this issue.