cmd/go: timestamp of files in module archive unset #34953
Comments
This is the behavior of |
dmitshur
changed the title
dmitshur
added
GoCommand
|
I took a quick look in https://play.golang.org/p/HOhsuCVXEuP https://github.com/golang/go/blob/master/src/archive/zip/struct.go#L128 |
|
Another small thing I found was that all files in the module from |
|
The I believe @jayconrod has an open bug somewhere to consider making that code available as a library. Other than that, I'm not sure what there is to do here. |
|
CL 202042 is a draft CL that moves module zip handling code to x/mod. It documents the restrictions on zip files that the Go command accepts and provides a way to build and extract zip files without We don't currently save any metadata other than file names, and we don't set file modes or dates when we extract files into the module cache. I'm not sure there's any concrete reason for this. Storing and setting timestamps and executability would not change module hashes, which only cover file names and content. It would change the SHA-256 hash of generated zip files. We haven't promised those are stable, but changing them would probably surprise some people. I would like |
I think we should actively document that they're not guaranteed to be stable, so that people don't begin to expect them to be stable. Otherwise, we'd be locking in the exact zip file format encoding, compression level, etc. I can't find it written down right now unfortunately, but my understanding of the decision to include only the file names and file contents in the hashes have been to prevent this problem. The only reference I'm able to find now is the documentation of
|
@jayconrod This is exactly what it wanted to ask, thanks! |
|
In my module proxy I create the zip file on the fly from a tar.gz stream that I receive from the Gitlab API in order to prevent downloading the full repository snapshot first. I think this use case would not be applicable with the API of https://go-review.googlesource.com/c/mod/+/202042/2/zip/zip.go . It would be really nice if the zip could be built by incrementally adding files, one after another. |
#34965 is closely related. (We should probably preserve the execute bit, at least.) |
There is a reason the hash in the |
In this CL, Either way, the whole stream needs to be read before we can start adding files to the zip. Given that, I think it's better to make the cost clear to the caller and accept a list of files, rather than a stream. |
Ok, that's a valid point and I haven't thought about filtering submodules, yet. |
|
Repeating my comment from #34965 (comment): We defined the Go module representation to be absolutely least common denominator across systems for portability. The Go module zip file is for dependencies that need to be downloaded and built by the go command. It is not meant to support anything more. By design, there are no permission bits, nor special files like symlinks, nor modification times. I still believe this is the right decision. We can go arbitrarily far in "preserving" file metadata, and the result would still miss some kinds of metadata and fail to recreate other metadata on some subset of systems (symlinks, odd file modification times, etc). Tests that need special files (anything beyond files containing bytes) can write them to a temporary directory and configure them as they wish. Install scripts can be invoked as I don't believe this decision was wrong. But even if it were, it is a fundamental design decision made years ago that would be very hard to change now - it would invalidate all the checksums (if this extra information is important enough to preserve it would have to be checksummed too). It would have to be very wrong to be worth the pain of correcting. |
|
Closing as working as intended. I'll include this in the module reference documentation and the documentation for |
What version of Go are you using (
go version)?Does this issue reproduce with the latest release?
Probably.
What operating system and processor architecture are you using (
go env)?go envOutputWhat did you do?
I am writing a custom module proxy at the moment and while testing equality of the zipped module with the output of
proxy.golang.orgI noticed that the official proxy does not set the timestamp for the files in the archive, i.e. they are all datedNov 30 1979.Is this the expected behavior?
What did you expect to see?
The actual file timestamps, i.e. the files in the archive should have the same timestamp as their source in the git repository.
What did you see instead?
$ curl -s https://proxy.golang.org/github.com/pkg/errors/@v/v0.8.1.zip | bsdtar -ztvf- -rw-rw-r-- 0 0 0 0 Nov 30 1979 github.com/pkg/errors@v0.8.1/.gitignore -rw-rw-r-- 0 0 0 0 Nov 30 1979 github.com/pkg/errors@v0.8.1/.travis.yml -rw-rw-r-- 0 0 0 0 Nov 30 1979 github.com/pkg/errors@v0.8.1/LICENSE -rw-rw-r-- 0 0 0 0 Nov 30 1979 github.com/pkg/errors@v0.8.1/README.md -rw-rw-r-- 0 0 0 0 Nov 30 1979 github.com/pkg/errors@v0.8.1/appveyor.yml -rw-rw-r-- 0 0 0 0 Nov 30 1979 github.com/pkg/errors@v0.8.1/bench_test.go -rw-rw-r-- 0 0 0 0 Nov 30 1979 github.com/pkg/errors@v0.8.1/errors.go -rw-rw-r-- 0 0 0 0 Nov 30 1979 github.com/pkg/errors@v0.8.1/errors_test.go -rw-rw-r-- 0 0 0 0 Nov 30 1979 github.com/pkg/errors@v0.8.1/example_test.go -rw-rw-r-- 0 0 0 0 Nov 30 1979 github.com/pkg/errors@v0.8.1/format_test.go -rw-rw-r-- 0 0 0 0 Nov 30 1979 github.com/pkg/errors@v0.8.1/stack.go -rw-rw-r-- 0 0 0 0 Nov 30 1979 github.com/pkg/errors@v0.8.1/stack_test.goThe text was updated successfully, but these errors were encountered: