Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proposal: use zip format inside .a and .o files #14386

Closed
rsc opened this issue Feb 18, 2016 · 10 comments
Closed

proposal: use zip format inside .a and .o files #14386

rsc opened this issue Feb 18, 2016 · 10 comments
Assignees
Milestone

Comments

@rsc
Copy link
Contributor

@rsc rsc commented Feb 18, 2016

I propose to change both Go package archives and Go object files to use the more standard zip archive format.
In contrast to ar archives, zip archives admit efficient random access to individual files within the archive
and also allow decisions about compression on a per-file basis.
The result for Go will be cleaner access to the parts of a package archive
and the ability later to add compression of individual parts
as appropriate.

See golang.org/design/14386-zip-package-archives for details.

@rsc rsc self-assigned this Feb 18, 2016
@rsc rsc added this to the Proposal milestone Feb 18, 2016
@rsc rsc added the Proposal label Feb 18, 2016
@rsc
Copy link
Contributor Author

@rsc rsc commented Feb 18, 2016

Updated link to design doc in original post, but here it is for people getting email updates: https://golang.org/design/14386-zip-package-archives.

@dsnet
Copy link
Member

@dsnet dsnet commented Feb 19, 2016

I don't have too much of an opinion regarding new .a and .o files (since I don't deal with compiler/linker stuff).

As the one who's been touching archive/tar lately, I support using zip over tar. The tar "format" is pretty fragmented and essentially consists of variety of popular and incompatible extensions (of which, only a few have a specification). And of the ones that have a specification, the behavior of the tar utility that introduced a given format doesn't always match the specified behavior. In contrast (as mentioned), the zip format is well defined (even if it is somewhat complicated) and Go's implementation of zip is less buggy then tar, IMHO.

What are the extensions to archive/zip that @rogpeppe is adding? Does it have to deal with trying to extend zip.File.Open to return an io.ReadCloser that satisfies io.ReaderAt and/or io.Seeker under for the zip.Store method? As it is today, zip.File.Open does not return something that can be random-accessed (like the proposal assumes is possible).

Edit: I figured out that random-access on individual files can be achieved by a combination of zip.File.DataOffset and zip.File.CompressedSize64. Disregard my concern.

@bradfitz
Copy link
Contributor

@bradfitz bradfitz commented Feb 19, 2016

The proposal says both:

I propose to change both Go package archives and Go object files to use the more standard zip archive format.

and also:

Go uses its own object file format, which we refer to as Go object files; that format is unchanged by this proposal.

That sounds contradictory.

@minux
Copy link
Member

@minux minux commented Feb 19, 2016

@rsc
Copy link
Contributor Author

@rsc rsc commented Feb 19, 2016

On Thu, Feb 18, 2016 at 9:29 PM, Minux Ma notifications@github.com wrote:

so the Go 1.7 compiler and linker will have to vendor
math/big, archive/zip, and potentially compress/flate?

could we instead make cmd/dist copy the vendored
code to pkg/bootstrap instead duplicating those
packages in cmd/? Too bad we didn't introduce the
vendoring experiment into Go 1.4, otherwise the
solution will be really simple.

Yes, I have a CL that already does this. cmd/dist already knows how to
rewrite imports; vendor directories would add very little.

Russ

@mwhudson
Copy link
Contributor

@mwhudson mwhudson commented Feb 29, 2016

I guess for a package containing assembly the assembler will produce .obj files directly and the go tool will simply add these to the zip file the compiler produced?

The abstract says "I propose to change both Go package archives and Go object files to use the more standard zip archive format." but is really mostly about package archives isn't it? I guess the .o files the compiler and assembler currently make are somewhat containerish, and the *.obj files will just contain the part that follows the \n!\n bit in the current .o files? (I wonder, for the sort of object file redesign I propose in #11123 maybe it would make sense to have the *.obj file be replaced by a directory in the zip containing files like symtable, strings, data, etc...)

@mdempsky
Copy link
Member

@mdempsky mdempsky commented Aug 5, 2016

Have we ruled out the possibility of just using the file system and storing compiled artifacts as individual files?

@adg
Copy link
Contributor

@adg adg commented Sep 26, 2016

@rsc ping!

@rsc
Copy link
Contributor Author

@rsc rsc commented Sep 30, 2016

This is a low-priority change for me. I don't see any objections here, so I'm assuming it's okay to implement. Whether I get to it in time for Go 1.8 remains to be seen.

@adg adg modified the milestones: Go1.9, Proposal Oct 31, 2016
@rsc
Copy link
Contributor Author

@rsc rsc commented Dec 19, 2016

This is a good idea, but right now we don't have the bandwidth to help redesign the object format and support the resulting churn in the code base. We should revisit the object format at some point, but it won't be soon. Marking this declined for now, with the expectation that we may come back to this later.

@rsc rsc closed this Dec 19, 2016
@golang golang locked and limited conversation to collaborators Dec 19, 2017
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
8 participants
You can’t perform that action at this time.