cmd/go: work when binaries are available but source is missing #2775

Closed
rsc opened this Issue Jan 24, 2012 · 53 comments
@rsc
cmd/go: work when binaries are available but source is missing
@rsc

Comment 1:

Labels changed: added go1-must.

@rsc
rsc commented Mar 1, 2012

Comment 2:

This issue was closed by revision b03a5f6.

Status changed to Fixed.

@gnanderson

Comment 3:

After building from the tip @ commit a461bcce05f6 which fixed this issue, I have a new
issue while trying to go install libraries from or into my GOPATH. I think I've managed
to isolate the circumstances under which this happens.

My GOPATH package dir is:

    /home/ganderson/go/pkg/linux_amd64

First, here's what actually works.

As long as there are no binary packages installed to ~/go/pkg then I can go build any
package whose source is located in ~/go/src:

    ☿ ganderson@excession [17:48:09] ~/src/go/golang  on default at tip?
    : go build -v github.com/ha/doozer
    code.google.com/p/goprotobuf/proto
    github.com/kr/pretty.go
    github.com/ha/doozer

I can successfully go install a single package which has no dependancies:

    ☿ ganderson@excession [17:48:12] ~/src/go/golang  on default at tip?
    : go install -v github.com/kr/pretty.go/
    github.com/kr/pretty.go

No here is where things start to break. Once I try to build or install a package which
has a dependency already installed as a package, the command fails... Here is the
structure of ~/go/pkg after the previous install command:

    ☿ ganderson@excession [17:52:27] ~/src/go/golang  on default at tip?
    : tree ~/go/pkg/linux_amd64/
    /home/ganderson/go/pkg/linux_amd64/                                                                                                                                                                                                          
    └── github.com                                                                                                                                                                                                                               
        └── kr                                                                                                                                                                                                                                   
            └── pretty.go.a                                                                                                                                                                                                                      
                                                                                                                                                                                                                                             
    2 directories, 1 file

Now if I try to either go build or go install a package which has
github.com/kr/pretty.go as a dependency the command fails, e.g.

    ☿ ganderson@excession [17:54:01] ~/src/go/golang  on default at tip?                                                                                                                                                                         
    : go build -v github.com/ha/doozer                                                                                                                                                                                                           
    code.google.com/p/goprotobuf/proto                                                                                                                                                                                                           
    github.com/ha/doozer                                                                                                                                                                                                                         
    # github.com/ha/doozer                                                                                                                                                                                                                       
    ../../../go/src/github.com/ha/doozer/conn.go:7: can't find import: "github.com/kr/pretty.go"

If I clean my ~/go/pkg dir and try to go install github.com/ha/doozer directly:

    ☿ ganderson@excession [17:55:55] ~/src/go/golang  on default at tip?                                                                                                                                                                         
    : go install -v github.com/ha/doozer
    code.google.com/p/goprotobuf/proto                                                                                                                                                                                                           
    github.com/kr/pretty.go                                                                                                                                                                                                                      
    github.com/ha/doozer                                                                                                                                                                                                                         
    # github.com/ha/doozer                                                                                                                                                                                                                       
    ../../../go/src/github.com/ha/doozer/conn.go:4: can't find import: "code.google.com/p/goprotobuf/proto"

Note that both pretty.go and protobuf are both compiled and installed to ~/go/pkg during
the command but trying to build doozer against these fails.

    ganderson@excession [17:58:26] ~/src/go/golang  on default at tip?                                                                                                                                                                         
    : tree ~/go/pkg/linux_amd64/                                                                                                                                                                                                         
    /home/ganderson/go/pkg/linux_amd64/                                                                                                                                                                                                          
    ├── code.google.com                                                                                                                                                                                                                          
    │   └── p                                                                                                                                                                                                                                    
    │       └── goprotobuf                                                                                                                                                                                                                       
    │           └── proto.a                                                                                                                                                                                                                      
    └── github.com                                                                                                                                                                                                                               
        └── kr                                                                                                                                                                                                                                   
            └── pretty.go.a                                                                                                                                                                                                                      
                                                                                                                                                                                                                                             
    5 directories, 2 files


To summarise the issue - I can go build or go install packages in GOPATH, as long as a
dependency is not already installed to $GOPATH/pkg. This issue does *not* appear to
affect GOROOT.
@gnanderson

Comment 4:

Here's my env, in case it's relevant.

GOBIN=/usr/bin
GOARCH=amd64
GOROOT=/usr/lib64/go
GOOS=linux
GOPATH=/home/ganderson/go
@rsc
rsc commented Mar 2, 2012

Comment 5:

Thanks for the detailed update.

Owner changed to builder@golang.org.

Status changed to Accepted.

@gnanderson

Comment 6:

I think one of your commits this week has resolved this, none of the issues I previously
reported are showing in weekly.2012-03-04. All packages are building ok whether its via
source or pre-built binary in GOPATH.
@rsc
rsc commented Mar 7, 2012

Comment 7:

We were passing a not-quite-right list of directories to 6g; it is possible that the fix
fixed this.

Status changed to Retracted.

@gnanderson

Comment 8:

This issue has re-appeared again in go1 release. It happens for all packages that import
a third party lib which is installed as as a binary only. The symptoms as described in
comment #3 fit exactly the problem as noticed in go1 release.

The previous release where this does not happen is weekly.2012-03-22
@gnanderson

Comment 9:

I've just noticed that unlike comment #3, this problem also now happens if the packages
are installed to $GOROOT as well or instead of $GOPATH
@gnanderson

Comment 10:

Sorry, a further clarification. If source is available the build completes. What I am
seeing then is that it's not possible to build using third party imports where the
import is installed only as binary package in either $GOROOT or $GOPATH:


    • ganderson@excession [01:05:33] ~ 
    : tree $GOPATH/pkg
    /home/ganderson/go/pkg
    └── linux_amd64
        ├── code.google.com
        │   └── p
        │       └── goprotobuf
        │           ├── proto.a
        │           └── protoc-gen-go
        │               ├── descriptor.a
        │               ├── generator.a
        │               └── plugin.a
        └── github.com
            └── kr
                └── pretty.a
    
    7 directories, 5 files

    • ganderson@excession [02:44:53] ~ 
    : go build -x -v github.com/ha/doozer...
    WORK=/tmp/go-build734680617
    conn.go:4:2: import "code.google.com/p/goprotobuf/proto": cannot find package
    conn.go:7:2: import "github.com/kr/pretty": cannot find package
@rsc
rsc commented Apr 2, 2012

Comment 11:

I removed support for this shortly before Go 1, because it was causing worse problems.
My intention is to allow this usage in Go 1.1, but only if there is no src directory 
in the GOPATH tree in question.  That is, if you create a directory listed in GOPATH
that has _no source at all_, then it will be treated as binary-only.

Labels changed: added priority-later, go1.1, removed priority-go1, go1-must.

Status changed to Accepted.

@rsc

Comment 12:

Labels changed: added size-l.

@davecheney

Comment 13:

Issue #4635 has been merged into this issue.

@mewmew

Comment 14:

This issue was one of the most serious limitations of the "Distributed Systems" course
[1] at Carnegie Mellon University.

Quote:
"You can't hand out precompiled library code and use the go build tools

This was one of the most serious limitations we encountered.  We tried several (hacky)
approaches and none worked reliably enough for the class environment (we need to try the
mtime hack again on AFS).  This meant that we had no way to hand out binary reference
code for the students to test against.  Nor could we hand out reference implementations
for students who couldn't get part of the project working." [2]

[1]: http://www.cs.cmu.edu/~dga/15-440/F12/index.html
[2]: http://da-data.blogspot.se/2013/02/teaching-distributed-systems-in-go.html
@davecheney

Comment 15:

@r.eklind: yeah, that sounds like a nasty limitation. What would be the best solution
for you that would solve this ?

Owner changed to ---.

@dave-andersen

Comment 16:

chiming in, @dave:  The mechanism that Russ discussed as a potential solution in Go1.1
would be just peachy.  We have total control over the source directory tree that the
students start out with, so we can basically use anything.  The limitations are that
they typically extract it from a tar file, we don't want to have to have them run any
additional commands (touch(1)'ing files), and we'd love it to work on a variety of
platforms and filesystems (many students run the code on AFS;  OSes used span the gamut,
etc).  That's why I point to the mtime hack as a bit too fragile for our environment.

My ideal solution would let us:
  - Hand out *one* tarball that contains precompiled packages for multiple architectures;
  - Assuming GOPATH is set properly, the students could simply import ("foo") and be able to use the package.

What other information in particular would be useful to you as far as "the best
solution"?
@rsc

Comment 17:

[The time for maybe has passed.]
@rsc

Comment 18:

Labels changed: added go1.1maybe, removed go1.1.

@gopherbot

Comment 19 by CundraMihail:

I don't want to pollute this issue, but since I've spent so much time and finally
stumbled onto this (apparently relevant issue) it seems relevant.  I've tried searching
quite a few times through every Go book I own, the golang.org site, as well as the
golang-nuts group and I couldn't find anything that mentioned anywhere that it wasn't
possible to use a Go package with no source available.  It seems like this simple fact
would be mentioned somewhere?
I fully expected to be able to have:
$GOPATH/
  pkg/arch/dummy.a
  src/test/test.go
have test.go contain 
import 'dummy'
and have it work.   Instead it only works if all source for dummy is available in
src/dummy.  What's even the point of having dummy.a then?
Go requiring all source to always be available for all packages seems like a pretty
significant requirement to mention somewhere, don't you think?
@robpike

Comment 20:

Labels changed: added go1.2maybe, removed go1.1maybe.

@rsc

Comment 21:

Labels changed: added feature.

@robpike

Comment 22:

Too late for Go 1.2.

Labels changed: added go1.3maybe, removed go1.2maybe.

@robpike

Comment 23:

Labels changed: removed go1.3maybe.

@rsc

Comment 24:

Labels changed: added go1.3maybe.

@rsc

Comment 25:

Labels changed: removed feature.

@rsc
rsc commented Dec 4, 2013

Comment 26:

Labels changed: added release-none, removed go1.3maybe.

@rsc
rsc commented Dec 4, 2013

Comment 27:

Labels changed: added repo-main.

@minux
Go member

Comment 28:

today i discovered one more issue worth considering when trying to fix this issue.

when a package contains c++ code is built and then distributed without source code
(or with a dummy document-only source file), cmd/go won't be able to figure out
to use -extld g++ to link the program that uses the package.

Either we record the ld requirement in object files, or we link the required libstdc++
into the cgo.o file.
@ianlancetaylor

Comment 29:

FYI, linking libstdc++ into a .o file won't work.  The C++ library internals change over
time.  The external interface remains backward compatible at the binary level, but you
can't reliably mix and match some internal code across libstdc++ versions.
@minux
Go member

Comment 30:

yeah, i also figured that we shouldn't relying on static linking libstdc++ anyway.

so our only option would be to record the ld requirement in the header.
@rsc rsc added this to the Unplanned milestone Apr 10, 2015
@aaronbee

The mtime hack appears to have stopped working in go1.5rc1.

@pborman

It appears the plumbing is there to allow binary only packages, but it is explicitly removed:

    // TODO: After Go 1, decide when to pass build.AllowBinary here.
    // See issue 3268 for mistakes to avoid.
    bp, err := buildContext.Import(path, srcDir, build.ImportComment)

It seems the problem is there is no way in the current setup to know if a package was released as binary only of if the sources were removed. Perhaps simply having an "binary only" package directory. The .a files in that package directory would not require sources. For example "pkgbin" or "pgk/binary". A flag to go build could be used by the vendor to specify the destination should be the binary directory but it should rebuild from sources.

@pborman

I think this could be addressed very simply in a variety of ways. Since I assume people want to keep the "ignore any .a for which there are no .go files" concept, we can define a way for the .go file to specify this. Here are some example contents:

package foo

If foo.go is the only source file for package foo, and it only contains these two words, then do not rebuild the .a file.

package foo
//go:norebuild

A directive to indicate the compiler should never rebuild this file. This would let the .go file contain documentation, possibly even stub function definitions so godoc would continue to work.

Or pick any other indicator.

@pborman

#12186 is the same bug as this.

@pborman

#12186 was closed, but it was marked as Go1.7Early while this one is not. Can we get this one marked as Go1.7Early as well?

@bradfitz
Go member

@pborman, Go1.NEarly means "if it's going to happen for Go 1.N, it better happen very early in the cycle". The Go 1.7 cycle is well past early at this point.

@rsc rsc modified the milestone: Go1.7, Unplanned Mar 30, 2016
@rsc

I intend to look at this if I can find time in the next few weeks, since Paul reached out to me with a real use case that I didn't realize was already happening in the wild (vendor-supplied package binaries).

@stemar94

@rsc: Wouldn't we have to fix the .a format for this first? Or will it be 1.7 .a files can only be used with 1.7 compiler/linker?

@rsc

It's already the case (very intentionally) that you can only use package binaries compiled with the exact same version of the toolchain you are using. So 1.6 package binaries only work with 1.6, and yes 1.7 package binaries will only work with 1.7. The use case assumes the producer and consumer agree on a go toolchain.

@minux
Go member
@pborman

Enabling binary only packages does not require go get. Once binary only packages are possible, someone could look at what to do with go get. I would not want enabling binary packages to be gated on a conversation about go get.

@eliquious
eliquious commented Apr 15, 2016 edited

We also have a need to use an archive file provided to us by another company. What about a linkarchive flag followed by a list of comma delimited libraries to use. Using a linkarchive flag would also be consistent with the new linkshared flag.


The go tool could either search the pkg dir or the current directory like so:

Search $GOPATH/pkg dir

Requires user to put the .a file in it's proper place.

Commands:

cp lib.a $GOPATH/pkg/darwin_amd64/github.com/user/lib
go build -linkarchive github.com/user/lib

Search $PWD/pkg

Searching the current directory would prevent the additional step of copying the archive and allow easier management of supporting multiple OSs.

Directory structure:

- $GOPATH/src/github.com/user/main
    \- pkg/darwin_amd64/
        \- github.com/user/lib
             - lib.a

Command:

go build -linkarchive github.com/user/lib
@gopherbot gopherbot pushed a commit to golang/proposal that referenced this issue Apr 25, 2016
@rsc rsc design: add design doc for binary-only packages
For golang/go#2775.

Change-Id: I44e1f8687c1b75381ff3cde5f7f659b13c717c44
Reviewed-on: https://go-review.googlesource.com/22431
Run-TryBot: Russ Cox <rsc@golang.org>
Reviewed-by: Russ Cox <rsc@golang.org>
6599001
@gopherbot

CL https://golang.org/cl/22433 mentions this issue.

@gopherbot

CL https://golang.org/cl/22432 mentions this issue.

@dlsniper
dlsniper commented Apr 26, 2016 edited

Hi,

I have a few issues with this proposal:

  • how will vendoring work in this context. One can argue that vendoring is already a half baked solution, but with this proposal ignoring vendoring it seems yet another burden to be solved. Concrete case: package A depends on package BinOnly, how does package C which vendors package A works?
  • the proposal says that code in the Go files is optional. This basically breaks any editor that has to work with source code. To examples that I have in mind are the Go plugin for IntelliJ which relies on the source code in GOPATH/src to work and the other is go-code which uses .a files to work. If I understand correctly, the proposal says that .a files are considered to be binary? If this is the case it means that the IntelliJ plugin will have to start parsing .a files as well, which would be ok but the do those packages include documentation for the exported types as well? Also, where's the documentation describing the .a file format (and please don't point me to the spurce code doing the work, that is not proper documentation / description)?
  • what will be the packaging for this? Like I've mentioned earlier, if this uses .a then some things will still be possible but if the files are going to be true compiled binaries then how does it work (with the above two concerns)
  • how does the binary package works for keeping it compatible between Go version? If I ship a Go 1.7 compiled binary will that work in 1.8?
  • this feels rushed in into 1.7. Would you consider waiting for 1.8 to have a proper discussion around it?
@robpike

I agree it seems rushed. Perhaps it should be marked as an experimental feature that will soak for one cycle before becoming official.

@rsc

This feature is not intended for general use. It is intended for the very specific case of "company X wants to sell company Y a binary version of a package (or the less commercial setting from @dave-andersen: teacher X wants to give student Y a reference solution)". I've heard from multiple, diverse sources that this is already happening today, with hacks of various kinds (see background in doc). We can't stop this from happening, but we can make sure that those people don't get stuck on Go 1.4 because the go command has gotten too good at noticing the source code is missing.

To answer the question "what about tool X that needs source?", as mentioned in the doc the answer is "tool X will not work with such packages, and hopefully that will help limit the use of such things." In particular this makes no sense in any open source context such as "go get" or Linux distributions, which is why for example "go get" will not support such arrangements. But again Go is used for more than just open source.

I'm happy to say the details here are experimental in case other details arise (not that the tools are covered by the compatibility guarantee anyway) or somehow it's not restrictive enough and starts to be used inappropriately, but there is a real need here, just one felt disproportionally by people outside the open source community and not on GitHub.

The IntelliJ plugin can stay as it is, or it might make sense to use go/types reading the binaries anyway, depending on the exact needs of the plugin. But it's fine if it doesn't work with binary-only packages. The more helpful tools that don't support binary-only packages and discourage their use, the better. But they do need to work for actually building a program.

@dlsniper
dlsniper commented Apr 26, 2016 edited

Thanks for feedback.

Again, I'm not against this feature landing in the Go distribution, I can see the benefits of having it.
However, Go, unlike C, doesn't have header files (at least until now).
From my understanding of the phrase:

Having a “fake” version of the source code also provides a way to supply documentation compatible with “go doc” and “godoc” even though the complete source code is missing.

It does seem that one will need to provide a minimal definition of the package, something like:

// Copyright 2016 dlsniper

//go:binary-only-package
package awesome

// This is entirely optional

// The Awesomizer interface allows you to make awesome code
type Awesomizer interface{
    CodeIt()
}

At which point I would rather prefer to enforce the stub package to contain the public definitions (and hopefully documentation for this) rather than just the package name and the special flag + other build flags. This would then mean that the documentation of the package has to be respected and it would help out users using such packages.

Speaking of flags, rather than introduce a new flag for the go tool, wouldn't it be better to introduce a new, static build tag? It can keep the name binary-only-package but then as a developer I could at least test out how this works in the code and still have the code for the package as well. This would also represent one less thing that Go developers have to learn and keep track of (not everyone knows the //go: flags (and yes, I know of go:generate)

Again, to address the 🐘 in the issue: what about vendoring? How would that work in this context? There are already problems with the current vendoring approach which came up a few times on the mailing list and have only resulted in workarounds / "developers should learn more / read / understand better" which is not very real-life unfortunately. Adding a binary only as a vendored dependency, even if internally, would not be something that people will skip, I'm positive this will happen.

The more helpful tools that don't support binary-only packages and discourage their use, the better.

I'm willing to wage a bet that within a couple of weeks of this feature landing people will ask for this feature to work. And like I've said, maybe for the IntelliJ plugin will do some support for it (or not, I can't speak of that right now) but if it's there, people will want to use it (as you've said, they already do it) as it would be a standard feature (which is exactly why people have editors like Atom, VSCode or IntelliJ with all sorts of plugins helping them write the code. But I don't wan to digress here on how people write code).

To sum up, imho:

  • public types should be present in the stub Go package (which is expected to be delivered apparently from the proposal)
  • vendoring needs to be addressed before merging this, not after (even if it means clear documentation of the use-cases and how to approach them)
  • maybe build tags could be used instead of the //go:binary-only "flag" or how we call those

Thank you.

EDIT:
I've changed the build flag to build tag, sorry for the wrong names.

@dlsniper

There's also of course the issue of ensuring that the binary package being compiled is the one that the authors intended to distribute so maybe having a checksum for the binary present in the stub package would help the go tool to validate it at compile time?

@pborman
@dlsniper

This has been an issue since 1.5 and has been ignored since then (it has always been "we will address in release N+1"). Kicking the can down the road has proven to not be helpful.

@pborman to be a bit mean, that's what they do with generics for the last 6 years, no? Not to mention other issues (multiple package coverage, etc...) I don't want to go there but this is not a technical argument for a good implementation of a solution imho.

They can provide the minimal [...] Or they can provide more useful information [...]

So why not force those who want this feature to do proper documentation / packaging for it? @rsc says that it's ok to make it painful for end-users, why not make it painful (and get real value of it) for producers as well?

Adding checksums is interesting, but in this case I would venture to say that versioning is more useful as it will immediately tell you which version is more up to date.

We don't want to have go get aware on versioning so why would binary only packages be different? Also checksums provide a different functionality than versioning.

I really thank Russ for doing this as I have been one of those affected by the deteriorating "support" for binary only packages (as a consumer, not a producer).

I too am grateful for his work and all the other contributors to the Go ecosystem, don't get me wrong. But like I've stated, this feels bad. I agree with what @robpike suggested to flag it as experimental. Meanwhile we could still discuss about this, no?

I feel I don't stress enough for this. I'm not against the idea of having this implemented, not even a bit. I'm against the idea of rushing it to get in for 1.7 just because it's something people are doing today. Vendoring has more problems and I can argue that it's a problem for more people than binary-only packages are. I don't want to force this comparison anymore but this doesn't sound inline with the Go vision of offering pragmatic solutions for problems people have.

I've raised a few points to at least have a decent discussion on how to solve them and, if solutions are found then I don't see any problem for getting this into 1.7.

@pborman
@mk0x9 mk0x9 pushed a commit to mk0x9/go that referenced this issue Apr 27, 2016
@rsc rsc cmd/go: add Package.StaleReason for debugging with go list
It comes up every few months that we can't understand why
the go command is rebuilding some package.
Add diagnostics so that the go command can explain itself
if asked.

For #2775, #3506, #12074.

Change-Id: I1c73b492589b49886bf31a8f9d05514adbd6ed70
Reviewed-on: https://go-review.googlesource.com/22432
Reviewed-by: Rob Pike <r@golang.org>
0b5fbf7
@gopherbot gopherbot pushed a commit that closed this issue Apr 29, 2016
@rsc rsc cmd/go, go/build: add support for binary-only packages
See https://golang.org/design/2775-binary-only-packages for design.

Fixes #2775.

Change-Id: I33e74eebffadc14d3340bba96083af0dec5172d5
Reviewed-on: https://go-review.googlesource.com/22433
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
af6aa0f
@gopherbot gopherbot closed this in af6aa0f Apr 29, 2016
@rsc

I submitted the CL. We will note that it is experimental in the release notes.

To reply briefly to some points:

  • It doesn't work to be a build tag. Build tags are about selection of files to compile, not about changing the meaning of the compilation. Independent concerns, independent mechanisms. And //go: for prefixes is the established standard for source annotations directed at the go toolchain (but we're not going to retroactively change // +build).
  • If the binary-only package has the wrong imports inside it, likely the best way to address that would be to go back to the person who sold you the package and get them fixed. But you could also rewrite the binary form to change import paths if needed. I am skeptical it will come to that. And note that if you're happy with the imports being used in the binary-only package, you can choose your own import path for it simply by moving it somewhere else in the file system.
  • I don't think there's anything interesting about vendoring here except that it's one rationale for wanting a different import path.
  • If any tools want to not support these binary blobs, that's fine with me: you can say "go get doesn't support them either."
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment