cmd/go: work when binaries are available but source is missing
Labels changed: added go1-must.
This issue was closed by revision b03a5f6.
Status changed to Fixed.
After building from the tip @ commit a461bcce05f6 which fixed this issue, I have a new
issue while trying to go install libraries from or into my GOPATH. I think I've managed
to isolate the circumstances under which this happens.
My GOPATH package dir is:
First, here's what actually works.
As long as there are no binary packages installed to ~/go/pkg then I can go build any
package whose source is located in ~/go/src:
☿ ganderson@excession [17:48:09] ~/src/go/golang on default at tip?
: go build -v github.com/ha/doozer
I can successfully go install a single package which has no dependancies:
☿ ganderson@excession [17:48:12] ~/src/go/golang on default at tip?
: go install -v github.com/kr/pretty.go/
No here is where things start to break. Once I try to build or install a package which
has a dependency already installed as a package, the command fails... Here is the
structure of ~/go/pkg after the previous install command:
☿ ganderson@excession [17:52:27] ~/src/go/golang on default at tip?
: tree ~/go/pkg/linux_amd64/
2 directories, 1 file
Now if I try to either go build or go install a package which has
github.com/kr/pretty.go as a dependency the command fails, e.g.
☿ ganderson@excession [17:54:01] ~/src/go/golang on default at tip?
: go build -v github.com/ha/doozer
../../../go/src/github.com/ha/doozer/conn.go:7: can't find import: "github.com/kr/pretty.go"
If I clean my ~/go/pkg dir and try to go install github.com/ha/doozer directly:
☿ ganderson@excession [17:55:55] ~/src/go/golang on default at tip?
: go install -v github.com/ha/doozer
../../../go/src/github.com/ha/doozer/conn.go:4: can't find import: "code.google.com/p/goprotobuf/proto"
Note that both pretty.go and protobuf are both compiled and installed to ~/go/pkg during
the command but trying to build doozer against these fails.
ganderson@excession [17:58:26] ~/src/go/golang on default at tip?
: tree ~/go/pkg/linux_amd64/
│ └── p
│ └── goprotobuf
│ └── proto.a
5 directories, 2 files
To summarise the issue - I can go build or go install packages in GOPATH, as long as a
dependency is not already installed to $GOPATH/pkg. This issue does *not* appear to
Here's my env, in case it's relevant.
Thanks for the detailed update.
Owner changed to email@example.com.
Status changed to Accepted.
I think one of your commits this week has resolved this, none of the issues I previously
reported are showing in weekly.2012-03-04. All packages are building ok whether its via
source or pre-built binary in GOPATH.
We were passing a not-quite-right list of directories to 6g; it is possible that the fix
Status changed to Retracted.
This issue has re-appeared again in go1 release. It happens for all packages that import
a third party lib which is installed as as a binary only. The symptoms as described in
comment #3 fit exactly the problem as noticed in go1 release.
The previous release where this does not happen is weekly.2012-03-22
I've just noticed that unlike comment #3, this problem also now happens if the packages
are installed to $GOROOT as well or instead of $GOPATH
Sorry, a further clarification. If source is available the build completes. What I am
seeing then is that it's not possible to build using third party imports where the
import is installed only as binary package in either $GOROOT or $GOPATH:
• ganderson@excession [01:05:33] ~
: tree $GOPATH/pkg
│ └── p
│ └── goprotobuf
│ ├── proto.a
│ └── protoc-gen-go
│ ├── descriptor.a
│ ├── generator.a
│ └── plugin.a
7 directories, 5 files
• ganderson@excession [02:44:53] ~
: go build -x -v github.com/ha/doozer...
conn.go:4:2: import "code.google.com/p/goprotobuf/proto": cannot find package
conn.go:7:2: import "github.com/kr/pretty": cannot find package
I removed support for this shortly before Go 1, because it was causing worse problems.
My intention is to allow this usage in Go 1.1, but only if there is no src directory
in the GOPATH tree in question. That is, if you create a directory listed in GOPATH
that has _no source at all_, then it will be treated as binary-only.
Labels changed: added priority-later, go1.1, removed priority-go1, go1-must.
Labels changed: added size-l.
Issue #4635 has been merged into this issue.
This issue was one of the most serious limitations of the "Distributed Systems" course
 at Carnegie Mellon University.
"You can't hand out precompiled library code and use the go build tools
This was one of the most serious limitations we encountered. We tried several (hacky)
approaches and none worked reliably enough for the class environment (we need to try the
mtime hack again on AFS). This meant that we had no way to hand out binary reference
code for the students to test against. Nor could we hand out reference implementations
for students who couldn't get part of the project working." 
@r.eklind: yeah, that sounds like a nasty limitation. What would be the best solution
for you that would solve this ?
Owner changed to ---.
chiming in, @dave: The mechanism that Russ discussed as a potential solution in Go1.1
would be just peachy. We have total control over the source directory tree that the
students start out with, so we can basically use anything. The limitations are that
they typically extract it from a tar file, we don't want to have to have them run any
additional commands (touch(1)'ing files), and we'd love it to work on a variety of
platforms and filesystems (many students run the code on AFS; OSes used span the gamut,
etc). That's why I point to the mtime hack as a bit too fragile for our environment.
My ideal solution would let us:
- Hand out *one* tarball that contains precompiled packages for multiple architectures;
- Assuming GOPATH is set properly, the students could simply import ("foo") and be able to use the package.
What other information in particular would be useful to you as far as "the best
[The time for maybe has passed.]
Labels changed: added go1.1maybe, removed go1.1.
Comment 19 by CundraMihail:
I don't want to pollute this issue, but since I've spent so much time and finally
stumbled onto this (apparently relevant issue) it seems relevant. I've tried searching
quite a few times through every Go book I own, the golang.org site, as well as the
golang-nuts group and I couldn't find anything that mentioned anywhere that it wasn't
possible to use a Go package with no source available. It seems like this simple fact
would be mentioned somewhere?
I fully expected to be able to have:
have test.go contain
and have it work. Instead it only works if all source for dummy is available in
src/dummy. What's even the point of having dummy.a then?
Go requiring all source to always be available for all packages seems like a pretty
significant requirement to mention somewhere, don't you think?
Labels changed: added go1.2maybe, removed go1.1maybe.
Labels changed: added feature.
Too late for Go 1.2.
Labels changed: added go1.3maybe, removed go1.2maybe.
Labels changed: removed go1.3maybe.
Labels changed: added go1.3maybe.
Labels changed: removed feature.
Labels changed: added release-none, removed go1.3maybe.
Labels changed: added repo-main.
today i discovered one more issue worth considering when trying to fix this issue.
when a package contains c++ code is built and then distributed without source code
(or with a dummy document-only source file), cmd/go won't be able to figure out
to use -extld g++ to link the program that uses the package.
Either we record the ld requirement in object files, or we link the required libstdc++
into the cgo.o file.
FYI, linking libstdc++ into a .o file won't work. The C++ library internals change over
time. The external interface remains backward compatible at the binary level, but you
can't reliably mix and match some internal code across libstdc++ versions.
yeah, i also figured that we shouldn't relying on static linking libstdc++ anyway.
so our only option would be to record the ld requirement in the header.
The mtime hack appears to have stopped working in go1.5rc1.
It appears the plumbing is there to allow binary only packages, but it is explicitly removed:
// TODO: After Go 1, decide when to pass build.AllowBinary here.
// See issue 3268 for mistakes to avoid.
bp, err := buildContext.Import(path, srcDir, build.ImportComment)
It seems the problem is there is no way in the current setup to know if a package was released as binary only of if the sources were removed. Perhaps simply having an "binary only" package directory. The .a files in that package directory would not require sources. For example "pkgbin" or "pgk/binary". A flag to go build could be used by the vendor to specify the destination should be the binary directory but it should rebuild from sources.