Skip to content


cmd/go: work when binaries are available but source is missing #2775

rsc opened this Issue · 32 comments
cmd/go: work when binaries are available but source is missing

Comment 1:

Labels changed: added go1-must.

rsc commented

Comment 2:

This issue was closed by revision b03a5f6.

Status changed to Fixed.


Comment 3:

After building from the tip @ commit a461bcce05f6 which fixed this issue, I have a new
issue while trying to go install libraries from or into my GOPATH. I think I've managed
to isolate the circumstances under which this happens.

My GOPATH package dir is:


First, here's what actually works.

As long as there are no binary packages installed to ~/go/pkg then I can go build any
package whose source is located in ~/go/src:

    ☿ ganderson@excession [17:48:09] ~/src/go/golang  on default at tip?
    : go build -v

I can successfully go install a single package which has no dependancies:

    ☿ ganderson@excession [17:48:12] ~/src/go/golang  on default at tip?
    : go install -v

No here is where things start to break. Once I try to build or install a package which
has a dependency already installed as a package, the command fails... Here is the
structure of ~/go/pkg after the previous install command:

    ☿ ganderson@excession [17:52:27] ~/src/go/golang  on default at tip?
    : tree ~/go/pkg/linux_amd64/
        └── kr                                                                                                                                                                                                                                   
            └── pretty.go.a                                                                                                                                                                                                                      
    2 directories, 1 file

Now if I try to either go build or go install a package which has as a dependency the command fails, e.g.

    ☿ ganderson@excession [17:54:01] ~/src/go/golang  on default at tip?                                                                                                                                                                         
    : go build -v                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     
    ../../../go/src/ can't find import: ""

If I clean my ~/go/pkg dir and try to go install directly:

    ☿ ganderson@excession [17:55:55] ~/src/go/golang  on default at tip?                                                                                                                                                                         
    : go install -v                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                
    ../../../go/src/ can't find import: ""

Note that both pretty.go and protobuf are both compiled and installed to ~/go/pkg during
the command but trying to build doozer against these fails.

    ganderson@excession [17:58:26] ~/src/go/golang  on default at tip?                                                                                                                                                                         
    : tree ~/go/pkg/linux_amd64/                                                                                                                                                                                                         
    │   └── p                                                                                                                                                                                                                                    
    │       └── goprotobuf                                                                                                                                                                                                                       
    │           └── proto.a                                                                                                                                                                                                                      
        └── kr                                                                                                                                                                                                                                   
            └── pretty.go.a                                                                                                                                                                                                                      
    5 directories, 2 files

To summarise the issue - I can go build or go install packages in GOPATH, as long as a
dependency is not already installed to $GOPATH/pkg. This issue does *not* appear to
affect GOROOT.

Comment 4:

Here's my env, in case it's relevant.

rsc commented

Comment 5:

Thanks for the detailed update.

Owner changed to

Status changed to Accepted.


Comment 6:

I think one of your commits this week has resolved this, none of the issues I previously
reported are showing in weekly.2012-03-04. All packages are building ok whether its via
source or pre-built binary in GOPATH.
rsc commented

Comment 7:

We were passing a not-quite-right list of directories to 6g; it is possible that the fix
fixed this.

Status changed to Retracted.


Comment 8:

This issue has re-appeared again in go1 release. It happens for all packages that import
a third party lib which is installed as as a binary only. The symptoms as described in
comment #3 fit exactly the problem as noticed in go1 release.

The previous release where this does not happen is weekly.2012-03-22

Comment 9:

I've just noticed that unlike comment #3, this problem also now happens if the packages
are installed to $GOROOT as well or instead of $GOPATH

Comment 10:

Sorry, a further clarification. If source is available the build completes. What I am
seeing then is that it's not possible to build using third party imports where the
import is installed only as binary package in either $GOROOT or $GOPATH:

    • ganderson@excession [01:05:33] ~ 
    : tree $GOPATH/pkg
    └── linux_amd64
        │   └── p
        │       └── goprotobuf
        │           ├── proto.a
        │           └── protoc-gen-go
        │               ├── descriptor.a
        │               ├── generator.a
        │               └── plugin.a
            └── kr
                └── pretty.a
    7 directories, 5 files

    • ganderson@excession [02:44:53] ~ 
    : go build -x -v
    conn.go:4:2: import "": cannot find package
    conn.go:7:2: import "": cannot find package
rsc commented

Comment 11:

I removed support for this shortly before Go 1, because it was causing worse problems.
My intention is to allow this usage in Go 1.1, but only if there is no src directory 
in the GOPATH tree in question.  That is, if you create a directory listed in GOPATH
that has _no source at all_, then it will be treated as binary-only.

Labels changed: added priority-later, go1.1, removed priority-go1, go1-must.

Status changed to Accepted.


Comment 12:

Labels changed: added size-l.


Comment 13:

Issue #4635 has been merged into this issue.


Comment 14:

This issue was one of the most serious limitations of the "Distributed Systems" course
[1] at Carnegie Mellon University.

"You can't hand out precompiled library code and use the go build tools

This was one of the most serious limitations we encountered.  We tried several (hacky)
approaches and none worked reliably enough for the class environment (we need to try the
mtime hack again on AFS).  This meant that we had no way to hand out binary reference
code for the students to test against.  Nor could we hand out reference implementations
for students who couldn't get part of the project working." [2]


Comment 15:

@r.eklind: yeah, that sounds like a nasty limitation. What would be the best solution
for you that would solve this ?

Owner changed to ---.


Comment 16:

chiming in, @dave:  The mechanism that Russ discussed as a potential solution in Go1.1
would be just peachy.  We have total control over the source directory tree that the
students start out with, so we can basically use anything.  The limitations are that
they typically extract it from a tar file, we don't want to have to have them run any
additional commands (touch(1)'ing files), and we'd love it to work on a variety of
platforms and filesystems (many students run the code on AFS;  OSes used span the gamut,
etc).  That's why I point to the mtime hack as a bit too fragile for our environment.

My ideal solution would let us:
  - Hand out *one* tarball that contains precompiled packages for multiple architectures;
  - Assuming GOPATH is set properly, the students could simply import ("foo") and be able to use the package.

What other information in particular would be useful to you as far as "the best

Comment 17:

[The time for maybe has passed.]

Comment 18:

Labels changed: added go1.1maybe, removed go1.1.


Comment 19 by CundraMihail:

I don't want to pollute this issue, but since I've spent so much time and finally
stumbled onto this (apparently relevant issue) it seems relevant.  I've tried searching
quite a few times through every Go book I own, the site, as well as the
golang-nuts group and I couldn't find anything that mentioned anywhere that it wasn't
possible to use a Go package with no source available.  It seems like this simple fact
would be mentioned somewhere?
I fully expected to be able to have:
have test.go contain 
import 'dummy'
and have it work.   Instead it only works if all source for dummy is available in
src/dummy.  What's even the point of having dummy.a then?
Go requiring all source to always be available for all packages seems like a pretty
significant requirement to mention somewhere, don't you think?

Comment 20:

Labels changed: added go1.2maybe, removed go1.1maybe.


Comment 21:

Labels changed: added feature.


Comment 22:

Too late for Go 1.2.

Labels changed: added go1.3maybe, removed go1.2maybe.


Comment 23:

Labels changed: removed go1.3maybe.


Comment 24:

Labels changed: added go1.3maybe.


Comment 25:

Labels changed: removed feature.

rsc commented

Comment 26:

Labels changed: added release-none, removed go1.3maybe.

rsc commented

Comment 27:

Labels changed: added repo-main.

Go member

Comment 28:

today i discovered one more issue worth considering when trying to fix this issue.

when a package contains c++ code is built and then distributed without source code
(or with a dummy document-only source file), cmd/go won't be able to figure out
to use -extld g++ to link the program that uses the package.

Either we record the ld requirement in object files, or we link the required libstdc++
into the cgo.o file.

Comment 29:

FYI, linking libstdc++ into a .o file won't work.  The C++ library internals change over
time.  The external interface remains backward compatible at the binary level, but you
can't reliably mix and match some internal code across libstdc++ versions.
Go member

Comment 30:

yeah, i also figured that we shouldn't relying on static linking libstdc++ anyway.

so our only option would be to record the ld requirement in the header.
@rsc rsc added this to the Unplanned milestone

The mtime hack appears to have stopped working in go1.5rc1.


It appears the plumbing is there to allow binary only packages, but it is explicitly removed:

    // TODO: After Go 1, decide when to pass build.AllowBinary here.
    // See issue 3268 for mistakes to avoid.
    bp, err := buildContext.Import(path, srcDir, build.ImportComment)

It seems the problem is there is no way in the current setup to know if a package was released as binary only of if the sources were removed. Perhaps simply having an "binary only" package directory. The .a files in that package directory would not require sources. For example "pkgbin" or "pgk/binary". A flag to go build could be used by the vendor to specify the destination should be the binary directory but it should rebuild from sources.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.