Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

cmd/go: work when binaries are available but source is missing #2775

Open
rsc opened this Issue · 32 comments
@rsc
Owner
cmd/go: work when binaries are available but source is missing
@rsc
Owner

Comment 1:

Labels changed: added go1-must.

@rsc
Owner
rsc commented

Comment 2:

This issue was closed by revision b03a5f6.

Status changed to Fixed.

@gnanderson

Comment 3:

After building from the tip @ commit a461bcce05f6 which fixed this issue, I have a new
issue while trying to go install libraries from or into my GOPATH. I think I've managed
to isolate the circumstances under which this happens.

My GOPATH package dir is:

    /home/ganderson/go/pkg/linux_amd64

First, here's what actually works.

As long as there are no binary packages installed to ~/go/pkg then I can go build any
package whose source is located in ~/go/src:

    ☿ ganderson@excession [17:48:09] ~/src/go/golang  on default at tip?
    : go build -v github.com/ha/doozer
    code.google.com/p/goprotobuf/proto
    github.com/kr/pretty.go
    github.com/ha/doozer

I can successfully go install a single package which has no dependancies:

    ☿ ganderson@excession [17:48:12] ~/src/go/golang  on default at tip?
    : go install -v github.com/kr/pretty.go/
    github.com/kr/pretty.go

No here is where things start to break. Once I try to build or install a package which
has a dependency already installed as a package, the command fails... Here is the
structure of ~/go/pkg after the previous install command:

    ☿ ganderson@excession [17:52:27] ~/src/go/golang  on default at tip?
    : tree ~/go/pkg/linux_amd64/
    /home/ganderson/go/pkg/linux_amd64/                                                                                                                                                                                                          
    └── github.com                                                                                                                                                                                                                               
        └── kr                                                                                                                                                                                                                                   
            └── pretty.go.a                                                                                                                                                                                                                      
                                                                                                                                                                                                                                             
    2 directories, 1 file

Now if I try to either go build or go install a package which has
github.com/kr/pretty.go as a dependency the command fails, e.g.

    ☿ ganderson@excession [17:54:01] ~/src/go/golang  on default at tip?                                                                                                                                                                         
    : go build -v github.com/ha/doozer                                                                                                                                                                                                           
    code.google.com/p/goprotobuf/proto                                                                                                                                                                                                           
    github.com/ha/doozer                                                                                                                                                                                                                         
    # github.com/ha/doozer                                                                                                                                                                                                                       
    ../../../go/src/github.com/ha/doozer/conn.go:7: can't find import: "github.com/kr/pretty.go"

If I clean my ~/go/pkg dir and try to go install github.com/ha/doozer directly:

    ☿ ganderson@excession [17:55:55] ~/src/go/golang  on default at tip?                                                                                                                                                                         
    : go install -v github.com/ha/doozer
    code.google.com/p/goprotobuf/proto                                                                                                                                                                                                           
    github.com/kr/pretty.go                                                                                                                                                                                                                      
    github.com/ha/doozer                                                                                                                                                                                                                         
    # github.com/ha/doozer                                                                                                                                                                                                                       
    ../../../go/src/github.com/ha/doozer/conn.go:4: can't find import: "code.google.com/p/goprotobuf/proto"

Note that both pretty.go and protobuf are both compiled and installed to ~/go/pkg during
the command but trying to build doozer against these fails.

    ganderson@excession [17:58:26] ~/src/go/golang  on default at tip?                                                                                                                                                                         
    : tree ~/go/pkg/linux_amd64/                                                                                                                                                                                                         
    /home/ganderson/go/pkg/linux_amd64/                                                                                                                                                                                                          
    ├── code.google.com                                                                                                                                                                                                                          
    │   └── p                                                                                                                                                                                                                                    
    │       └── goprotobuf                                                                                                                                                                                                                       
    │           └── proto.a                                                                                                                                                                                                                      
    └── github.com                                                                                                                                                                                                                               
        └── kr                                                                                                                                                                                                                                   
            └── pretty.go.a                                                                                                                                                                                                                      
                                                                                                                                                                                                                                             
    5 directories, 2 files


To summarise the issue - I can go build or go install packages in GOPATH, as long as a
dependency is not already installed to $GOPATH/pkg. This issue does *not* appear to
affect GOROOT.
@gnanderson

Comment 4:

Here's my env, in case it's relevant.

GOBIN=/usr/bin
GOARCH=amd64
GOROOT=/usr/lib64/go
GOOS=linux
GOPATH=/home/ganderson/go
@rsc
Owner
rsc commented

Comment 5:

Thanks for the detailed update.

Owner changed to builder@golang.org.

Status changed to Accepted.

@gnanderson

Comment 6:

I think one of your commits this week has resolved this, none of the issues I previously
reported are showing in weekly.2012-03-04. All packages are building ok whether its via
source or pre-built binary in GOPATH.
@rsc
Owner
rsc commented

Comment 7:

We were passing a not-quite-right list of directories to 6g; it is possible that the fix
fixed this.

Status changed to Retracted.

@gnanderson

Comment 8:

This issue has re-appeared again in go1 release. It happens for all packages that import
a third party lib which is installed as as a binary only. The symptoms as described in
comment #3 fit exactly the problem as noticed in go1 release.

The previous release where this does not happen is weekly.2012-03-22
@gnanderson

Comment 9:

I've just noticed that unlike comment #3, this problem also now happens if the packages
are installed to $GOROOT as well or instead of $GOPATH
@gnanderson

Comment 10:

Sorry, a further clarification. If source is available the build completes. What I am
seeing then is that it's not possible to build using third party imports where the
import is installed only as binary package in either $GOROOT or $GOPATH:


    • ganderson@excession [01:05:33] ~ 
    : tree $GOPATH/pkg
    /home/ganderson/go/pkg
    └── linux_amd64
        ├── code.google.com
        │   └── p
        │       └── goprotobuf
        │           ├── proto.a
        │           └── protoc-gen-go
        │               ├── descriptor.a
        │               ├── generator.a
        │               └── plugin.a
        └── github.com
            └── kr
                └── pretty.a
    
    7 directories, 5 files

    • ganderson@excession [02:44:53] ~ 
    : go build -x -v github.com/ha/doozer...
    WORK=/tmp/go-build734680617
    conn.go:4:2: import "code.google.com/p/goprotobuf/proto": cannot find package
    conn.go:7:2: import "github.com/kr/pretty": cannot find package
@rsc
Owner
rsc commented

Comment 11:

I removed support for this shortly before Go 1, because it was causing worse problems.
My intention is to allow this usage in Go 1.1, but only if there is no src directory 
in the GOPATH tree in question.  That is, if you create a directory listed in GOPATH
that has _no source at all_, then it will be treated as binary-only.

Labels changed: added priority-later, go1.1, removed priority-go1, go1-must.

Status changed to Accepted.

@rsc
Owner

Comment 12:

Labels changed: added size-l.

@davecheney
Collaborator

Comment 13:

Issue #4635 has been merged into this issue.

@mewmew

Comment 14:

This issue was one of the most serious limitations of the "Distributed Systems" course
[1] at Carnegie Mellon University.

Quote:
"You can't hand out precompiled library code and use the go build tools

This was one of the most serious limitations we encountered.  We tried several (hacky)
approaches and none worked reliably enough for the class environment (we need to try the
mtime hack again on AFS).  This meant that we had no way to hand out binary reference
code for the students to test against.  Nor could we hand out reference implementations
for students who couldn't get part of the project working." [2]

[1]: http://www.cs.cmu.edu/~dga/15-440/F12/index.html
[2]: http://da-data.blogspot.se/2013/02/teaching-distributed-systems-in-go.html
@davecheney
Collaborator

Comment 15:

@r.eklind: yeah, that sounds like a nasty limitation. What would be the best solution
for you that would solve this ?

Owner changed to ---.

@dave-andersen

Comment 16:

chiming in, @dave:  The mechanism that Russ discussed as a potential solution in Go1.1
would be just peachy.  We have total control over the source directory tree that the
students start out with, so we can basically use anything.  The limitations are that
they typically extract it from a tar file, we don't want to have to have them run any
additional commands (touch(1)'ing files), and we'd love it to work on a variety of
platforms and filesystems (many students run the code on AFS;  OSes used span the gamut,
etc).  That's why I point to the mtime hack as a bit too fragile for our environment.

My ideal solution would let us:
  - Hand out *one* tarball that contains precompiled packages for multiple architectures;
  - Assuming GOPATH is set properly, the students could simply import ("foo") and be able to use the package.

What other information in particular would be useful to you as far as "the best
solution"?
@rsc
Owner

Comment 17:

[The time for maybe has passed.]
@rsc
Owner

Comment 18:

Labels changed: added go1.1maybe, removed go1.1.

@gopherbot
Collaborator

Comment 19 by CundraMihail:

I don't want to pollute this issue, but since I've spent so much time and finally
stumbled onto this (apparently relevant issue) it seems relevant.  I've tried searching
quite a few times through every Go book I own, the golang.org site, as well as the
golang-nuts group and I couldn't find anything that mentioned anywhere that it wasn't
possible to use a Go package with no source available.  It seems like this simple fact
would be mentioned somewhere?
I fully expected to be able to have:
$GOPATH/
  pkg/arch/dummy.a
  src/test/test.go
have test.go contain 
import 'dummy'
and have it work.   Instead it only works if all source for dummy is available in
src/dummy.  What's even the point of having dummy.a then?
Go requiring all source to always be available for all packages seems like a pretty
significant requirement to mention somewhere, don't you think?
@robpike
Collaborator

Comment 20:

Labels changed: added go1.2maybe, removed go1.1maybe.

@rsc
Owner

Comment 21:

Labels changed: added feature.

@robpike
Collaborator

Comment 22:

Too late for Go 1.2.

Labels changed: added go1.3maybe, removed go1.2maybe.

@robpike
Collaborator

Comment 23:

Labels changed: removed go1.3maybe.

@rsc
Owner

Comment 24:

Labels changed: added go1.3maybe.

@rsc
Owner

Comment 25:

Labels changed: removed feature.

@rsc
Owner
rsc commented

Comment 26:

Labels changed: added release-none, removed go1.3maybe.

@rsc
Owner
rsc commented

Comment 27:

Labels changed: added repo-main.

@minux
Collaborator

Comment 28:

today i discovered one more issue worth considering when trying to fix this issue.

when a package contains c++ code is built and then distributed without source code
(or with a dummy document-only source file), cmd/go won't be able to figure out
to use -extld g++ to link the program that uses the package.

Either we record the ld requirement in object files, or we link the required libstdc++
into the cgo.o file.
@ianlancetaylor

Comment 29:

FYI, linking libstdc++ into a .o file won't work.  The C++ library internals change over
time.  The external interface remains backward compatible at the binary level, but you
can't reliably mix and match some internal code across libstdc++ versions.
@minux
Collaborator

Comment 30:

yeah, i also figured that we shouldn't relying on static linking libstdc++ anyway.

so our only option would be to record the ld requirement in the header.
@rsc rsc added this to the Unplanned milestone
@aaronbee

The mtime hack appears to have stopped working in go1.5rc1.

@pborman

It appears the plumbing is there to allow binary only packages, but it is explicitly removed:

    // TODO: After Go 1, decide when to pass build.AllowBinary here.
    // See issue 3268 for mistakes to avoid.
    bp, err := buildContext.Import(path, srcDir, build.ImportComment)

It seems the problem is there is no way in the current setup to know if a package was released as binary only of if the sources were removed. Perhaps simply having an "binary only" package directory. The .a files in that package directory would not require sources. For example "pkgbin" or "pgk/binary". A flag to go build could be used by the vendor to specify the destination should be the binary directory but it should rebuild from sources.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.