cmd/go: work when binaries are available but source is missing #2775
Comments
|
This issue was closed by revision b03a5f6. Status changed to Fixed. |
After building from the tip @ commit a461bcce05f6 which fixed this issue, I have a new
issue while trying to go install libraries from or into my GOPATH. I think I've managed
to isolate the circumstances under which this happens.
My GOPATH package dir is:
/home/ganderson/go/pkg/linux_amd64
First, here's what actually works.
As long as there are no binary packages installed to ~/go/pkg then I can go build any
package whose source is located in ~/go/src:
☿ ganderson@excession [17:48:09] ~/src/go/golang on default at tip?
: go build -v github.com/ha/doozer
code.google.com/p/goprotobuf/proto
github.com/kr/pretty.go
github.com/ha/doozer
I can successfully go install a single package which has no dependancies:
☿ ganderson@excession [17:48:12] ~/src/go/golang on default at tip?
: go install -v github.com/kr/pretty.go/
github.com/kr/pretty.go
No here is where things start to break. Once I try to build or install a package which
has a dependency already installed as a package, the command fails... Here is the
structure of ~/go/pkg after the previous install command:
☿ ganderson@excession [17:52:27] ~/src/go/golang on default at tip?
: tree ~/go/pkg/linux_amd64/
/home/ganderson/go/pkg/linux_amd64/
└── github.com
└── kr
└── pretty.go.a
2 directories, 1 file
Now if I try to either go build or go install a package which has
github.com/kr/pretty.go as a dependency the command fails, e.g.
☿ ganderson@excession [17:54:01] ~/src/go/golang on default at tip?
: go build -v github.com/ha/doozer
code.google.com/p/goprotobuf/proto
github.com/ha/doozer
# github.com/ha/doozer
../../../go/src/github.com/ha/doozer/conn.go:7: can't find import: "github.com/kr/pretty.go"
If I clean my ~/go/pkg dir and try to go install github.com/ha/doozer directly:
☿ ganderson@excession [17:55:55] ~/src/go/golang on default at tip?
: go install -v github.com/ha/doozer
code.google.com/p/goprotobuf/proto
github.com/kr/pretty.go
github.com/ha/doozer
# github.com/ha/doozer
../../../go/src/github.com/ha/doozer/conn.go:4: can't find import: "code.google.com/p/goprotobuf/proto"
Note that both pretty.go and protobuf are both compiled and installed to ~/go/pkg during
the command but trying to build doozer against these fails.
ganderson@excession [17:58:26] ~/src/go/golang on default at tip?
: tree ~/go/pkg/linux_amd64/
/home/ganderson/go/pkg/linux_amd64/
├── code.google.com
│ └── p
│ └── goprotobuf
│ └── proto.a
└── github.com
└── kr
└── pretty.go.a
5 directories, 2 files
To summarise the issue - I can go build or go install packages in GOPATH, as long as a
dependency is not already installed to $GOPATH/pkg. This issue does *not* appear to
affect GOROOT.
|
Thanks for the detailed update. Owner changed to builder@golang.org. Status changed to Accepted. |
This issue has re-appeared again in go1 release. It happens for all packages that import a third party lib which is installed as as a binary only. The symptoms as described in comment #3 fit exactly the problem as noticed in go1 release. The previous release where this does not happen is weekly.2012-03-22 |
Sorry, a further clarification. If source is available the build completes. What I am
seeing then is that it's not possible to build using third party imports where the
import is installed only as binary package in either $GOROOT or $GOPATH:
• ganderson@excession [01:05:33] ~
: tree $GOPATH/pkg
/home/ganderson/go/pkg
└── linux_amd64
├── code.google.com
│ └── p
│ └── goprotobuf
│ ├── proto.a
│ └── protoc-gen-go
│ ├── descriptor.a
│ ├── generator.a
│ └── plugin.a
└── github.com
└── kr
└── pretty.a
7 directories, 5 files
• ganderson@excession [02:44:53] ~
: go build -x -v github.com/ha/doozer...
WORK=/tmp/go-build734680617
conn.go:4:2: import "code.google.com/p/goprotobuf/proto": cannot find package
conn.go:7:2: import "github.com/kr/pretty": cannot find package
|
I removed support for this shortly before Go 1, because it was causing worse problems. My intention is to allow this usage in Go 1.1, but only if there is no src directory in the GOPATH tree in question. That is, if you create a directory listed in GOPATH that has _no source at all_, then it will be treated as binary-only. Labels changed: added priority-later, go1.1, removed priority-go1, go1-must. Status changed to Accepted. |
|
Issue #4635 has been merged into this issue. |
This issue was one of the most serious limitations of the "Distributed Systems" course [1] at Carnegie Mellon University. Quote: "You can't hand out precompiled library code and use the go build tools This was one of the most serious limitations we encountered. We tried several (hacky) approaches and none worked reliably enough for the class environment (we need to try the mtime hack again on AFS). This meant that we had no way to hand out binary reference code for the students to test against. Nor could we hand out reference implementations for students who couldn't get part of the project working." [2] [1]: http://www.cs.cmu.edu/~dga/15-440/F12/index.html [2]: http://da-data.blogspot.se/2013/02/teaching-distributed-systems-in-go.html |
chiming in, @dave: The mechanism that Russ discussed as a potential solution in Go1.1
would be just peachy. We have total control over the source directory tree that the
students start out with, so we can basically use anything. The limitations are that
they typically extract it from a tar file, we don't want to have to have them run any
additional commands (touch(1)'ing files), and we'd love it to work on a variety of
platforms and filesystems (many students run the code on AFS; OSes used span the gamut,
etc). That's why I point to the mtime hack as a bit too fragile for our environment.
My ideal solution would let us:
- Hand out *one* tarball that contains precompiled packages for multiple architectures;
- Assuming GOPATH is set properly, the students could simply import ("foo") and be able to use the package.
What other information in particular would be useful to you as far as "the best
solution"?
|
I don't want to pollute this issue, but since I've spent so much time and finally stumbled onto this (apparently relevant issue) it seems relevant. I've tried searching quite a few times through every Go book I own, the golang.org site, as well as the golang-nuts group and I couldn't find anything that mentioned anywhere that it wasn't possible to use a Go package with no source available. It seems like this simple fact would be mentioned somewhere? I fully expected to be able to have: $GOPATH/ pkg/arch/dummy.a src/test/test.go have test.go contain import 'dummy' and have it work. Instead it only works if all source for dummy is available in src/dummy. What's even the point of having dummy.a then? Go requiring all source to always be available for all packages seems like a pretty significant requirement to mention somewhere, don't you think? |
today i discovered one more issue worth considering when trying to fix this issue. when a package contains c++ code is built and then distributed without source code (or with a dummy document-only source file), cmd/go won't be able to figure out to use -extld g++ to link the program that uses the package. Either we record the ld requirement in object files, or we link the required libstdc++ into the cgo.o file. |
|
I think this could be addressed very simply in a variety of ways. Since I assume people want to keep the "ignore any .a for which there are no .go files" concept, we can define a way for the .go file to specify this. Here are some example contents: If foo.go is the only source file for package foo, and it only contains these two words, then do not rebuild the .a file. A directive to indicate the compiler should never rebuild this file. This would let the .go file contain documentation, possibly even stub function definitions so godoc would continue to work. Or pick any other indicator. |
|
#12186 is the same bug as this. |
|
#12186 was closed, but it was marked as Go1.7Early while this one is not. Can we get this one marked as Go1.7Early as well? |
|
@pborman, Go1.NEarly means "if it's going to happen for Go 1.N, it better happen very early in the cycle". The Go 1.7 cycle is well past early at this point. |
|
I intend to look at this if I can find time in the next few weeks, since Paul reached out to me with a real use case that I didn't realize was already happening in the wild (vendor-supplied package binaries). |
|
@rsc: Wouldn't we have to fix the .a format for this first? Or will it be 1.7 .a files can only be used with 1.7 compiler/linker? |
|
It's already the case (very intentionally) that you can only use package binaries compiled with the exact same version of the toolchain you are using. So 1.6 package binaries only work with 1.6, and yes 1.7 package binaries will only work with 1.7. The use case assumes the producer and consumer agree on a go toolchain. |
|
If we are going to do this, could we do it in a way such that go get don't
support downloading and building binary only packages (at least not by
default)?
I don't want to check if any packages I jut go get contains any binary only
packages every time I run go get -u.
This almost rules out putting pkg.a inside package source directories.
|
|
Enabling binary only packages does not require go get. Once binary only packages are possible, someone could look at what to do with go get. I would not want enabling binary packages to be gated on a conversation about go get. |
|
We also have a need to use an archive file provided to us by another company. What about a The go tool could either search the pkg dir or the current directory like so: Search $GOPATH/pkg dir Requires user to put the Commands: Search $PWD/pkg Searching the current directory would prevent the additional step of copying the archive and allow easier management of supporting multiple OSs. Directory structure: Command: |
For golang/go#2775. Change-Id: I44e1f8687c1b75381ff3cde5f7f659b13c717c44 Reviewed-on: https://go-review.googlesource.com/22431 Run-TryBot: Russ Cox <rsc@golang.org> Reviewed-by: Russ Cox <rsc@golang.org>
|
CL https://golang.org/cl/22433 mentions this issue. |
|
CL https://golang.org/cl/22432 mentions this issue. |
|
Hi, I have a few issues with this proposal:
|
|
I agree it seems rushed. Perhaps it should be marked as an experimental feature that will soak for one cycle before becoming official. |
|
This feature is not intended for general use. It is intended for the very specific case of "company X wants to sell company Y a binary version of a package (or the less commercial setting from @dave-andersen: teacher X wants to give student Y a reference solution)". I've heard from multiple, diverse sources that this is already happening today, with hacks of various kinds (see background in doc). We can't stop this from happening, but we can make sure that those people don't get stuck on Go 1.4 because the go command has gotten too good at noticing the source code is missing. To answer the question "what about tool X that needs source?", as mentioned in the doc the answer is "tool X will not work with such packages, and hopefully that will help limit the use of such things." In particular this makes no sense in any open source context such as "go get" or Linux distributions, which is why for example "go get" will not support such arrangements. But again Go is used for more than just open source. I'm happy to say the details here are experimental in case other details arise (not that the tools are covered by the compatibility guarantee anyway) or somehow it's not restrictive enough and starts to be used inappropriately, but there is a real need here, just one felt disproportionally by people outside the open source community and not on GitHub. The IntelliJ plugin can stay as it is, or it might make sense to use go/types reading the binaries anyway, depending on the exact needs of the plugin. But it's fine if it doesn't work with binary-only packages. The more helpful tools that don't support binary-only packages and discourage their use, the better. But they do need to work for actually building a program. |
|
Thanks for feedback. Again, I'm not against this feature landing in the Go distribution, I can see the benefits of having it.
It does seem that one will need to provide a minimal definition of the package, something like: // Copyright 2016 dlsniper
//go:binary-only-package
package awesome
// This is entirely optional
// The Awesomizer interface allows you to make awesome code
type Awesomizer interface{
CodeIt()
}At which point I would rather prefer to enforce the stub package to contain the public definitions (and hopefully documentation for this) rather than just the package name and the special flag + other build flags. This would then mean that the documentation of the package has to be respected and it would help out users using such packages. Speaking of flags, rather than introduce a new flag for the go tool, wouldn't it be better to introduce a new, static build tag? It can keep the name Again, to address the 🐘 in the issue: what about vendoring? How would that work in this context? There are already problems with the current vendoring approach which came up a few times on the mailing list and have only resulted in workarounds / "developers should learn more / read / understand better" which is not very real-life unfortunately. Adding a binary only as a vendored dependency, even if internally, would not be something that people will skip, I'm positive this will happen.
I'm willing to wage a bet that within a couple of weeks of this feature landing people will ask for this feature to work. And like I've said, maybe for the IntelliJ plugin will do some support for it (or not, I can't speak of that right now) but if it's there, people will want to use it (as you've said, they already do it) as it would be a standard feature (which is exactly why people have editors like Atom, VSCode or IntelliJ with all sorts of plugins helping them write the code. But I don't wan to digress here on how people write code). To sum up, imho:
Thank you. EDIT: |
|
There's also of course the issue of ensuring that the binary package being compiled is the one that the authors intended to distribute so maybe having a checksum for the binary present in the stub package would help the go tool to validate it at compile time? |
|
This has been an issue since 1.5 and has been ignored since then (it has //go:binary-only-package package mine Or they can provide more useful information: //go:binary-only-package // Package mine implements our secret Sauce. package mine // A Value contains information about a value. // Int returns the integer value of v. // Mine is a super secret function that takes This will actually work with go doc: $ go doc func Mine(x Value) (y Value) func Mine(x Value) (y Value) I don't think the latter needs to be required, though I would hope most Adding checksums is interesting, but in this case I would venture to say I really thank Russ for doing this as I have been one of those affected by On Tue, Apr 26, 2016 at 7:06 AM, Russ Cox notifications@github.com wrote:
|
@pborman to be a bit mean, that's what they do with generics for the last 6 years, no? Not to mention other issues (multiple package coverage, etc...) I don't want to go there but this is not a technical argument for a good implementation of a solution imho.
So why not force those who want this feature to do proper documentation / packaging for it? @rsc says that it's ok to make it painful for end-users, why not make it painful (and get real value of it) for producers as well?
We don't want to have
I too am grateful for his work and all the other contributors to the Go ecosystem, don't get me wrong. But like I've stated, this feels bad. I agree with what @robpike suggested to flag it as experimental. Meanwhile we could still discuss about this, no? I feel I don't stress enough for this. I'm not against the idea of having this implemented, not even a bit. I'm against the idea of rushing it to get in for 1.7 just because it's something people are doing today. Vendoring has more problems and I can argue that it's a problem for more people than binary-only packages are. I don't want to force this comparison anymore but this doesn't sound inline with the Go vision of offering pragmatic solutions for problems people have. I've raised a few points to at least have a decent discussion on how to solve them and, if solutions are found then I don't see any problem for getting this into 1.7. |
|
I don't think anyone has said "no, it should not be marked experimental". On Tue, Apr 26, 2016 at 8:18 AM, Florin Pățan notifications@github.com
At least generics work as well today as they did in 1.4.2 and 1.0 :-) I The reason this is being "rushed" into 1.7 is because it was not done for It seems you are saying that Russ did not go far enough and those questions So, by all means, let's mark it experimental, let's have more discussions |
It comes up every few months that we can't understand why the go command is rebuilding some package. Add diagnostics so that the go command can explain itself if asked. For golang#2775, golang#3506, golang#12074. Change-Id: I1c73b492589b49886bf31a8f9d05514adbd6ed70 Reviewed-on: https://go-review.googlesource.com/22432 Reviewed-by: Rob Pike <r@golang.org>
|
I submitted the CL. We will note that it is experimental in the release notes. To reply briefly to some points:
|
The text was updated successfully, but these errors were encountered: