Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Already on GitHub? Sign in to your account

cmd/go: work when binaries are available but source is missing #2775

Closed
rsc opened this Issue Jan 24, 2012 · 53 comments

Comments

Projects
None yet
Contributor

rsc commented Jan 24, 2012

cmd/go: work when binaries are available but source is missing
Contributor

rsc commented Jan 30, 2012

Comment 1:

Labels changed: added go1-must.

Contributor

rsc commented Mar 1, 2012

Comment 2:

This issue was closed by revision b03a5f6.

Status changed to Fixed.

Comment 3:

After building from the tip @ commit a461bcce05f6 which fixed this issue, I have a new
issue while trying to go install libraries from or into my GOPATH. I think I've managed
to isolate the circumstances under which this happens.
My GOPATH package dir is:
    /home/ganderson/go/pkg/linux_amd64
First, here's what actually works.
As long as there are no binary packages installed to ~/go/pkg then I can go build any
package whose source is located in ~/go/src:
    ☿ ganderson@excession [17:48:09] ~/src/go/golang  on default at tip?
    : go build -v github.com/ha/doozer
    code.google.com/p/goprotobuf/proto
    github.com/kr/pretty.go
    github.com/ha/doozer
I can successfully go install a single package which has no dependancies:
    ☿ ganderson@excession [17:48:12] ~/src/go/golang  on default at tip?
    : go install -v github.com/kr/pretty.go/
    github.com/kr/pretty.go
No here is where things start to break. Once I try to build or install a package which
has a dependency already installed as a package, the command fails... Here is the
structure of ~/go/pkg after the previous install command:
    ☿ ganderson@excession [17:52:27] ~/src/go/golang  on default at tip?
    : tree ~/go/pkg/linux_amd64/
    /home/ganderson/go/pkg/linux_amd64/                                                                                                                                                                                                          
    └── github.com                                                                                                                                                                                                                               
        └── kr                                                                                                                                                                                                                                   
            └── pretty.go.a                                                                                                                                                                                                                      
                                                                                                                                                                                                                                             
    2 directories, 1 file
Now if I try to either go build or go install a package which has
github.com/kr/pretty.go as a dependency the command fails, e.g.
    ☿ ganderson@excession [17:54:01] ~/src/go/golang  on default at tip?                                                                                                                                                                         
    : go build -v github.com/ha/doozer                                                                                                                                                                                                           
    code.google.com/p/goprotobuf/proto                                                                                                                                                                                                           
    github.com/ha/doozer                                                                                                                                                                                                                         
    # github.com/ha/doozer                                                                                                                                                                                                                       
    ../../../go/src/github.com/ha/doozer/conn.go:7: can't find import: "github.com/kr/pretty.go"
If I clean my ~/go/pkg dir and try to go install github.com/ha/doozer directly:
    ☿ ganderson@excession [17:55:55] ~/src/go/golang  on default at tip?                                                                                                                                                                         
    : go install -v github.com/ha/doozer
    code.google.com/p/goprotobuf/proto                                                                                                                                                                                                           
    github.com/kr/pretty.go                                                                                                                                                                                                                      
    github.com/ha/doozer                                                                                                                                                                                                                         
    # github.com/ha/doozer                                                                                                                                                                                                                       
    ../../../go/src/github.com/ha/doozer/conn.go:4: can't find import: "code.google.com/p/goprotobuf/proto"
Note that both pretty.go and protobuf are both compiled and installed to ~/go/pkg during
the command but trying to build doozer against these fails.
    ganderson@excession [17:58:26] ~/src/go/golang  on default at tip?                                                                                                                                                                         
    : tree ~/go/pkg/linux_amd64/                                                                                                                                                                                                         
    /home/ganderson/go/pkg/linux_amd64/                                                                                                                                                                                                          
    ├── code.google.com                                                                                                                                                                                                                          
    │   └── p                                                                                                                                                                                                                                    
    │       └── goprotobuf                                                                                                                                                                                                                       
    │           └── proto.a                                                                                                                                                                                                                      
    └── github.com                                                                                                                                                                                                                               
        └── kr                                                                                                                                                                                                                                   
            └── pretty.go.a                                                                                                                                                                                                                      
                                                                                                                                                                                                                                             
    5 directories, 2 files
To summarise the issue - I can go build or go install packages in GOPATH, as long as a
dependency is not already installed to $GOPATH/pkg. This issue does *not* appear to
affect GOROOT.

Comment 4:

Here's my env, in case it's relevant.
GOBIN=/usr/bin
GOARCH=amd64
GOROOT=/usr/lib64/go
GOOS=linux
GOPATH=/home/ganderson/go
Contributor

rsc commented Mar 2, 2012

Comment 5:

Thanks for the detailed update.

Owner changed to builder@golang.org.

Status changed to Accepted.

Comment 6:

I think one of your commits this week has resolved this, none of the issues I previously
reported are showing in weekly.2012-03-04. All packages are building ok whether its via
source or pre-built binary in GOPATH.
Contributor

rsc commented Mar 7, 2012

Comment 7:

We were passing a not-quite-right list of directories to 6g; it is possible that the fix
fixed this.

Status changed to Retracted.

Comment 8:

This issue has re-appeared again in go1 release. It happens for all packages that import
a third party lib which is installed as as a binary only. The symptoms as described in
comment #3 fit exactly the problem as noticed in go1 release.
The previous release where this does not happen is weekly.2012-03-22

Comment 9:

I've just noticed that unlike comment #3, this problem also now happens if the packages
are installed to $GOROOT as well or instead of $GOPATH

Comment 10:

Sorry, a further clarification. If source is available the build completes. What I am
seeing then is that it's not possible to build using third party imports where the
import is installed only as binary package in either $GOROOT or $GOPATH:
    • ganderson@excession [01:05:33] ~ 
    : tree $GOPATH/pkg
    /home/ganderson/go/pkg
    └── linux_amd64
        ├── code.google.com
        │   └── p
        │       └── goprotobuf
        │           ├── proto.a
        │           └── protoc-gen-go
        │               ├── descriptor.a
        │               ├── generator.a
        │               └── plugin.a
        └── github.com
            └── kr
                └── pretty.a
    
    7 directories, 5 files
    • ganderson@excession [02:44:53] ~ 
    : go build -x -v github.com/ha/doozer...
    WORK=/tmp/go-build734680617
    conn.go:4:2: import "code.google.com/p/goprotobuf/proto": cannot find package
    conn.go:7:2: import "github.com/kr/pretty": cannot find package
Contributor

rsc commented Apr 2, 2012

Comment 11:

I removed support for this shortly before Go 1, because it was causing worse problems.
My intention is to allow this usage in Go 1.1, but only if there is no src directory 
in the GOPATH tree in question.  That is, if you create a directory listed in GOPATH
that has _no source at all_, then it will be treated as binary-only.

Labels changed: added priority-later, go1.1, removed priority-go1, go1-must.

Status changed to Accepted.

Contributor

rsc commented Dec 10, 2012

Comment 12:

Labels changed: added size-l.

Contributor

davecheney commented Jan 9, 2013

Comment 13:

Issue #4635 has been merged into this issue.

mewmew commented Feb 12, 2013

Comment 14:

This issue was one of the most serious limitations of the "Distributed Systems" course
[1] at Carnegie Mellon University.
Quote:
"You can't hand out precompiled library code and use the go build tools
This was one of the most serious limitations we encountered.  We tried several (hacky)
approaches and none worked reliably enough for the class environment (we need to try the
mtime hack again on AFS).  This meant that we had no way to hand out binary reference
code for the students to test against.  Nor could we hand out reference implementations
for students who couldn't get part of the project working." [2]
[1]: http://www.cs.cmu.edu/~dga/15-440/F12/index.html
[2]: http://da-data.blogspot.se/2013/02/teaching-distributed-systems-in-go.html
Contributor

davecheney commented Feb 12, 2013

Comment 15:

@r.eklind: yeah, that sounds like a nasty limitation. What would be the best solution
for you that would solve this ?

Owner changed to ---.

Contributor

dave-andersen commented Feb 15, 2013

Comment 16:

chiming in, @dave:  The mechanism that Russ discussed as a potential solution in Go1.1
would be just peachy.  We have total control over the source directory tree that the
students start out with, so we can basically use anything.  The limitations are that
they typically extract it from a tar file, we don't want to have to have them run any
additional commands (touch(1)'ing files), and we'd love it to work on a variety of
platforms and filesystems (many students run the code on AFS;  OSes used span the gamut,
etc).  That's why I point to the mtime hack as a bit too fragile for our environment.
My ideal solution would let us:
  - Hand out *one* tarball that contains precompiled packages for multiple architectures;
  - Assuming GOPATH is set properly, the students could simply import ("foo") and be able to use the package.
What other information in particular would be useful to you as far as "the best
solution"?
Contributor

rsc commented Mar 12, 2013

Comment 17:

[The time for maybe has passed.]
Contributor

rsc commented Mar 12, 2013

Comment 18:

Labels changed: added go1.1maybe, removed go1.1.

Comment 19 by CundraMihail:

I don't want to pollute this issue, but since I've spent so much time and finally
stumbled onto this (apparently relevant issue) it seems relevant.  I've tried searching
quite a few times through every Go book I own, the golang.org site, as well as the
golang-nuts group and I couldn't find anything that mentioned anywhere that it wasn't
possible to use a Go package with no source available.  It seems like this simple fact
would be mentioned somewhere?
I fully expected to be able to have:
$GOPATH/
  pkg/arch/dummy.a
  src/test/test.go
have test.go contain 
import 'dummy'
and have it work.   Instead it only works if all source for dummy is available in
src/dummy.  What's even the point of having dummy.a then?
Go requiring all source to always be available for all packages seems like a pretty
significant requirement to mention somewhere, don't you think?
Contributor

robpike commented May 18, 2013

Comment 20:

Labels changed: added go1.2maybe, removed go1.1maybe.

Contributor

rsc commented Jul 30, 2013

Comment 21:

Labels changed: added feature.

Contributor

robpike commented Aug 16, 2013

Comment 22:

Too late for Go 1.2.

Labels changed: added go1.3maybe, removed go1.2maybe.

Contributor

robpike commented Aug 20, 2013

Comment 23:

Labels changed: removed go1.3maybe.

Contributor

rsc commented Nov 27, 2013

Comment 24:

Labels changed: added go1.3maybe.

Contributor

rsc commented Nov 27, 2013

Comment 25:

Labels changed: removed feature.

Contributor

rsc commented Dec 4, 2013

Comment 26:

Labels changed: added release-none, removed go1.3maybe.

Contributor

rsc commented Dec 4, 2013

Comment 27:

Labels changed: added repo-main.

Member

minux commented Dec 19, 2013

Comment 28:

today i discovered one more issue worth considering when trying to fix this issue.
when a package contains c++ code is built and then distributed without source code
(or with a dummy document-only source file), cmd/go won't be able to figure out
to use -extld g++ to link the program that uses the package.
Either we record the ld requirement in object files, or we link the required libstdc++
into the cgo.o file.
Contributor

ianlancetaylor commented Dec 19, 2013

Comment 29:

FYI, linking libstdc++ into a .o file won't work.  The C++ library internals change over
time.  The external interface remains backward compatible at the binary level, but you
can't reliably mix and match some internal code across libstdc++ versions.
Member

minux commented Dec 19, 2013

Comment 30:

yeah, i also figured that we shouldn't relying on static linking libstdc++ anyway.
so our only option would be to record the ld requirement in the header.

@rsc rsc added this to the Unplanned milestone Apr 10, 2015

The mtime hack appears to have stopped working in go1.5rc1.

Contributor

pborman commented Aug 28, 2015

It appears the plumbing is there to allow binary only packages, but it is explicitly removed:

    // TODO: After Go 1, decide when to pass build.AllowBinary here.
    // See issue 3268 for mistakes to avoid.
    bp, err := buildContext.Import(path, srcDir, build.ImportComment)

It seems the problem is there is no way in the current setup to know if a package was released as binary only of if the sources were removed. Perhaps simply having an "binary only" package directory. The .a files in that package directory would not require sources. For example "pkgbin" or "pgk/binary". A flag to go build could be used by the vendor to specify the destination should be the binary directory but it should rebuild from sources.

Contributor

pborman commented Mar 29, 2016

I think this could be addressed very simply in a variety of ways. Since I assume people want to keep the "ignore any .a for which there are no .go files" concept, we can define a way for the .go file to specify this. Here are some example contents:

package foo

If foo.go is the only source file for package foo, and it only contains these two words, then do not rebuild the .a file.

package foo
//go:norebuild

A directive to indicate the compiler should never rebuild this file. This would let the .go file contain documentation, possibly even stub function definitions so godoc would continue to work.

Or pick any other indicator.

Contributor

pborman commented Mar 29, 2016

#12186 is the same bug as this.

Contributor

pborman commented Mar 30, 2016

#12186 was closed, but it was marked as Go1.7Early while this one is not. Can we get this one marked as Go1.7Early as well?

Owner

bradfitz commented Mar 30, 2016

@pborman, Go1.NEarly means "if it's going to happen for Go 1.N, it better happen very early in the cycle". The Go 1.7 cycle is well past early at this point.

@rsc rsc modified the milestone: Go1.7, Unplanned Mar 30, 2016

Contributor

rsc commented Mar 30, 2016

I intend to look at this if I can find time in the next few weeks, since Paul reached out to me with a real use case that I didn't realize was already happening in the wild (vendor-supplied package binaries).

@rsc: Wouldn't we have to fix the .a format for this first? Or will it be 1.7 .a files can only be used with 1.7 compiler/linker?

Contributor

rsc commented Mar 30, 2016

It's already the case (very intentionally) that you can only use package binaries compiled with the exact same version of the toolchain you are using. So 1.6 package binaries only work with 1.6, and yes 1.7 package binaries will only work with 1.7. The use case assumes the producer and consumer agree on a go toolchain.

Member

minux commented Mar 30, 2016

Contributor

pborman commented Mar 30, 2016

Enabling binary only packages does not require go get. Once binary only packages are possible, someone could look at what to do with go get. I would not want enabling binary packages to be gated on a conversation about go get.

eliquious commented Apr 15, 2016 edited

We also have a need to use an archive file provided to us by another company. What about a linkarchive flag followed by a list of comma delimited libraries to use. Using a linkarchive flag would also be consistent with the new linkshared flag.


The go tool could either search the pkg dir or the current directory like so:

Search $GOPATH/pkg dir

Requires user to put the .a file in it's proper place.

Commands:

cp lib.a $GOPATH/pkg/darwin_amd64/github.com/user/lib
go build -linkarchive github.com/user/lib

Search $PWD/pkg

Searching the current directory would prevent the additional step of copying the archive and allow easier management of supporting multiple OSs.

Directory structure:

- $GOPATH/src/github.com/user/main
    \- pkg/darwin_amd64/
        \- github.com/user/lib
             - lib.a

Command:

go build -linkarchive github.com/user/lib

@gopherbot gopherbot pushed a commit to golang/proposal that referenced this issue Apr 25, 2016

@rsc rsc design: add design doc for binary-only packages
For golang/go#2775.

Change-Id: I44e1f8687c1b75381ff3cde5f7f659b13c717c44
Reviewed-on: https://go-review.googlesource.com/22431
Run-TryBot: Russ Cox <rsc@golang.org>
Reviewed-by: Russ Cox <rsc@golang.org>
6599001

CL https://golang.org/cl/22433 mentions this issue.

CL https://golang.org/cl/22432 mentions this issue.

Contributor

dlsniper commented Apr 26, 2016 edited

Hi,

I have a few issues with this proposal:

  • how will vendoring work in this context. One can argue that vendoring is already a half baked solution, but with this proposal ignoring vendoring it seems yet another burden to be solved. Concrete case: package A depends on package BinOnly, how does package C which vendors package A works?
  • the proposal says that code in the Go files is optional. This basically breaks any editor that has to work with source code. To examples that I have in mind are the Go plugin for IntelliJ which relies on the source code in GOPATH/src to work and the other is go-code which uses .a files to work. If I understand correctly, the proposal says that .a files are considered to be binary? If this is the case it means that the IntelliJ plugin will have to start parsing .a files as well, which would be ok but the do those packages include documentation for the exported types as well? Also, where's the documentation describing the .a file format (and please don't point me to the spurce code doing the work, that is not proper documentation / description)?
  • what will be the packaging for this? Like I've mentioned earlier, if this uses .a then some things will still be possible but if the files are going to be true compiled binaries then how does it work (with the above two concerns)
  • how does the binary package works for keeping it compatible between Go version? If I ship a Go 1.7 compiled binary will that work in 1.8?
  • this feels rushed in into 1.7. Would you consider waiting for 1.8 to have a proper discussion around it?
Contributor

robpike commented Apr 26, 2016

I agree it seems rushed. Perhaps it should be marked as an experimental feature that will soak for one cycle before becoming official.

Contributor

rsc commented Apr 26, 2016

This feature is not intended for general use. It is intended for the very specific case of "company X wants to sell company Y a binary version of a package (or the less commercial setting from @dave-andersen: teacher X wants to give student Y a reference solution)". I've heard from multiple, diverse sources that this is already happening today, with hacks of various kinds (see background in doc). We can't stop this from happening, but we can make sure that those people don't get stuck on Go 1.4 because the go command has gotten too good at noticing the source code is missing.

To answer the question "what about tool X that needs source?", as mentioned in the doc the answer is "tool X will not work with such packages, and hopefully that will help limit the use of such things." In particular this makes no sense in any open source context such as "go get" or Linux distributions, which is why for example "go get" will not support such arrangements. But again Go is used for more than just open source.

I'm happy to say the details here are experimental in case other details arise (not that the tools are covered by the compatibility guarantee anyway) or somehow it's not restrictive enough and starts to be used inappropriately, but there is a real need here, just one felt disproportionally by people outside the open source community and not on GitHub.

The IntelliJ plugin can stay as it is, or it might make sense to use go/types reading the binaries anyway, depending on the exact needs of the plugin. But it's fine if it doesn't work with binary-only packages. The more helpful tools that don't support binary-only packages and discourage their use, the better. But they do need to work for actually building a program.

Contributor

dlsniper commented Apr 26, 2016 edited

Thanks for feedback.

Again, I'm not against this feature landing in the Go distribution, I can see the benefits of having it.
However, Go, unlike C, doesn't have header files (at least until now).
From my understanding of the phrase:

Having a “fake” version of the source code also provides a way to supply documentation compatible with “go doc” and “godoc” even though the complete source code is missing.

It does seem that one will need to provide a minimal definition of the package, something like:

// Copyright 2016 dlsniper

//go:binary-only-package
package awesome

// This is entirely optional

// The Awesomizer interface allows you to make awesome code
type Awesomizer interface{
    CodeIt()
}

At which point I would rather prefer to enforce the stub package to contain the public definitions (and hopefully documentation for this) rather than just the package name and the special flag + other build flags. This would then mean that the documentation of the package has to be respected and it would help out users using such packages.

Speaking of flags, rather than introduce a new flag for the go tool, wouldn't it be better to introduce a new, static build tag? It can keep the name binary-only-package but then as a developer I could at least test out how this works in the code and still have the code for the package as well. This would also represent one less thing that Go developers have to learn and keep track of (not everyone knows the //go: flags (and yes, I know of go:generate)

Again, to address the 🐘 in the issue: what about vendoring? How would that work in this context? There are already problems with the current vendoring approach which came up a few times on the mailing list and have only resulted in workarounds / "developers should learn more / read / understand better" which is not very real-life unfortunately. Adding a binary only as a vendored dependency, even if internally, would not be something that people will skip, I'm positive this will happen.

The more helpful tools that don't support binary-only packages and discourage their use, the better.

I'm willing to wage a bet that within a couple of weeks of this feature landing people will ask for this feature to work. And like I've said, maybe for the IntelliJ plugin will do some support for it (or not, I can't speak of that right now) but if it's there, people will want to use it (as you've said, they already do it) as it would be a standard feature (which is exactly why people have editors like Atom, VSCode or IntelliJ with all sorts of plugins helping them write the code. But I don't wan to digress here on how people write code).

To sum up, imho:

  • public types should be present in the stub Go package (which is expected to be delivered apparently from the proposal)
  • vendoring needs to be addressed before merging this, not after (even if it means clear documentation of the use-cases and how to approach them)
  • maybe build tags could be used instead of the //go:binary-only "flag" or how we call those

Thank you.

EDIT:
I've changed the build flag to build tag, sorry for the wrong names.

Contributor

dlsniper commented Apr 26, 2016

There's also of course the issue of ensuring that the binary package being compiled is the one that the authors intended to distribute so maybe having a checksum for the binary present in the stub package would help the go tool to validate it at compile time?

Contributor

pborman commented Apr 26, 2016

This has been an issue since 1.5 and has been ignored since then (it has
always been "we will address in release N+1"). Kicking the can down the
road has proven to not be helpful. What is nice about what Russ has
implemented is that the owner of such code can pick the level of
interoperability they desire. They can provide the minimal:

//go:binary-only-package

package mine

Or they can provide more useful information:

//go:binary-only-package

// Package mine implements our secret Sauce.

package mine

// A Value contains information about a value.
type Value struct {
Name string // name of the value
_ struct{}
}

// Int returns the integer value of v.
func (v *Value) Int() int

// Mine is a super secret function that takes
// x and produces y using the confidential
// Sauce function:
func Mine(x Value) (y Value)

This will actually work with go doc:

$ go doc
package mine // import "c"

func Mine(x Value) (y Value)
type Value struct { ... }
$ go doc Mine
func Mine(x Value) (y Value)
Mine is a super secret function that takes x and produces y using the
confidential Sauce function:
$ go doc Value
type Value struct {
Name string // name of the value
// Has unexported fields.
}
A Value contains information about a value.

func Mine(x Value) (y Value)
func (v *Value) Int() int

I don't think the latter needs to be required, though I would hope most
producers would include more detail than less. Perhaps someone will even
write a tool that takes full source code, strips out everything not needed
by go doc, and adds the //go:binary-only-package comment to the top.

Adding checksums is interesting, but in this case I would venture to say
that versioning is more useful as it will immediately tell you which
version is more up to date.

I really thank Russ for doing this as I have been one of those affected by
the deteriorating "support" for binary only packages (as a consumer, not a
producer). I think he has implemented a solution that has the least impact
and is the easiest to move forward with new features, if needed. Once out
there, those who need binary only, non-open source packages will be able to
tell us how well it is working and if there are changes that should be
made. I think breaking how this works between releases is still better
than the status-quo. Being called experimental is fine, as long as it
satisfies the primary goal (making it possible to release binary only
packages).

-Paul

On Tue, Apr 26, 2016 at 7:06 AM, Russ Cox notifications@github.com wrote:

This feature is not intended for general use. It is intended for the
very specific case of "company X wants to sell company Y a binary version
of a package (or the less commercial setting from @dave-andersen
https://github.com/dave-andersen: teacher X wants to give student Y a
reference solution)". I've heard from multiple, diverse sources that this
is already happening today, with hacks of various kinds (see background in
doc). We can't stop this from happening, but we can make sure that
those people don't get stuck on Go 1.4 because the go command has gotten
too good at noticing the source code is missing.

To answer the question "what about tool X that needs source?", as
mentioned in the doc the answer is "tool X will not work with such
packages, and hopefully that will help limit the use of such things." In
particular this makes no sense in any open source context such as "go get"
or Linux distributions, which is why for example "go get" will not support
such arrangements. But again Go is used for more than just open source.

I'm happy to say the details here are experimental in case other details
arise (not that the tools are covered by the compatibility guarantee
anyway) or somehow it's not restrictive enough and starts to be used
inappropriately, but there is a real need here, just one felt
disproportionally by people outside the open source community and not on
GitHub.

The IntelliJ plugin can stay as it is, or it might make sense to use
go/types https://goto.google.com/types reading the binaries anyway,
depending on the exact needs of the plugin. But it's fine if it doesn't
work with binary-only packages. The more helpful tools that don't support
binary-only packages and discourage their use, the better. But they do need
to work for actually building a program.


You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#2775 (comment)

Contributor

dlsniper commented Apr 26, 2016

This has been an issue since 1.5 and has been ignored since then (it has always been "we will address in release N+1"). Kicking the can down the road has proven to not be helpful.

@pborman to be a bit mean, that's what they do with generics for the last 6 years, no? Not to mention other issues (multiple package coverage, etc...) I don't want to go there but this is not a technical argument for a good implementation of a solution imho.

They can provide the minimal [...] Or they can provide more useful information [...]

So why not force those who want this feature to do proper documentation / packaging for it? @rsc says that it's ok to make it painful for end-users, why not make it painful (and get real value of it) for producers as well?

Adding checksums is interesting, but in this case I would venture to say that versioning is more useful as it will immediately tell you which version is more up to date.

We don't want to have go get aware on versioning so why would binary only packages be different? Also checksums provide a different functionality than versioning.

I really thank Russ for doing this as I have been one of those affected by the deteriorating "support" for binary only packages (as a consumer, not a producer).

I too am grateful for his work and all the other contributors to the Go ecosystem, don't get me wrong. But like I've stated, this feels bad. I agree with what @robpike suggested to flag it as experimental. Meanwhile we could still discuss about this, no?

I feel I don't stress enough for this. I'm not against the idea of having this implemented, not even a bit. I'm against the idea of rushing it to get in for 1.7 just because it's something people are doing today. Vendoring has more problems and I can argue that it's a problem for more people than binary-only packages are. I don't want to force this comparison anymore but this doesn't sound inline with the Go vision of offering pragmatic solutions for problems people have.

I've raised a few points to at least have a decent discussion on how to solve them and, if solutions are found then I don't see any problem for getting this into 1.7.

Contributor

pborman commented Apr 26, 2016

I don't think anyone has said "no, it should not be marked experimental".
That is a fine idea. One really nice thing about the solution is that if
you don't need binary-only packages then you will not notice this at all.
If you do need binary-only packages, you will be very grateful that this is
part of 1.7. This will also make Go more appealing for commercial
products. The GNU C compiler was going nowhere fast (in the commercial
world) until RMS relented and introduced the LGPL.

On Tue, Apr 26, 2016 at 8:18 AM, Florin Pățan notifications@github.com
wrote:

This has been an issue since 1.5 and has been ignored since then (it has
always been "we will address in release N+1"). Kicking the can down the
road has proven to not be helpful.

@pborman https://github.com/pborman to be a bit mean, that's what they
do with generics for the last 6 years, no? Not to mention other issues
(multiple package coverage, etc...) I don't want to go there but this is
not a technical argument for a good implementation of a solution imho.

At least generics work as well today as they did in 1.4.2 and 1.0 :-) I
can't really say that about binary only package releases.

The reason this is being "rushed" into 1.7 is because it was not done for
1.6 after 1.5 broken the existing recipes, and people are being held back
to 1.4.2. This is something I have had discussions about with various
members of the Go team since 1.5. There have been issues filed about it.

It seems you are saying that Russ did not go far enough and those questions
(how to make go get work, how does IntelliJ work with it, what about
vendoring) must be answered before anything goes in. I disagree, I think
Russ took the appropriate minimal solution for 1.7. All those other things
can be discussed and addressed once we have more experience. I don't think
it would be possible to get a good answer for vendoring at this point as
the vendoring we have already has enough questions. I can see an easy
possible answer to vendoring, but it assumes that binary-only packages are
freely redistributable, but I don't think that will be the case. You are
correct that that vendoring needs more thought.

So, by all means, let's mark it experimental, let's have more discussions
and debates, but let's not let those things from precluding this common
sense minimal solution for 1.7. Right now it feels that the discussion has
been how to put up roadblocks to getting this minimal solution into 1.7.

-Paul

@mk0x9 mk0x9 pushed a commit to mk0x9/go that referenced this issue Apr 27, 2016

@rsc rsc cmd/go: add Package.StaleReason for debugging with go list
It comes up every few months that we can't understand why
the go command is rebuilding some package.
Add diagnostics so that the go command can explain itself
if asked.

For #2775, #3506, #12074.

Change-Id: I1c73b492589b49886bf31a8f9d05514adbd6ed70
Reviewed-on: https://go-review.googlesource.com/22432
Reviewed-by: Rob Pike <r@golang.org>
0b5fbf7

@gopherbot gopherbot closed this in af6aa0f Apr 29, 2016

Contributor

rsc commented Apr 29, 2016

I submitted the CL. We will note that it is experimental in the release notes.

To reply briefly to some points:

  • It doesn't work to be a build tag. Build tags are about selection of files to compile, not about changing the meaning of the compilation. Independent concerns, independent mechanisms. And //go: for prefixes is the established standard for source annotations directed at the go toolchain (but we're not going to retroactively change // +build).
  • If the binary-only package has the wrong imports inside it, likely the best way to address that would be to go back to the person who sold you the package and get them fixed. But you could also rewrite the binary form to change import paths if needed. I am skeptical it will come to that. And note that if you're happy with the imports being used in the binary-only package, you can choose your own import path for it simply by moving it somewhere else in the file system.
  • I don't think there's anything interesting about vendoring here except that it's one rationale for wanting a different import path.
  • If any tools want to not support these binary blobs, that's fine with me: you can say "go get doesn't support them either."

@gopherbot gopherbot locked and limited conversation to collaborators Apr 29, 2017

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.