New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: drop support for binary-only packages #28152

Open
rsc opened this Issue Oct 11, 2018 · 29 comments

Comments

Projects
None yet
@rsc
Contributor

rsc commented Oct 11, 2018

Binary-only packages are increasingly hard to support safely. There is no guarantee that the compilation of the binary-only package used the same versions of the dependencies that the final link does (and it would probably be too onerous to insist on that). As a result, the binary-only package may have been compiled using escape analysis results or inline function bodies for dependencies that are no longer accurate. The result can be silent memory corruption.

My memory is that we added binary-only packages originally so that a professor could hand out binary solution sets so that a student who didn't finish lab 2 could still move on to lab 3. I don't know why else anyone uses them anymore, but they're probably going to break more and more as the compiler gets more sophisticated.

Should we just remove them? If not, why not? What do people need them for?

See also #28146 (less severe).

/cc @bcmills

@gopherbot gopherbot added this to the Proposal milestone Oct 11, 2018

@gopherbot gopherbot added the Proposal label Oct 11, 2018

@bradfitz

This comment has been minimized.

Member

bradfitz commented Oct 11, 2018

I'd like to see them removed, mostly for code simplicity.

For the professor use case, perhaps source obfuscation of some form would be sufficient.

@rsc

This comment has been minimized.

Contributor

rsc commented Oct 11, 2018

Looking for people who use binary-only-package and might be able to explain what they use it for.

@ianlancetaylor filed #23473 (but presumably passing along another report).
@triztian filed #26590.
@hilyjiang filed #24318.
@motomux filed #26875.
@joshlf filed #16841.
@jerrybean filed #21451.

/cc @dneil @dsnet for knowledge of any potential Google-internal usage.

@bcmills bcmills added the GoCommand label Oct 11, 2018

@neild

This comment has been minimized.

Contributor

neild commented Oct 11, 2018

I'm not aware of anyone using binary-only packages inside Google. Doesn't mean someone isn't, of course.

@cznic

This comment has been minimized.

Contributor

cznic commented Oct 11, 2018

I'm in favor of the proposal, nonetheless I guess the important property of binary packages is they make some closed source package vendors feel "safe" about their IP.

@booss

This comment has been minimized.

booss commented Oct 11, 2018

I'm in favor of the proposal, nonetheless I guess the important property of binary packages is they make some closed source package vendors feel "safe" about their IP.

That or plugins need to not panic (when loading) and complain about different versions of the same package and also offer unloading support (i.e: find a way to sever the link with the runtime and free evreything).

Vendoring binary packages is useful if you're selling an SDK or toolkit.

@triztian

This comment has been minimized.

triztian commented Oct 14, 2018

Hi there, thank you for considering us!,

In our case we use it as part of our offering to Enterprise customers. Being able to provide non-source distributions and as a way to hide some of our I.P clears us with the IT departments of such companies.

Our biggest use is being able to provide a static library (the BOP package) to third parties without providing our source, we provide an SDK that should work without any connection to our Servers, hence we package the SDK as binary-only-package that includes some proprietary I.P. that is licensed.

While that approach is not the most secure way to protect our source, it is the right balance right now, we just don't include more sensitive functionality in what we distribute.

Another important characteristic is that being able to have a higher level of friction of preventing modification to what we distribute; i.e. even if we provide non-IP that has to function in a specific way, not having BOP's would make easier for third parties to modify the SDK.

I'm not entirely in favor of dropping support for binary-only-packages because I believe that many other languages support a way of distributing packages that don't have the original source in them:

  • Java/Android - Jar/aar files
  • Swift - Frameworks
  • C/C++ - static libs + headers
@ianlancetaylor

This comment has been minimized.

Contributor

ianlancetaylor commented Oct 14, 2018

#23473 was for an internal Google use case (internal reference number 72160357) that was resolved by changing to not use binary packages.

@rsc

This comment has been minimized.

Contributor

rsc commented Oct 18, 2018

Posted to golang-nuts to cast a wider net for use cases.
https://groups.google.com/d/msg/golang-nuts/juPzaRDVB9c/6PYP56-0CAAJ

@wsc1

This comment has been minimized.

wsc1 commented Oct 19, 2018

In general I'm in favour of giving incentive to provide source. But there are legitimate binary-only cases related to not only IP but security.

What relationship would removing support for BOPs this have to plugins and related build modes (shared/c-shared/c-archive)? Could one still provide a binary in any of these forms and be able to use that? If so, then a simple solution for at least some BOP use cases would be to use a wrapper package that loads a plugin, or uses -linkshared or a c-archive via cgo and the plugin or c-archive would be outside of go get/modules and distributed separately as binary.

I have no plans to distribute such things, but would like to be able offer the capacity to others in some form. If something like the above workarounds worked, then that would provide a way to drop BOP
support and say "use workaround ..." for those who want it.

At the same time I understand there are still some issues with these other binary distribution mechanisms and modules.

@rsc

This comment has been minimized.

Contributor

rsc commented Oct 19, 2018

Thanks for asking about plugins. There are no plans to remove plugin support, and suggesting the use of plugins as binary blobs instead of binary-only packages makes sense to me. Plugins have many of the same concerns, but they already have a proper check for mismatched API (avoiding the silent memory corruption problem) and it's always better to have one mechanism than two.

@joshlf

This comment has been minimized.

joshlf commented Oct 19, 2018

Are there any performance concerns for plugins? E.g., I would imagine that you can certain link-time optimizations with binary-only packages that aren't available for dynamically loaded code?

@booss

This comment has been minimized.

booss commented Oct 19, 2018

I agree that binary-only-packages are problematic and should be removed.
I tried to use plugins but you can't unload them.
I decided to live with that and restart the process in case the plugin had to be reloaded but then:
You can't load a plugin if you vendor your dependencies and one of this dependency is shared with the host.
Right now I'm using go tools (compile and link) and reuse the .a but that's kinda ugly

@ianlancetaylor

This comment has been minimized.

Contributor

ianlancetaylor commented Oct 19, 2018

@joshlf It's hard for me to imagine any real case in which plugins are noticeably less efficient than binary-only packages. It's possible to construct an artificial case, but I doubt that would match any real implementation.

@wsc1

This comment has been minimized.

wsc1 commented Oct 19, 2018

Thanks for asking about plugins. There are no plans to remove plugin support, and suggesting the use of plugins as binary blobs instead of binary-only packages makes sense to me. Plugins have many of the same concerns, but they already have a proper check for mismatched API (avoiding the silent memory corruption problem) and it's always better to have one mechanism than two.

Great to hear. It seems to me it wouldn't be too much to adapt the source stubs from a BOP to wrap a plugin and an init() to load it from somewhere, and well worth it to both BOP users and Go maintenance.

@rsc

This comment has been minimized.

Contributor

rsc commented Oct 24, 2018

Will leave this open for another week but it sounds like we can retire binary-only packages in favor of using plugins. We could advertise in the Go 1.12 release notes that it will be the last release to support binary-only packages and they will be removed in Go 1.13.

@andybons

This comment has been minimized.

Member

andybons commented Oct 31, 2018

Accepted since we waited a week and nothing came up/no objections were raised.

Per discussion with @golang/proposal-review

@andybons andybons changed the title from proposal: cmd/go: drop support for binary-only packages to cmd/go: drop support for binary-only packages Oct 31, 2018

@edburns

This comment has been minimized.

edburns commented Nov 13, 2018

As a corporate programmer, the closed-source use case is very important. A language that is so RMS idealism friendly that it doesn't even allow one to use closed-source dependencies is a very hard sell in many corporate environments.

@mvdan

This comment has been minimized.

Member

mvdan commented Nov 13, 2018

@edburns see the discussion above; using compiled binaries will still be possible via plugins. binary-only packages are harder to develop, maintain, and use, as can be seen above.

The later stages of this thread were about finding any reason why plugins aren't a good replacement for binary-only packages, to which there wasn't a clear answer.

@ianlancetaylor

This comment has been minimized.

Contributor

ianlancetaylor commented Nov 13, 2018

@edburns Thanks for the comment. Do you have direct experience of this? How do those corporate environments handle languages like Python or Javascript?

@edburns

This comment has been minimized.

edburns commented Nov 14, 2018

Consider the case when a corporation is using a licensed closed-source software package as a dependency on an internal project. Right now, if the vendor of a closed-source package wanted to make their licensed package useful to go programmers, their only option is to author it as a plugin. As stated elsewhere, this has drawbacks: complexity of .so linking, lack of non Linux support, etc.

@mvdan

This comment has been minimized.

Member

mvdan commented Nov 14, 2018

make their licensed package useful to go programmers

I'd argue that using a binary-only package was never really an option. The binary would require an exact version of Go, as well as an exact version of all the packages it depends on. It's certainly possible, but I wouldn't say it's ever been a good or easy way to distribute software. The intended purpose, the professor scenario, is a much more restricted use case.

complexity of .so linking

Please expand on this. If you believe there's a bug or something to improve in the plugin package or the go tool, you should open a separate issue.

lack of non Linux support

What particular platform is noticeably missing plugin support? Assuming you mean Windows, that's being tracked in #19282. I believe effort would be better invested in issues like that one, instead of on maintaining binary-only package support.

@ianlancetaylor

This comment has been minimized.

Contributor

ianlancetaylor commented Nov 14, 2018

@edburns Thanks. Personally I am more interested in direct experience than in hypotheticals. We can construct a number of hypothetical situations, but in the end they won't tell us what to do.

@oszika

This comment has been minimized.

oszika commented Nov 15, 2018

It is possible to use vendoring with plugins (without set package import path)? The use of multiple plugins is hard because they need to use exactly the same version of common dependencies. Would it be possible to use the latest minor version for each dependency otherwise?
Edit: the last point seems to be impossible to do because we cannot unload a plugin.

@twyvip

This comment has been minimized.

twyvip commented Nov 28, 2018

  1. I have one file in /root/work/go/src/say named say.go
    package say
    import "fmt"
    func Say(s string) string {
    return fmt.Sprintf("hello %s", s)
    }

  2. I try to build it to .a file go build -i -o $GOPATH/pkg/linux_amd64/say.a

  3. change /root/work/go/src/say/say.go to

//go:binary-only-package

package say

  1. another file /root/work/go/src/say/work/v1.go
    package v1
    import (
    "fmt"
    "say"
    )
    func SayToBob() {
    fmt.Println(say.Say("Bob"))
    }

  2. the main file /root/work/go/src/say/work/main.go
    package main
    import (
    "say/work/v1"
    )
    func main() {
    v1.SayToBob()
    }

  3. run main.go go run /root/work/go/src/say/work/main.go
    go run main.go
    hello Bob
    that's right

7.but in fact, i need to build SayToBob as plugin to hot reload,run
go build -buildmode=plugin -o /root/work/go/src/say/work/v1/plugin.so /root/work/go/src/say/work/v1/v1.go
missing or invalid package binary for binary-only package say

@bcmills

This comment has been minimized.

Member

bcmills commented Nov 28, 2018

@twyvip, how does your comment relate to this proposal?

@massimiliano-mantione

This comment has been minimized.

massimiliano-mantione commented Dec 6, 2018

FWIW, I have a "convenience" use case for binary-only packages: cross compiled cgo packages that are inconvenient to compile on the host because everybody would have to install the relevant C cross compiler (for the target).

Concrete case: I am starting to write a go wrapper for the librobotcontrol C library for the beaglebone blue.

That C library makes no sense on the host (be it windows, mac or linux).
AFAIK to cross compile the wrapper I'd need to copy the relevant header files and libraries on the host, and also provide cgo a C cross compiler.

I was strongly considering building the wrapper natively on the beaglebone (very easy to do), and just redistribute the compiled object file so that users do not need to set up the cgo cross compilation environment.

In this case "users" would be a group of 15 years old highschool students I am mentoring for a school-related robotics project.

Anyway I can probably do this cgo wrapper as a plugin, with a bit of boilerplate code to export a type safe interface to the wrapped functions.

Do you still think to deprecate binary-only packages?
If yes I'll go the plugin way, since that would be the only supported option in the future.

@bcmills

This comment has been minimized.

Member

bcmills commented Dec 6, 2018

@massimiliano-mantione, a plugin sounds like the right approach for your use-case.

@gopherbot

This comment has been minimized.

gopherbot commented Dec 6, 2018

Change https://golang.org/cl/152918 mentions this issue: doc: announce the end of support for binary-only packages

@ianlancetaylor

This comment has been minimized.

Contributor

ianlancetaylor commented Dec 6, 2018

@massimiliano-mantione Another approach you can use is .syso files. Compile all the C code into a single object file per support platform, and name it something like c_GOOS_GOARCH.syso. Then the Go code can be the same as always, and can use cgo to call into the C code provided as a .syso file.

gopherbot pushed a commit that referenced this issue Dec 7, 2018

doc: announce the end of support for binary-only packages
Updates #28152

Change-Id: If859221afc683b392f649e79d7ff0a06125cbe10
Reviewed-on: https://go-review.googlesource.com/c/152918
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment