Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: drop support for binary-only packages #28152

Closed
rsc opened this Issue Oct 11, 2018 · 72 comments

Comments

@rsc
Copy link
Contributor

rsc commented Oct 11, 2018

Binary-only packages are increasingly hard to support safely. There is no guarantee that the compilation of the binary-only package used the same versions of the dependencies that the final link does (and it would probably be too onerous to insist on that). As a result, the binary-only package may have been compiled using escape analysis results or inline function bodies for dependencies that are no longer accurate. The result can be silent memory corruption.

My memory is that we added binary-only packages originally so that a professor could hand out binary solution sets so that a student who didn't finish lab 2 could still move on to lab 3. I don't know why else anyone uses them anymore, but they're probably going to break more and more as the compiler gets more sophisticated.

Should we just remove them? If not, why not? What do people need them for?

See also #28146 (less severe).

/cc @bcmills

@gopherbot gopherbot added this to the Proposal milestone Oct 11, 2018

@gopherbot gopherbot added the Proposal label Oct 11, 2018

@bradfitz

This comment has been minimized.

Copy link
Member

bradfitz commented Oct 11, 2018

I'd like to see them removed, mostly for code simplicity.

For the professor use case, perhaps source obfuscation of some form would be sufficient.

@rsc

This comment has been minimized.

Copy link
Contributor Author

rsc commented Oct 11, 2018

Looking for people who use binary-only-package and might be able to explain what they use it for.

@ianlancetaylor filed #23473 (but presumably passing along another report).
@triztian filed #26590.
@hilyjiang filed #24318.
@motomux filed #26875.
@joshlf filed #16841.
@jerrybean filed #21451.

/cc @dneil @dsnet for knowledge of any potential Google-internal usage.

@bcmills bcmills added the GoCommand label Oct 11, 2018

@neild

This comment has been minimized.

Copy link
Contributor

neild commented Oct 11, 2018

I'm not aware of anyone using binary-only packages inside Google. Doesn't mean someone isn't, of course.

@cznic

This comment has been minimized.

Copy link
Contributor

cznic commented Oct 11, 2018

I'm in favor of the proposal, nonetheless I guess the important property of binary packages is they make some closed source package vendors feel "safe" about their IP.

@booss

This comment has been minimized.

Copy link

booss commented Oct 11, 2018

I'm in favor of the proposal, nonetheless I guess the important property of binary packages is they make some closed source package vendors feel "safe" about their IP.

That or plugins need to not panic (when loading) and complain about different versions of the same package and also offer unloading support (i.e: find a way to sever the link with the runtime and free evreything).

Vendoring binary packages is useful if you're selling an SDK or toolkit.

@triztian

This comment has been minimized.

Copy link

triztian commented Oct 14, 2018

Hi there, thank you for considering us!,

In our case we use it as part of our offering to Enterprise customers. Being able to provide non-source distributions and as a way to hide some of our I.P clears us with the IT departments of such companies.

Our biggest use is being able to provide a static library (the BOP package) to third parties without providing our source, we provide an SDK that should work without any connection to our Servers, hence we package the SDK as binary-only-package that includes some proprietary I.P. that is licensed.

While that approach is not the most secure way to protect our source, it is the right balance right now, we just don't include more sensitive functionality in what we distribute.

Another important characteristic is that being able to have a higher level of friction of preventing modification to what we distribute; i.e. even if we provide non-IP that has to function in a specific way, not having BOP's would make easier for third parties to modify the SDK.

I'm not entirely in favor of dropping support for binary-only-packages because I believe that many other languages support a way of distributing packages that don't have the original source in them:

  • Java/Android - Jar/aar files
  • Swift - Frameworks
  • C/C++ - static libs + headers
@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

ianlancetaylor commented Oct 14, 2018

#23473 was for an internal Google use case (internal reference number 72160357) that was resolved by changing to not use binary packages.

@rsc

This comment has been minimized.

Copy link
Contributor Author

rsc commented Oct 18, 2018

Posted to golang-nuts to cast a wider net for use cases.
https://groups.google.com/d/msg/golang-nuts/juPzaRDVB9c/6PYP56-0CAAJ

@wsc1

This comment has been minimized.

Copy link

wsc1 commented Oct 19, 2018

In general I'm in favour of giving incentive to provide source. But there are legitimate binary-only cases related to not only IP but security.

What relationship would removing support for BOPs this have to plugins and related build modes (shared/c-shared/c-archive)? Could one still provide a binary in any of these forms and be able to use that? If so, then a simple solution for at least some BOP use cases would be to use a wrapper package that loads a plugin, or uses -linkshared or a c-archive via cgo and the plugin or c-archive would be outside of go get/modules and distributed separately as binary.

I have no plans to distribute such things, but would like to be able offer the capacity to others in some form. If something like the above workarounds worked, then that would provide a way to drop BOP
support and say "use workaround ..." for those who want it.

At the same time I understand there are still some issues with these other binary distribution mechanisms and modules.

@rsc

This comment has been minimized.

Copy link
Contributor Author

rsc commented Oct 19, 2018

Thanks for asking about plugins. There are no plans to remove plugin support, and suggesting the use of plugins as binary blobs instead of binary-only packages makes sense to me. Plugins have many of the same concerns, but they already have a proper check for mismatched API (avoiding the silent memory corruption problem) and it's always better to have one mechanism than two.

@joshlf

This comment has been minimized.

Copy link

joshlf commented Oct 19, 2018

Are there any performance concerns for plugins? E.g., I would imagine that you can certain link-time optimizations with binary-only packages that aren't available for dynamically loaded code?

@booss

This comment has been minimized.

Copy link

booss commented Oct 19, 2018

I agree that binary-only-packages are problematic and should be removed.
I tried to use plugins but you can't unload them.
I decided to live with that and restart the process in case the plugin had to be reloaded but then:
You can't load a plugin if you vendor your dependencies and one of this dependency is shared with the host.
Right now I'm using go tools (compile and link) and reuse the .a but that's kinda ugly

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

ianlancetaylor commented Oct 19, 2018

@joshlf It's hard for me to imagine any real case in which plugins are noticeably less efficient than binary-only packages. It's possible to construct an artificial case, but I doubt that would match any real implementation.

@wsc1

This comment has been minimized.

Copy link

wsc1 commented Oct 19, 2018

Thanks for asking about plugins. There are no plans to remove plugin support, and suggesting the use of plugins as binary blobs instead of binary-only packages makes sense to me. Plugins have many of the same concerns, but they already have a proper check for mismatched API (avoiding the silent memory corruption problem) and it's always better to have one mechanism than two.

Great to hear. It seems to me it wouldn't be too much to adapt the source stubs from a BOP to wrap a plugin and an init() to load it from somewhere, and well worth it to both BOP users and Go maintenance.

@rsc

This comment has been minimized.

Copy link
Contributor Author

rsc commented Oct 24, 2018

Will leave this open for another week but it sounds like we can retire binary-only packages in favor of using plugins. We could advertise in the Go 1.12 release notes that it will be the last release to support binary-only packages and they will be removed in Go 1.13.

@andybons

This comment has been minimized.

Copy link
Member

andybons commented Oct 31, 2018

Accepted since we waited a week and nothing came up/no objections were raised.

Per discussion with @golang/proposal-review

@andybons andybons changed the title proposal: cmd/go: drop support for binary-only packages cmd/go: drop support for binary-only packages Oct 31, 2018

@edburns

This comment has been minimized.

Copy link

edburns commented Nov 13, 2018

As a corporate programmer, the closed-source use case is very important. A language that is so RMS idealism friendly that it doesn't even allow one to use closed-source dependencies is a very hard sell in many corporate environments.

@mvdan

This comment has been minimized.

Copy link
Member

mvdan commented Nov 13, 2018

@edburns see the discussion above; using compiled binaries will still be possible via plugins. binary-only packages are harder to develop, maintain, and use, as can be seen above.

The later stages of this thread were about finding any reason why plugins aren't a good replacement for binary-only packages, to which there wasn't a clear answer.

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

ianlancetaylor commented Nov 13, 2018

@edburns Thanks for the comment. Do you have direct experience of this? How do those corporate environments handle languages like Python or Javascript?

@edburns

This comment has been minimized.

Copy link

edburns commented Nov 14, 2018

Consider the case when a corporation is using a licensed closed-source software package as a dependency on an internal project. Right now, if the vendor of a closed-source package wanted to make their licensed package useful to go programmers, their only option is to author it as a plugin. As stated elsewhere, this has drawbacks: complexity of .so linking, lack of non Linux support, etc.

@mvdan

This comment has been minimized.

Copy link
Member

mvdan commented Nov 14, 2018

make their licensed package useful to go programmers

I'd argue that using a binary-only package was never really an option. The binary would require an exact version of Go, as well as an exact version of all the packages it depends on. It's certainly possible, but I wouldn't say it's ever been a good or easy way to distribute software. The intended purpose, the professor scenario, is a much more restricted use case.

complexity of .so linking

Please expand on this. If you believe there's a bug or something to improve in the plugin package or the go tool, you should open a separate issue.

lack of non Linux support

What particular platform is noticeably missing plugin support? Assuming you mean Windows, that's being tracked in #19282. I believe effort would be better invested in issues like that one, instead of on maintaining binary-only package support.

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

ianlancetaylor commented Nov 14, 2018

@edburns Thanks. Personally I am more interested in direct experience than in hypotheticals. We can construct a number of hypothetical situations, but in the end they won't tell us what to do.

@oszika

This comment has been minimized.

Copy link

oszika commented Nov 15, 2018

It is possible to use vendoring with plugins (without set package import path)? The use of multiple plugins is hard because they need to use exactly the same version of common dependencies. Would it be possible to use the latest minor version for each dependency otherwise?
Edit: the last point seems to be impossible to do because we cannot unload a plugin.

@fabian-f

This comment has been minimized.

Copy link

fabian-f commented Feb 27, 2019

I'm in a similar, albeit of lower magnitude, situation to @cpipero.

@ianlancetaylor I totally get the need to burn a bad code base to the ground, but there needs to be a suitable drop-in (or close to) replacement that A) has the same functionality and B) works in a similar way.

Whilst plugins provide, on an abstract level, similar functionality, they are lacking completely when it comes to type definitions. It's just not there at all. Then there's the objectively horrible syntax which is just begging to be generated (which is a possible but not ideal workaround).

@triztian

This comment has been minimized.

Copy link

triztian commented Feb 27, 2019

@ianlancetaylor would the "plugin" package eventually be able to at least support loading structs declared in the plugin?.

I think that would further make plugins a replacement for binary-only-packages.

@cpipero

This comment has been minimized.

Copy link

cpipero commented Feb 27, 2019

We looked at plugin and it's not so practical for our use.
You need to call plugin.Lookup("FooType") and then cast the result to FooType to be able to construct a type. We've got a lot of types (we are selling an SDK).
Again, it's all in the economics: this issue is costing us a lot...

@yslim

This comment has been minimized.

Copy link

yslim commented Feb 27, 2019

Today, I studied plugin.
I think plugin concept is very similar to swig.
All sharing types should be treated as interface.

func main() {
shared.LoadGreetingPlugin("plugin/hello.so")

greeter1 := shared.NewGreeting("test1")
greeter2 := shared.NewGreeting("test2")
greeter2.SetName("test3")

greeter1.SayHello()
greeter2.SayHello()
}

https://github.com/yslim/go_plugin_test

In my case, plugin will be a good option for replacing binary-only-package.

@aaveidt

This comment has been minimized.

Copy link

aaveidt commented Feb 27, 2019

In my case, plugin will be a good option for replacing binary-only-package.

@yslim Have you used binary-only package? Its 100 times shorter.
https://github.com/yslim/go_plugin_test/blob/master/shared/greeting.go

Your main.go and shared/greating.go files have 45-50 lines of code to call only 1 SayHello() method.
Using binary-only package only costs 4

package main
import hello
greeter := hello.NewGreeting("test1")
greeter.SayHello()

Then i can call any exported methods effortlessly without writing any line of code: greeter.A() , greeter.B() ...
What would you do with plugin, you have to redeclare each of them in your Greeting interface.

One framework or big package can have hundred types, and exported methods.
Try to port any package like gin, chi, gorm, beego to plugins to see the difference.
Using plugin isn't practical at all.

@gen0cide

This comment has been minimized.

Copy link

gen0cide commented Feb 27, 2019

I completely disagree with your position @aaveidt and @fabian-f for a number of reasons.

You're continuing to assert that binary-only packages are the only option because they allow your users easy access to code that you write, but that you will not share with them. Where I struggle to sympathize is in that you're doing this exclusively for Go packages. This isn't for c-shared libraries that have symbols exported and documented. You're asking Go developers to implement your binaries under the following conditions:

  1. No access to the underlying source code. Hopefully you generated some kind of a Godoc, but even then, I'd find myself frustrated as Go developer giving you money to use your code. Not a day goes by that I don't click through an Anchor into code in Godoc. And I think that's a paradigm that has made Go successful - it has the best documentation story of any language.

  2. Very poor compatibility. As has been noted by many issues here, I'd be extremely upset as a developer if my vendor continues to provide me libraries that suffer from inconsistent, buggy behavior.

What I'm getting at is that while it is 100% your choice to decide that theoretical intellectual property protection is more important than your developers experience, I do not understand how somehow removing BOPs when Go has provided you with a much more stable Plugin system[1].

Your biggest complaints about the Plugin system relate to "effort". It definitely will require some work to evolve your codebases to comply with the new supported standards, but repeatedly arguments have been made about how this would be an undue burden on your users. Ultimately, your users, who are Go developers, are burdened by your choices, not the Go ecosystem's. If you valued their experience above all else, writing some go/ast, go/parser and go/printer code to programmatically generate intermediate plugin interfaces that you could share with users doesn't seem like too much to ask.

Especially when the result will be more stability, better documentation and usability, and easier forward compatibility. I support @ianlancetaylor and @rsc on the job done. As much as they value API stability and compatibility, this is the type of deadweight that drives poor developer experiences in the ecosystem. They recognized it and it's time for it to deprecate into the past.

[1] I realize that right now that Go doesn't have Plugin support for Windows. But so far, a significant amount of push back has been from people talking about "web frameworks" which overwhelmingly are developed and deployed on Linux.

@cpipero

This comment has been minimized.

Copy link

cpipero commented Feb 27, 2019

@gen0cide thank you for the valid points, which speak to ultimate stability and developer well-being.
Our problem is more of a practical nature, where a known feature was the base for us to lay out a legal and commercial framework for the distribution of a product.

That framework ceased to exist once no equivalent feature could be provided by us to our customer.

Again, I did not pay for golang development so I am not going to blame the ultimate decision. I just voiced the reason why I would vote for keeping a feature that works for us without any issue, understanding the risks but not having a whole lot of choices given the significant investment.

You say that writing a go/ast, go/parser, etc. is no big ask. In our case, we would probably refactor our entire API into a language that 1) supports the legal agreements in place (for us and our users), and 2) eliminates the risk of similar decisions by enforcing a more formal ISO/IEC process to make changes.

And in doing that (or if I were to follow your advice) it means to come up with a lot of money (of course, that subjective but for me anything more than 20K is a big ask).

Therefore, the point here is how to deal with existing SLA's.

With all that said, all I can do was to relate a real world scenario.

Thanks again for the advice.

@fabian-f

This comment has been minimized.

Copy link

fabian-f commented Feb 27, 2019

@gen0cide Writing some AST magic to generate a proxy layer is no big ask. I've already started it in anticipation of this change. The problem is plugins aren't ready to act as a replacement yet. They're only supported on Linux and Mac (3 of my larger clients have to deploy to Windows), and don't support struct type definitions. I don't get why a handful of people are so eager to deprecate this feature without a single full replacement.

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

ianlancetaylor commented Feb 27, 2019

@fabian-f I want to express again that, as we've said, the reason to deprecate binary-only packages is that they are not reliable. If accidentally misused, they can lead to silent memory corruption during execution. That is both unfortunate and difficult to fix.

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

ianlancetaylor commented Feb 27, 2019

Let me add that I do understand that you have a use case for binary packages. Months ago @rsc asked for people to give examples of use cases that were not supported by plugins. Only in the last few days have they come forward. I understand that not everybody can track the Go mailing lists. Now that a couple of use cases have come forward, we will re-evaluate the decision and see if we can find a way to move forward. If anybody wants to address the technical issues that make binary packages difficult to support, that would be very helpful.

@cpipero

This comment has been minimized.

Copy link

cpipero commented Feb 27, 2019

@rsc

This comment has been minimized.

Copy link
Contributor Author

rsc commented Feb 27, 2019

@cpipero and @aaveidt, from a "code hiding" point of view, the compiled .a file has a lot of source information embedded in it. A LOT. It might be equally practically effective just to run a code obfuscator on your code and distribute source. Perhaps much better, since it would hide more of the internal names.

If such a code obfuscator tool existed, would that work as a replacement for binary-only packages for you?

@mirtchovski

This comment has been minimized.

Copy link
Contributor

mirtchovski commented Feb 27, 2019

If such a code obfuscator tool existed, would that work as a replacement for binary-only packages for you?

I don't know about the current cases, but the malware community would be very happy with such a tool to make reverse engineers' tasks slightly more difficult. cf. https://blog.malwarebytes.com/threat-analysis/2019/02/new-golang-brute-forcer-discovered-amid-rise-e-commerce-attacks/

@cpipero

This comment has been minimized.

Copy link

cpipero commented Feb 27, 2019

@bradfitz

This comment has been minimized.

Copy link
Member

bradfitz commented Feb 27, 2019

@cpipero, you might need to help explain things to your legal team, then. You can't expect them to know how all (or any) compilers work.

@gen0cide

This comment has been minimized.

Copy link

gen0cide commented Feb 27, 2019

@rsc, I'm supportive of exploration of Go obfuscation in general, but curious if it isn't trending a more broadly applicable solution near @fabian-f and @cpipero's comments regarding auto-generation of plugin scaffolding.

I spent years reverse engineering obfuscators for MMORPGs who used a wide variety of Java obfuscators, open source and commercial alike. Bytecode is easily represented in source - so in effect, even the best Java obfuscator is basically what you're proposing - a source obfuscator. Reversing these obfuscation techniques were common practice by myself and other teenagers. Also, at the end of the day, much of the obfuscation techniques that work well will diminish the optimization strategies of the compiler, as well as negatively effect performance of the compiled binary. This is now how most enterprise mobile obfuscators work today.

Nowadays, I work in information security and actively research and develop techniques of binary obfuscation using Go. I've spent a lot o time leveraging the power of go/ast and go/parser to this end. I love the theory of a richly featured obfuscation utility, but having gone down that road, I would anticipate supporting such a utility would be more cost than the benefits in this particular issue. Basically, this boils down to the fact that to really check the boxes for anti-introspection, anti-tampering, and preventing reverse engineering (especially in a situation where you're distributing obfuscated source), you'd need to write rules governing the re-implementation of every ast.Node in the tree. Note that to be effective, you'd need a wide library of these rules to look at not just individual nodes, but parent nodes and reference calling. Then, the obfuscator must randomly (or keyed with an secret IV) select strategies at every level, wrapping, rinsing, and repeating. This "code mixing" is all for not if the reverse engineer can simply document your re-write rules to perform the inverse on identified nodes.

I still believe the stability of the language ecosystem and compiler requires BOPs to be deprecated, but I do think there is room to innovate something on behalf of those who have use cases. How do you feel @rsc and @ianlancetaylor about the concepts of some sort of plugin scaffolding generator? I don't know where this lives, but I see the feature working something like this:

  1. DeveloperA develops a plugin using currently recommended practices.
  2. DeveloperA marks through a comment //plugin:export the types and functions they wish to expose.
  3. DeveloperA runs builds the plugin but with an additional -export flag. The compiler produces 2 files - plugin.so (the compiled plugin), and plugin_export.go.

What happened? The compiler walked the AST looking for plugin exports, then generated a scaffolded export that implements those exported types and function calls using a standardized entrypoint. This go code, along with the plugin, could then be distributed to developers.

Thoughts?

[edits for grammer/spelling]

@cpipero

This comment has been minimized.

Copy link

cpipero commented Feb 27, 2019

@gen0cide

This comment has been minimized.

Copy link

gen0cide commented Feb 27, 2019

If such a code obfuscator tool existed, would that work as a replacement for binary-only packages for you?

I don't know about the current cases, but the malware community would be very happy with such a tool to make reverse engineers' tasks slightly more difficult. cf. https://blog.malwarebytes.com/threat-analysis/2019/02/new-golang-brute-forcer-discovered-amid-rise-e-commerce-attacks/

@mirtchovski, I would encourage you to try and separate the deficiencies of the security industry from the needs of software engineering as a whole. The information security industry is a billion dollar industry that by in large, is filled with vaporware. The cognitive dissidence is on full display when technology is discouraged due to "what it will mean in the hands of hackers". This same mindset fueled anti-encryption sentiment in the 80s and 90s, and continues today around the need to invade user privacy more and more.

Hackers do not need this hypothetical obfuscator to lay waste to even the most sophisticated anti-malware systems. Those system's are ineffective, not because the hackers have magic tools, but because they are attempting to solve a non-sequiter. They have to be flexible enough to not cause false positive detection/prevention, but also detect nefarious behavior. Since the possible permutations of this is different not just from company to company, but from employee to employee, and even day to day per employee, heuristics based detection will never be the primary defensive strategy.

Good security is about empowering the user to be inherently secure. Go does a great job of this in their x/crypto/ssh package, as well as attempting to make certificate validation default in net/http.

Sorry for the long position, but as a security professional and an avid Go user, I've got an interest in not letting security theater discourage innovation.

[edit to add the forgotten mention to reply]

@fabian-f

This comment has been minimized.

Copy link

fabian-f commented Feb 27, 2019

@gen0cide I believe this code generator idea to be a good workaround, and was my plan for when this change goes ahead anyway. It being part of Go's core tools would definitely soften the impact of losing BOPs. However, plugins need to work on all main platforms before they can ever be considered an alternative.

Also aside from trimpath and -s -w is there any more complete ways to remove source information from the compiled binary?

@mvdan

This comment has been minimized.

Copy link
Member

mvdan commented Feb 27, 2019

I think we should focus on the technical arguments against adopting plugins. So far, it seems like the only missing critical features in plugins are Windows support and the ability to expose type definitions. Is that correct?

If so, perhaps we should move the discussion towards looking at those issues, and perhaps even blocking this proposal until they've been considered.

I don't think verbosity is nearly as important as the other issues, nor do I think it's a stopgap. Having to write an extra 50 lines is certainly not ideal, but there are many factors at play here. Perhaps Go2 type parameters would allow getting rid of some of the reflect boilerplate.

@fabian-f

This comment has been minimized.

Copy link

fabian-f commented Feb 27, 2019

@mvdan I agree. Plugins combined with a command line tool could replace BOPs, if plugins have the OS limitation fixed, support type definitions, and provide an option to export constant values (although the command line tool could simply copy constant definitions to the shared code).

All of the boilerplate code is so generic and repetitive that it could be generated.

@rsc

This comment has been minimized.

Copy link
Contributor Author

rsc commented Feb 28, 2019

I'm not terribly worried about the "malware community" one way or another. In the page you linked, the binary hadn't even been stripped.

@rsc

This comment has been minimized.

Copy link
Contributor Author

rsc commented Feb 28, 2019

The fact is binary-only packages do not work with Go 1.12 except in limited circumstances. (And I understand that some people are working within those limited circumstances successfully, if luckily.) In Go 1.13 they won't work at all. The limited cases that do work in Go 1.12 will continue to work in Go 1.12 - we aren't going to retroactively remove them from Go 1.12. People who need extra time to move to what's next can keep using Go 1.12 as long as they like, of course.

It seems like there are two paths forward. First, people could work on an obfuscator of sorts. Second, people could work on improving plugin support.

@fabian-f

This comment has been minimized.

Copy link

fabian-f commented Feb 28, 2019

@rsc Why not wait until plugins are at a sensible point before deprecating a solution that does work?

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

ianlancetaylor commented Mar 5, 2019

@fabian-f The unfortunate fact is that it doesn't work today, as described earlier. And it's going to break even worse with future changes such as module support.

@gopherbot

This comment has been minimized.

Copy link

gopherbot commented Mar 6, 2019

Change https://golang.org/cl/165746 mentions this issue: cmd/go: drop support for binary-only packages

@gopherbot gopherbot closed this in 1650f1b Mar 6, 2019

@fabian-f

This comment has been minimized.

Copy link

fabian-f commented Mar 6, 2019

@ianlancetaylor But you've heard from 3 different people who have clients who successfully use them in a production environment, so it does work, today.

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

ianlancetaylor commented Mar 6, 2019

@fabian-f There are several options that the client could pass to go build or go install that will break the build with binary-only packages. So, yes, it works in limited circumstances.

@mikeschinkel

This comment has been minimized.

Copy link

mikeschinkel commented Mar 8, 2019

Reading through all of these comments, it seems the question has evolved to "Can plugins replace binary-only packages?" and "What is needed to make binary-only packages viable?" Given that, I am hoping my comments are not off-topic, but if they are please forgive me and feel free to mark this comment as off-topic like one of the ones above.

We are not actually using binary-only packages but very recently explored using plugins to allow a 3rd party extension mechanism for a product we are building that would be marketed to people who are not-Go developers. IOW, we are looking for an extensibility mechanism that would allow 3rd parties to build and deliver binary "add-ons" for our product to our mutual customers. Unfortunately we found plugins lacking, for two reasons:

  1. They do not work for Windows (although I do understand that is supposed to be resolved.)

  2. Plugins evidently require being built by the same version of the compiler that the main application is built using, and that unfortunately is a non-starter given we won't always be coordinating with 3rd party developers nor would we want to have to.

So, I understand that plugins possibly — by design — will always require version parity between the main app and the plugin(s). But if that is the case then I think Go really needs a standardized extensibility mechanism that allows Go apps to load 3rd party extensions that were compiled with any compiler, maybe with communication over named pipes — can you do (something like) that in Windows? — or via protobuf over TCP/IP?

We can, of course, develop our own, but I think this really should be a widely applicable use-case — imaging 3rd party extension for Docker? — and I doubt our team and many others would have the expertise and experience to envision all the edge cases that might be found by users, so I am asking that the Go team consider creating a "blessed" standard package for extensibility that does not have the limitations that I understand plugins have.

That said, if I am misunderstanding anything about how plugins work, please forgive me for that as well. I have only been actively coding in Go for less than 6 months.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.