Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: drop runtime from plugins #17150

Open
rasky opened this issue Sep 18, 2016 · 11 comments

Comments

@rasky
Copy link
Member

commented Sep 18, 2016

It looks like plugins contain a copy of the runtime. I guess that copy is unused, and could probably be dropped, reducing binary size.

@rasky

This comment has been minimized.

Copy link
Member Author

commented Sep 18, 2016

/cc @crawshaw

@crawshaw

This comment has been minimized.

Copy link
Contributor

commented Sep 18, 2016

Yup, I had a TODO about this I lost somewhere in the CL patchsets.

It should be straightforward, it's already possible to build -linkshared binaries without a runtime. I expect most of the work is careful argument construction in cmd/go.

@crawshaw crawshaw changed the title Drop runtime from plugins cmd/go: drop runtime from plugins Sep 18, 2016

@quentinmit quentinmit added the NeedsFix label Oct 3, 2016

@quentinmit quentinmit added this to the Go1.8 milestone Oct 3, 2016

@rsc

This comment has been minimized.

Copy link
Contributor

commented Oct 21, 2016

@crawshaw, still want to do this for Go 1.8?

@crawshaw

This comment has been minimized.

Copy link
Contributor

commented Oct 21, 2016

Pushing to 1.9, so I can spend what cycles I have on bugs.

@gopherbot

This comment has been minimized.

Copy link

commented Sep 11, 2017

Change https://golang.org/cl/62352 mentions this issue: cmd/link: correct symbol table order for plugin-use code on darwin/amd64

@crawshaw

This comment has been minimized.

Copy link
Contributor

commented Sep 23, 2017

@hirochachacha at one point you had a CL exploring removing the runtime from plugins. Are you still working on it?

@hirochachacha

This comment has been minimized.

Copy link
Contributor

commented Sep 23, 2017

@crawshaw No, I'm afraid not.

@rsc rsc modified the milestones: Go1.10, Unplanned Dec 1, 2017

@montao

This comment has been minimized.

Copy link

commented Feb 4, 2019

I think I have this issue too in my code https://gitlab.com/montao/forbasile

The background is that we want to generate a large number of plugins, and statically linked objects would take too much space. If the runtime is removed from the plugin then I think that the problem would be solved and we could use Go. Right now we are looking at other choices of implementation (C, Rust, custom...) instead.

@bstarynk

This comment has been minimized.

Copy link

commented Feb 7, 2019

For what it is worth, my manydl.c is generating "randomly" the code of many plugins in C and compiling it and loading these many plugins. That program demonstrates that on Linux, one can have hundreds of thousands of dlopen-ed plugins. I see no reason why that would not be possible in Go (and I guess that writing the equivalent manydl.go program is reasonably easy)
And generating plugins at runtime is sometimes interesting. My old GCC MELT project did that (generating lots of plugins in C++). My current bismon project is doing that (generating lots of plugins in C), and it would be nice to be able to start a project generating lots of plugins in Go (which has, as a language for generated code, some advantages over C and C++)

So I believe it would be great if Go would practically support many plugins (in practice generated ones). The current implementation does that poorly (it is designed for a few, hand-written, plugins in mind).
In its current implementation, each generated *.so would have its own copy of common standard functions (e.g. fmt.Printf), so if we generate thousands of plugins, the binary code of fmt.Printf gets duplicated a thousand times in the virtual address space. That does not scale.

@montao

This comment has been minimized.

Copy link

commented Feb 10, 2019

@hirochachacha at one point you had a CL exploring removing the runtime from plugins. Are you still working on it?

I found this CL, I think: https://go-review.googlesource.com/c/go/+/61090 Not sure what is the problem with it but it was abandoned. Any clues?

@hirochachacha

This comment has been minimized.

Copy link
Contributor

commented Feb 11, 2019

I don't remember exactly, but I couldn't recommend investing that approach since that is a half-baked solution after all. It only cover runtime package, it doesn't cover other standard libraries. For example, if there are two plugin A and B that use strings package, both plugins still need to bundle strings package. Ideally, this issue should be solved by re-implementing plugin system on top of buildmode=shared. I guess that it requires hard work though.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
9 participants
You can’t perform that action at this time.