plugin: using stdlib package plugin from c-archive package doesn't work as expected

[typetetris]

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go version devel +5d1b53a Wed Nov 30 19:46:00 2016 +0000 linux/amd64

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/erwo/golang-plugin"
GORACE=""
GOROOT="/home/erwo/go-devel"
GOTOOLDIR="/home/erwo/go-devel/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build451133793=/tmp/go-build -gno-record-gcc-switches"
CXX="g++"
CGO_ENABLED="1"
PKG_CONFIG="pkg-config"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"

What did you do?

Wrote a main package to be used as c-archive within a c host app, which uses stdlib package plugin to load other go plugins.

https://github.com/erwo42/golang-plugin

Just run from within a workdir of above repo

go build -buildmode=plugin testplugin
go build -buildmode=c-archive indirect_with_c
gcc -o indirect_with_c c_host_app.c indirect_with_c.a -ldl -lpthread
./indirect_with_c

What did you expect to see?

Yay!

What did you see instead?

fatal error: runtime: no plugin module data

goroutine 17 [running, locked to thread]:
runtime.throw(0x4a9ba3, 0x1e)
        /home/erwo/go-devel/src/runtime/panic.go:596 +0x97 fp=0xc42003ca70 sp=0xc42003ca50
plugin.lastmoduleinit(0xebe420, 0xc42000e028, 0xebf460, 0x26, 0x766f00)
        /home/erwo/go-devel/src/runtime/plugin.go:13 +0xc27 fp=0xc42003cba0 sp=0xc42003ca70
plugin.open(0x4a6a5f, 0xd, 0x0, 0x0, 0x0)
        /home/erwo/go-devel/src/plugin/plugin_dlopen.go:72 +0x25f fp=0xc42003cde0 sp=0xc42003cba0
plugin.Open(0x4a6a5f, 0xd, 0x0, 0x2, 0x3)
        /home/erwo/go-devel/src/plugin/plugin.go:30 +0x37 fp=0xc42003ce18 sp=0xc42003cde0
main.RunGoPlugin()
        /home/erwo/golang-plugin/src/indirect_with_c/indirect_with_c.go:12 +0x52 fp=0xc42003cea8 sp=0xc42003ce18
main._cgoexpwrap_8c4bdecea134_RunGoPlugin()
        indirect_with_c/_obj/_cgo_gotypes.go:45 +0x16 fp=0xc42003ceb0 sp=0xc42003cea8
runtime.call32(0x0, 0x7fff7c9c1108, 0x7fff7c9c11af, 0x0)
        /home/erwo/go-devel/src/runtime/asm_amd64.s:501 +0x4a fp=0xc42003cee0 sp=0xc42003ceb0
runtime.cgocallbackg1(0x0)
        /home/erwo/go-devel/src/runtime/cgocall.go:297 +0x1a1 fp=0xc42003cf58 sp=0xc42003cee0
runtime.cgocallbackg(0x0)
        /home/erwo/go-devel/src/runtime/cgocall.go:184 +0x86 fp=0xc42003cfc0 sp=0xc42003cf58
runtime.cgocallback_gofunc(0x0, 0x0, 0x0, 0x0)
        /home/erwo/go-devel/src/runtime/asm_amd64.s:754 +0x71 fp=0xc42003cfe0 sp=0xc42003cfc0
runtime.goexit()
        /home/erwo/go-devel/src/runtime/asm_amd64.s:2184 +0x1 fp=0xc42003cfe8 sp=0xc42003cfe0

goroutine 18 [syscall, locked to thread]:
runtime.goexit()
        /home/erwo/go-devel/src/runtime/asm_amd64.s:2184 +0x1
Aborted

Other maybe related issues

#17928
#17150
#18120

[bradfitz]

/cc @crawshaw

[ianlancetaylor]

This is hard. We have not figured out how to implement a program that has multiple partial copies of the Go runtime. The immediate problem here is that the c-archive does not export the symbols that the plugin is looking for, so we have multiple copies of those symbols in the program space. But addressing that problem piecemeal is not going to work without an understanding of where we are headed, and I don't know the answer to that.

I don't see this happening for 1.8 unless @crawshaw can figure out a way.

[typetetris]

I have no formal training/experience in that field, so my suggestions might be garbage. Please tell me if it is so, and why.

Couldn't you just declare all symbols that make up the runtime as weak, when -buildmode=plugin is used and advise users to link the c program with --export-dynamic/--dynamic-list if they want to use plugin from a c-archive? So if the plugin is loaded, the symbols from the c-archive overrule those in the plugin? Voila -> one runtime ? At least for elf platforms?

You could even make the go tool emit a list of symbols to be used with --dynamic-list like go list_runtime_symbols > file_for_dynamic_list.

[ianlancetaylor]

I agree that that would probably work, but given that there is no reason to believe that the person writing the program is the person writing the plugin I think that expecting the program to reliably use --dynamic-list correctly is less than ideal.

[crawshaw]

My thinking here is that 1. c-archive/c-shared should include all the necessary runtime symbols, and 2. plugins should not include the runtime (#17150).

That's not a general solution though. Loading multiple c-archive/c-shared libraries should work.

[typetetris]

I assumed there was a fixed list of symbols the writer of the program must list for --dynamic-list which is independent of the plugins being used. If that assumption is correct, the two needn't know each other. The writer of the plugin and the writer of the program that is.

If the writer of the program doesn't anticipate the use of stdlib package plugin from within the c-archives he uses to link his program with, I would argue it is a security feature, if plugins stop to work, if he didn't use --dynamic-list. Ok, maybe that is not a good argument.

On the other hand, if you could just do

gcc -o hostapp hostapp.c go-archive.a $(go link_c_archive_flags)

why would that be worse then using pkg-config which you already do all the time?

[paultag]

Is there any way out of this bug? I don't think I quite understand the failure mode here (as a user)

I have a .so I built as ac-shared that tries to do a plugin.Open to an ELF that was built using buildmode=plugin. This seems to describe this issue, but I'm not quite sure on the best way to work around this.

I tried adding -dynamic-list and -dynamic-export to the LDFLAGS, but I was having trouble setting them correctly since I don't quite understand the magic behind plugin. I guess in the meantime I can just use c-shared and dlopen it, but I kinda hate having to rewrite hunks of stdlib in my app

(using go1.8.3 on a linux/amd64)

[typetetris]

I don't know a way out of this bug. But a alternative solution for the use case (extending a host application, which can use shared libraries) I would suggest to create an rpc mechanism and live with the network (unix domain sockets are fast) boundary.

[JohnStarich]

I believe I'm hitting a similar issue, but only on Linux. It works on macOS, surprisingly. I've created a repo to help reproduce the issue: https://github.com/JohnStarich/go-plugin-issues

The repo contains tests for normal Go executables loading plugins and also the indirect approach with a compiled C program loading a c-shared lib, which then tries to load/run a plugin.
The direct approach works fine, but starting with the C program and loading through a c-shared lib fails on Linux.

Perhaps this working on macOS could be a clue to a possible solution. Is there anything I can do or provide that may help?

macOS environment info

➜ go version
go version go1.11.5 darwin/amd64
➜ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/johnstarich/Library/Caches/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/johnstarich/Documents/Projects/go"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/Cellar/go/1.11.5/libexec"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.11.5/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/2x/wnf9b7qx7rv9vbhw0gzbffdr0000gn/T/go-build584225572=/tmp/go-build -gno-record-gcc-switches -fno-common"

Linux environment info

(Travis CI build log)

$ travis_setup_go
go version go1.11.5 linux/amd64
$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/travis/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/travis/gopath"
GOPROXY=""
GORACE=""
GOROOT="/home/travis/.gimme/versions/go1.11.5.linux.amd64"
GOTMPDIR=""
GOTOOLDIR="/home/travis/.gimme/versions/go1.11.5.linux.amd64/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build948331263=/tmp/go-build -gno-record-gcc-switches"

Edit: If this is not the same issue, let me know and I'll open a new one instead 😄

[JohnStarich]

@ianlancetaylor @crawshaw Any update on this or #17150?

[bradfitz]

@JohnStarich, no updates. Nobody is working on the plugin package.