plugin: panic: plugin was built with a different version of package

[juhwany]

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go version go1.11 darwin/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN="/Users/juhwany/go_workspace/bin"
GOCACHE="/Users/juhwany/Library/Caches/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/juhwany/go_workspace"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/go"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/juhwany/gos/go_server/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/mk/2tyywdsn0jq6l0vl8094tss80000gp/T/go-build380269681=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

1. Clone the blow repo in each directory. (not under GOPATH)
   server (main program) : https://github.com/juhwany/go_server
   plugin : https://github.com/juhwany/go_plugin

2. On same machine, build server & plugin in each directory
   Build server : go build -o main
   Build plugin : go build -buildmode=plugin -o plugin.so

3. Copy plugin.so to server directory and run
   ./main

4. See error message when main program try to open plugin.

panic: plugin.Open("plugin"): plugin was built with a different version of package github.com/juhwany/go_server/v2/apis

What did you expect to see?

Main program should load plugin.so successfully.
I could not understand why it fails even though the referenced package version is same between main program and plugin.
Am I using plugin in wrong way?

What did you see instead?

plugin was built with a different version of package error message

[randall77]

Related (dup?) #26759 .

[juhwany]

@randall77

No. I think building environment is slightly different from #26759.
I didn't build main program and plugin under GOPATH and the above sample project uses Go module for specifying dependency package. Go version is also different.

[BryceDFisher]

I do think they are all related though, including #27062. I think it all boils down to changing (absolute) paths at build time between the plugin build and the app build.

[bcmills]

CC @ianlancetaylor @cherrymui

[tamasfe]

I'm not sure if anyone's already mentioned it, but putting the common(apis) package to a separate module outside the server one will work. It's definitely cumbersome, but can be used as a workaround until this issue is fixed.

[oszika]

The command "go list -m -json all" shows version of go_server on go_plugin, but it's not defined on go_server itself. I have a similar problem using multiples modules on the same repository using "replace" directive. I don't know how to have the same version of the module inside and outside the repository.

[oszika]

In #28983, @bcmills suggests to build plugins from within the main module.

// server/go.mod
module github.com/juhwany/go_server/v2
require github.com/juhwany/plugin v0.0.0
replace github.com/juhwany/plugin => ../go_plugin

$ go build -buildmode=plugin github.com/juhwany/plugin
$ go run main.go 
DoSomething called!

[jbrukh]

Just throwing in my two cents here. Both the @tamasf97 and @oszika are workarounds.

However, @tamasf97's workaround only works if you can factor the shared code out without creating a circular dependency (not always possible).

@oszika's workaround using the "replace" directive is pretty inconvenient for clients of the shared code, as you need that local repo.

[itsmontoya]

We are unable to run our plugin-based service with gomod due to this issue.

[dhalman]

We are experiencing this issue after implementing go.mod in both the server and plugin

[bcmills]

@itsmontoya, @dhalman: what specific commands did you run to build the plugin?

(Did you follow the approach in #27751 (comment)? Did you build with -trimpath? Both of those may be necessary.)

[itsmontoya]

@bcmills we will try -trimpath and the other approach and get back to you ASAP! 👍

[dhalman]

added -trimpath and the error moved from our microservice dependency lib to now internal/cpu:

plugin.Open("plugins/thePlugin"): plugin was built with a different version of package internal/cpu

[novabyte]

@dhaimeen We've used the Go plugin API for quite a while now (over a year) and the error "different version of package internal/cpu" occurs when you've not used the -trimpath flag on both the shared object you build (plugin) and the main binary. Both bundle a copy of the Go runtime (although there's an open issue on the tracker about not packaging a copy of the Go runtime with plugin code). Hope this helps.

[brightzheng100]

I encountered the same issue while trying out this repo: https://github.com/vladimirvivien/gosh

To reproduce it, this is the process:

go version      # go version go1.16 darwin/amd64
dlv version     # Version: 1.6.0

git clone https://github.com/vladimirvivien/gosh
cd gosh

go mod init github.com/vladimirvivien/gosh  # to fix the running issue

This is all good:

go build -buildmode=plugin -o plugins/splash_command.so plugins/00_splash.go
go build -buildmode=plugin -o plugins/sys_command.so plugins/syscmd.go
go run gosh.go  # all good

But when try debugging by setting the breakpoint at /gosh.go, line 59, I always get an error:

"plugin.Open("plugins/splash_command"): plugin was built with a different version of package runtime/internal/sys"

If I try this instead by adding the -trimpath:

go build -trimpath -buildmode=plugin -o plugins/splash_command.so plugins/00_splash.go
go build -trimpath -buildmode=plugin -o plugins/sys_command.so plugins/syscmd.go
go run gosh.go

I would get this error:

failed to open plugin splash_command.so: plugin.Open("plugins/splash_command"): plugin was built with a different version of package internal/unsafeheader
failed to open plugin sys_command.so: plugin.Open("plugins/sys_command"): plugin was built with a different version of package internal/unsafeheader

Why?

[kriptor]

We have developed an open source solution for syncing go.mod repositories across large projects with internal dependencies. We find it especially useful for managing large collections of plugins along with their private imports.

Happy to share it if requested, but would not want to advertise in this forum

Hey @dhalman,

could you post the link to the open source syncing solution of yours. I find developing any a bit non-trivial plugin for any a bit non-trivial app a total nightmare. Maybe your solution might help me out. :)

Cheers

[dhalman]

@kriptor i havent tried it in a while, but im here to help support if you do end up using it

https://github.com/gomuserver/gomu

[paralin]

I posted some comments here about this issue previously, right now you can view the workarounds I've built as well as a work in progress hot reloading system (which is hampered by this bug) here: https://github.com/aperturerobotics/controllerbus/tree/master/example/plugin-demo

[henri9813]

Thank you @paralin , but my problem is ... very ennoyinh.

Package A include C version 1.1 ( go mod used ).

Package B ( the plugin ) include C version 1.1 go mod used.

Result, Plugin B was buillt with a different version of C.

My "workaround" was to remove the go.mod used in the B package to make it works. Very dirty solution in my opinion.

[ProgramComputer]

In #28983, @bcmills suggests to build plugins from within the main module.

// server/go.mod
module github.com/juhwany/go_server/v2
require github.com/juhwany/plugin v0.0.0
replace github.com/juhwany/plugin => ../go_plugin

$ go build -buildmode=plugin github.com/juhwany/plugin
$ go run main.go 
DoSomething called!

I had initially built the plugin with the same go.mod file as the main module. I encountered the issue when I moved the plugin into its own module. I tried --trimpath to build both the plugin and main module but did not succeed. The reference is a place to look.

@juhwany Does this issue still exist for you?

[knuppe]

Apparently disabling optimizations and inlining -gcflags="all=-N -l" "resolves" the issue for me, but the output files are huge :(

go build -buildmode=plugin -gcflags="all=-N -l" -o plugins/test.so plugins/test.go

[MatteoGioioso]

I have basically tried everything and in the end I have this error: plugin was built with a different version of package internal/goarch". I am using exact same dependencies, go version, built environments, flags, etc...

[cloudintegrator]

Well, I am still stuck with the same problem. Not sure if there will be any update on this issue. Exactly for this issue we had to choose another language for the modular system we were working on.

[itsmontoya]

I had to move Vroomy away from Plugins which was unfortunate.

[venkat-ninja]

Any update on this issue

[seankhliao]

Closing as I don't think there is any specific issue with the plugin package itself, it all appears to be application build environment differences.