cmd/go: plugin versions do not match when built within vs. outside a module #31354

zimnx · 2019-04-08T23:23:52Z

What did you do?

I have simple application that loads Go plugins and allows them to communicate with app via exported interface.

Simplified version can be found here.
Shared interface is stored under ext directory, real implementation is located underplugin directory.
Application passes pointer to real implementation to dynamically loaded plugin which expects interface.

Example plugin code can be found here.

Unfortunately combination of Go Modules and Go Plugins doesn't work unless go.mod of plugins module has replace entry with local relative path for shared interface path.
When I use remote location in plugins module, application crashes because of wrong version of packages.

Build app:

~/tmp/go-plugin-bug/central on  master ⌚ 16:12:19
$ go clean -modcache 

~/tmp/go-plugin-bug/central on  master ⌚ 16:12:21
$ git checkout v1.0.0
Note: checking out 'v1.0.0'.

~/tmp/go-plugin-bug/central on  f1d7e9f ⌚ 16:12:22
$ go install -a

Then build plugin

~/tmp/go-plugin-bug/central on  f1d7e9f ⌚ 16:12:27
$ cd ../plugins 

~/tmp/go-plugin-bug/plugins on  master ⌚ 16:12:28
$ cat go.mod 
module github.com/zimnx/plugins

go 1.12

require github.com/zimnx/central v1.0.0


~/tmp/go-plugin-bug/plugins on  master ⌚ 16:12:30
$ go build -buildmode=plugin -o plugin.so
go: finding github.com/zimnx/central v1.0.0
go: downloading github.com/zimnx/central v1.0.0
go: extracting github.com/zimnx/central v1.0.0

~/tmp/go-plugin-bug/plugins on  master ⌚ 16:12:39
$ central plugin.so 
2019/04/08 16:12:42 cant open plugin: plugin.Open("plugin"): plugin was built with a different version of package github.com/zimnx/central/ext

When I change plugins go mod to use local path instead of remote one everything works.

~/tmp/go-plugin-bug/plugins on  master ⌚ 16:12:42
$ cat go.mod                             
module github.com/zimnx/plugins

go 1.12

require github.com/zimnx/central v1.0.0

replace github.com/zimnx/central => ../central

~/tmp/go-plugin-bug/plugins on  master! ⌚ 16:14:30
$ go build -buildmode=plugin -o plugin.so

~/tmp/go-plugin-bug/plugins on  master! ⌚ 16:14:33
$ central plugin.so 
hello = world

What did you expect to see?

Go Modules and plugins working fine when remote path is used.

What did you see instead?

Error about different package versions.

Does this issue reproduce with the latest release (go1.12.3)?

Yes.

System details

go version go1.12.3 linux/amd64
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/maciej/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/maciej/work"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/go"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/maciej/tmp/go-plugin-bug/plugins/go.mod"
GOROOT/bin/go version: go version go1.12.3 linux/amd64
GOROOT/bin/go tool compile -V: compile version go1.12.3
uname -sr: Linux 4.14.13-041413-generic
Distributor ID:	Ubuntu
Description:	Ubuntu 17.10
Release:	17.10
Codename:	artful
/lib/x86_64-linux-gnu/libc.so.6: GNU C Library (Ubuntu GLIBC 2.26-0ubuntu2.1) stable release version 2.26, by Roland McGrath et al.
gdb --version: GNU gdb (Ubuntu 8.0.1-0ubuntu1) 8.0.1

The text was updated successfully, but these errors were encountered:

zimnx · 2019-04-10T16:35:26Z

Bump. Any plans to fix this? Seems that algorithm which verifies package version integrity needs alignment.

bcmills · 2019-04-10T21:08:48Z

This is the same underlying problem as #29814.

The workaround should be to build both the plugin and the main binary from outside of their respective repositories, or (as you note) to use a replace directive when building locally.

bcmills · 2019-04-10T21:10:04Z

CC @jayconrod

bcmills · 2019-05-07T20:24:55Z

See also #31278, of which this may be a duplicate.

juhwany · 2019-06-14T13:59:41Z

@bcmills

Is this issue related to #16860 ?

Building plugin and main binary with -trimpath flag doesn't solve this issue even using a go toolchain built from head.

xxxserxxx · 2020-03-07T21:14:50Z

There are several similar tickets; apologies if this is not the right place to comment.

If the path is different but the version of the libraries are identical, we get version mismatch errors. A way of replicating this is to build the calling program in a container, and build the plugin outside of the container. The paths may look like this:

# Calling
/go/pkg/mod/github.com/shirou/gopsutil@v2.18.11+incompatible/internal/common/common.go
# Called
/home/ser/go/pkg/mod/github.com/shirou/gopsutil@v2.18.11+incompatible/internal/common/common.go

Building locally, and where:

the plugin is built using a go.mod with a specific date+revision version for the calling main program, and
the calling application is built from a checkout at that version, verified that the version in the plugin go.mod matches the checked-out version with:

echo $(TZ=UTC git show -s --format=%cd --date=format-local:%Y%m%d%H%M%S HEAD | cut -b -19 |  tr -cd '[:digit:]')-$(git rev-parse HEAD | cut -b -12)

and both programs being compiled with the same go executable, and
using --trimpath on both, and
both go.mod list the same go version dependency
I get the same version incompatibility error with a different package:

# Error:
2:31:33 main.go:485: plugin.Open("dummy"): plugin was built with a different version of package github.com/xxxserxxx/gotop/v3/devices
$ #######################################################
$ strings ./gotop | grep 'v3/devices$'
go.link.pkghashbytes.github.com/xxxserxxx/gotop/v3/devices
go.link.pkghash.github.com/xxxserxxx/gotop/v3/devices
%github.com/xxxserxxx/gotop/v3/devices
go.link.pkghashbytes.github.com/xxxserxxx/gotop/v3/devices
go.link.pkghash.github.com/xxxserxxx/gotop/v3/devices
$ #######################################################
$ strings dummy.so | grep 'v3/devices$'
go.link.pkghashbytes.github.com/xxxserxxx/gotop/v3/devices
go.link.pkghash.github.com/xxxserxxx/gotop/v3/devices
github.com/xxxserxxx/gotop-dummygithub.com/xxxserxxx/gotop-dummygithub.com/xxxserxxx/gotop/v3/devices
%github.com/xxxserxxx/gotop/v3/devices
go.link.pkghashbytes.github.com/xxxserxxx/gotop/v3/devices
go.link.pkghash.github.com/xxxserxxx/gotop/v3/devices

I've played with copying the go.sum into the plugin directory so that it's identical to the other project, and have made sure that the entries in both go.mods are the same (except for the plugin's dependency on the main program).

If I add the following to the plugin's go.mod:

replace github.com/xxxserxxx/gotop/v3 => ../gotop

then it works.

Edit

Rewrote some of the text to be more readable, I hope.

Edit 2

I made an error: I need to remove the -trimpath argument in addition to adding the replace directive in the go.mod, or else I still get a version incompatibility error. -trimpath appears to make things worse.

xplodwild · 2020-03-22T12:20:07Z

Same thing happens for me for a go.mod-based project regarding the -trimpath option. When building both the main app and the plugin with -trimpath enabled, I get a plugin was built with a different version of package github.com/me/myapp/mypackage. Removing -trimpath makes it load fine, however I'm afraid people with other setups might have trouble getting their plugins loaded due to paths inconsistencies.

xxxserxxx · 2020-03-24T18:23:49Z

Another data point which I can't even begin to interpret:

$ git clone https://github.com/xxxserxxx/gotop
$ git clone https://github.com/xxxserxxx/gotop-dummy
$ cd gotop && git checkout v3.5.0
$ go build -o gotop ./cmd/gotop
$ cd ../gotop-dummy && grep gotop go.mod
require github.com/xxxserxxx/gotop/v3 v3.5.0
$ go mod download
$ go mod vendor
$ go build --buildmode=plugin -o ../gotop/dummy.so .
$ cd ../gotop
$ ./gotop -X dummy    # this hangs for some reason; ^C to break
$ cat ~/.local/state/gotop/errors.log
13:03:16 main.go:485: plugin.Open("dummy"): plugin was built with a different version of package github.com/shirou/gopsutil/internal/common
$ go mod graph | grep gopsutil
github.com/xxxserxxx/gotop/v3 github.com/shirou/gopsutil@v2.18.11+incompatible
➜  gotop git:(be42ba5) ✗ git log -1
commit be42ba538cefc892d1dc3d18c783483f4875baff (HEAD, tag: v3.5.0)
$ cd ../gotop-dummy
$ go mod graph | grep gopsutil
github.com/xxxserxxx/gotop/v3@v3.5.0 github.com/shirou/gopsutil@v2.18.11+incompatible

These are the identical dependencies. Why is the system claiming that they are not?

pnegahdar · 2020-03-31T14:17:05Z

Still an issue we're running into as well.

hoijui · 2020-05-05T09:03:26Z

This is the same underlying problem as #29814.

The workaround should be to build both the plugin and the main binary from outside of their respective repositories, or (as you note) to use a replace directive when building locally.

do you mean to say, that the ext folder should be its own module, separate from the main app?
would that work?

mytototo · 2020-06-09T07:49:04Z

Has anybody found a way to build reproducible plugins for different machines?

knipknap · 2020-07-31T13:40:28Z

@mytototo There is no reliable way, as the problem is by design. Every dependency of every package used in both plugin and main app have to be exactly the same version, making plugin and app extremely highly coupled. Even if you have control over both, plugin and app, this is very hard or impossible to maintain because you don't have control over what 3rd party modules might import.

Unfortunately that means there are very few practical use cases for Go plugins, as far as I can see.

mark4z · 2020-12-25T16:05:28Z

A solution for this problem:

There are a project MAIN, If u want build some plugin for MAIN. You can create a project A and import the MAIN as its DockerFile, and create another project B and import the MAIN to build the plugin.
Then A and B import the same version for MAIN, every thing will be work perfect.

xplodwild · 2020-12-26T18:30:04Z

A solution for this problem:

There are a project MAIN, If u want build some plugin for MAIN. You can create a project A and import the MAIN as its DockerFile, and create another project B and import the MAIN to build the plugin.
Then A and B import the same version for MAIN, every thing will be work perfect.

@mark4z This is not a proper workaround, for me at least. I need to be able to let external developers build plugins for my app, based on a specific set of shared interfaces and structures, without having to (re)build everything inside a single environment.

mark4z · 2020-12-27T02:25:19Z

A solution for this problem:

There are a project MAIN, If u want build some plugin for MAIN. You can create a project A and import the MAIN as its DockerFile, and create another project B and import the MAIN to build the plugin.

Then A and B import the same version for MAIN, every thing will be work perfect.

@mark4z This is not a proper workaround, for me at least. I need to be able to let external developers build plugins for my app, based on a specific set of shared interfaces and structures, without having to (re)build everything inside a single environment.

Sure, maybe the last way is extraction everything the plug-in need and both projects import it. I'm working on it now.

simar7 · 2021-04-12T23:25:00Z

I'm on go 1.16 and still seeing this problem.

mattn · 2021-05-26T14:38:05Z

The current plugin mechanism does not allow you to avoid mixed package hashes. For example, this error occurs when a main application that references package A loads plugins that references A. I often use gophernotes to try to write notebook with Go. gophernotes reference mattn/go-runewidth. So gophernotes can not import mattn/go-runewidth on the notebooks. I have to disable Go code to avoid this.

diff --git a/src/runtime/plugin.go b/src/runtime/plugin.go
index cd7fc5f848..ace1b13ed9 100644
--- a/src/runtime/plugin.go
+++ b/src/runtime/plugin.go
@@ -48,12 +48,14 @@ func plugin_lastmoduleinit() (path string, syms map[string]interface{}, errstr s
                        throw("plugin: new module data overlaps with previous moduledata")
                }
        }
-       for _, pkghash := range md.pkghashes {
-               if pkghash.linktimehash != *pkghash.runtimehash {
-                       md.bad = true
-                       return "", nil, "plugin was built with a different version of package " + pkghash.modulename
+       /*
+               for _, pkghash := range md.pkghashes {
+                       if pkghash.linktimehash != *pkghash.runtimehash {
+                               md.bad = true
+                               return "", nil, "plugin was built with a different version of package " + pkghash.modulename
+                       }
                }
-       }
+       */

        // Initialize the freshly loaded module.
        modulesinit()

bcmills changed the title ~~plugin: Plugins doesn't work when remote path of dependency is used in Go Module~~ cmd/go: plugin versions do not match when built within vs. outside a module Apr 10, 2019

bcmills added NeedsInvestigation modules labels Apr 10, 2019

bcmills added this to the Go1.13 milestone Apr 10, 2019

bcmills mentioned this issue May 13, 2019

plugin: unable to load plugin if go mod is not enabled #31951

Closed

ashutosh-narkar mentioned this issue Jun 26, 2019

Document how plugin can be used with vendored and GOPATH OPA dependency. open-policy-agent/opa-envoy-plugin#94

Closed

bcmills modified the milestones: Go1.13, Go1.14 Jun 28, 2019

rsc modified the milestones: Go1.14, Backlog Oct 9, 2019

gszr mentioned this issue Feb 12, 2020

Custom Go plugins on Kong 2.0 base Kong/docker-kong#326

Closed

xxxserxxx mentioned this issue Mar 8, 2020

plugin was built with a different version of package internal/cpu xxxserxxx/gotop#78

Closed

bcmills mentioned this issue Apr 20, 2020

cmd/go: executable created using linkshared option is not linking .so created by buildmode=shared with go modules enabled #38499

Open

liamsi mentioned this issue Feb 16, 2021

Use custom multihash and refactor plugin lazyledger/lazyledger-core#155

Merged

5 of 6 tasks complete

nyanpassu mentioned this issue Mar 22, 2021

Supporting remote plugin for tasks etc. containerd/containerd#5241

Open

powerkimhub mentioned this issue Apr 7, 2021

[Static build mode vs Plugin build mode and Private Repos Driver] cloud-barista/cb-spider#343

Closed

golang / go

cmd/go: plugin versions do not match when built within vs. outside a module #31354

cmd/go: plugin versions do not match when built within vs. outside a module #31354

zimnx commented Apr 8, 2019 •

edited

zimnx commented Apr 10, 2019

bcmills commented Apr 10, 2019

bcmills commented Apr 10, 2019

bcmills commented May 7, 2019

juhwany commented Jun 14, 2019 •

edited

xxxserxxx commented Mar 7, 2020 •

edited

xplodwild commented Mar 22, 2020

xxxserxxx commented Mar 24, 2020

pnegahdar commented Mar 31, 2020

hoijui commented May 5, 2020

mytototo commented Jun 9, 2020

knipknap commented Jul 31, 2020

mark4z commented Dec 25, 2020

xplodwild commented Dec 26, 2020

mark4z commented Dec 27, 2020

simar7 commented Apr 12, 2021

mattn commented May 26, 2021

golang / go

cmd/go: plugin versions do not match when built within vs. outside a module #31354

cmd/go: plugin versions do not match when built within vs. outside a module #31354

Comments

zimnx commented Apr 8, 2019 • edited

What did you do?

What did you expect to see?

What did you see instead?

Does this issue reproduce with the latest release (go1.12.3)?

System details

zimnx commented Apr 10, 2019

bcmills commented Apr 10, 2019

bcmills commented Apr 10, 2019

bcmills commented May 7, 2019

juhwany commented Jun 14, 2019 • edited

xxxserxxx commented Mar 7, 2020 • edited

Edit

Edit 2

xplodwild commented Mar 22, 2020

xxxserxxx commented Mar 24, 2020

pnegahdar commented Mar 31, 2020

hoijui commented May 5, 2020

mytototo commented Jun 9, 2020

knipknap commented Jul 31, 2020

mark4z commented Dec 25, 2020

xplodwild commented Dec 26, 2020

mark4z commented Dec 27, 2020

simar7 commented Apr 12, 2021

mattn commented May 26, 2021

zimnx commented Apr 8, 2019 •

edited

juhwany commented Jun 14, 2019 •

edited

xxxserxxx commented Mar 7, 2020 •

edited