Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

plugin: gopath type != vendor type #18827

Open
fsenart opened this issue Jan 28, 2017 · 17 comments
Open

plugin: gopath type != vendor type #18827

fsenart opened this issue Jan 28, 2017 · 17 comments

Comments

@fsenart
Copy link

@fsenart fsenart commented Jan 28, 2017

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go version go1.8rc3 linux/amd64

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/fsenart/projects"
GORACE=""
GOROOT="/home/fsenart/tools/go1.8rc3"
GOTOOLDIR="/home/fsenart/tools/go1.8rc3/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build101924921=/tmp/go-build -gno-record-gcc-switches"
CXX="g++"
CGO_ENABLED="1"
PKG_CONFIG="pkg-config"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"

What did you do?

If possible, provide a recipe for reproducing the error.
A complete runnable program is good.
A link on play.golang.org is best.

Scenario:

  1. Build a main program with some dependency located in the GOPATH
  2. Build a plugin program with the same dependency vendored
  3. Load the plugin from main

Result:

  • At runtime, the GOPATH dependency is not the same as the vendored depedency.
// $GOPATH/src/issues/issue_vendor/main .go

package main

import (
	"foo"
	"plugin"
	"reflect"
)

func main() {
	p, err := plugin.Open("plugin.so")
	if err != nil {
		panic(err)
	}
	f, err := p.Lookup("F")
	if err != nil {
		panic(err)
	}

	println(reflect.TypeOf(f).Elem() == reflect.TypeOf((*foo.Bar)(nil)))
}
// $GOPATH/src/issues/issue_vendor/plugin.go

package main

import "C"
import "foo"

var F *foo.Bar
// Copy the file one time in the GOPATH and one time in the vendor directory
// $GOPATH/src/foo/bar.go
// $GOPATH/src/issues/issue_vendor/vendor/foo/bar.go

package foo

type Bar struct{}
# $GOPATH/src/issues/issue_vendor/Makefile

test1: build1
	@./main

test2: build2
	@./main

build1: clean
	@go build -o main main.go
	@go build -buildmode=plugin -o plugin.so plugin.go

build2: clean
	@mv vendor _vendor
	@go build -o main main.go
	@mv _vendor vendor
	@go build -buildmode=plugin -o plugin.so plugin.go

clean:
	@rm -rf main

What did you expect to see?

  • make test1 should print true
  • make test2 should print true

What did you see instead?

  • make test1 prints true
  • make test2 prints false
@fsenart

This comment has been minimized.

Copy link
Author

@fsenart fsenart commented Feb 20, 2017

@ianlancetaylor is there any chance to add this issue to the 1.8.1 milestone as it makes the usage of vendoring nearly impossible in the context of plugins?

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Feb 20, 2017

Adding the 1.8.1 milestone to this issue won't fix the problem. When and if we have a fix on tip, then if the fix is simple and safe, we can consider adding it to 1.8.1.

@edaniels

This comment has been minimized.

Copy link
Contributor

@edaniels edaniels commented Feb 23, 2017

As mentioned in #19233, this sounds like the behavior described at #12432. Unless this behavior is set to change in 1.9, this seems like a non-issue, albeit annoying in the context of plugins.

@fsenart

This comment has been minimized.

Copy link
Author

@fsenart fsenart commented Feb 23, 2017

@edaniels it really seems like the behavior you mentioned but the nuance resides in the fact that this issue happens on run time. I understand that this is the "plugin" version of the behavior, but this is also far more annoying for end users as it can't be detected at compile time.
Plugins come with a lot of advantages but also open the pandora's box. IMHO, the main advantage is to allow the separation between a core program and a myriad of run time functionalities developed by different folks but this kind of issues constraint both parties to know about each others and conform to a set of common dependencies rules.
Maybe we should come up with a freshen look about vendoring in the context of plugins.

@edaniels

This comment has been minimized.

Copy link
Contributor

@edaniels edaniels commented Feb 23, 2017

I totally agree. It's probably a discussion for another issue but it seems like this is an issue for vendoring in general and maybe the go tool chain should be more aware of package identity in the future. To make plugins more useful something will need to be done.

@crawshaw

This comment has been minimized.

Copy link
Contributor

@crawshaw crawshaw commented Sep 3, 2017

If you built this as a single program, there would be two separate packages for foo. There would be two instances of any global variables, and the types you are comparing would be different.

It seems very strange to me that plugins would de-duplicate vendored packages. What if a plugin has vendored a different version of a package because it wants to use a different version? The only thing we could do in plugin.Open is fail in this case, because the packages wouldn't be compatible.

It would also be impossible to build a plugin that vendors two different versions of a package under different subtrees. I can't say I've ever done this, but all of this is possible with the vendor directory in usual programs, so I think it should work in plugins also.

@odeke-em

This comment has been minimized.

Copy link
Member

@odeke-em odeke-em commented Dec 11, 2017

Sorry that we didn't look at this issue in the Go1.10 cycle, but I think we might have to move this to Go1.11. What do you think @crawshaw?

@bradfitz

This comment has been minimized.

Copy link
Contributor

@bradfitz bradfitz commented Dec 13, 2017

@crawshaw is on leave, so that's a yes: moving to Go 1.11.

@bradfitz bradfitz modified the milestones: Go1.10, Go1.11 Dec 13, 2017
@odeke-em

This comment has been minimized.

Copy link
Member

@odeke-em odeke-em commented Dec 13, 2017

Oh, right, I had forgotten. Thanks @bradfitz.

@frankgreco frankgreco mentioned this issue Jan 27, 2018
0 of 8 tasks complete
@frankgreco

This comment has been minimized.

Copy link

@frankgreco frankgreco commented Jan 29, 2018

As this issue has been assigned to a milestone, what exactly is proposed to be changed/addressed?

@crawshaw says:

It would also be impossible to build a plugin that vendors two different versions of a package under different subtrees.

If this is the case, I'm struggling to find the relevance in the Plugin package entirely. If plugins are to always be tightly coupled with the dependencies of the project that loads it, why would I not just include that code in the parent project's source.

alegen added a commit to emersyx/emersyx_router that referenced this issue Mar 3, 2018
It is currently not possible to have API interfaces in the "vendor" folder. If two or more projects share the same interfaces via their own vendor folders, the go compiler will not treat the interfaces as equal. The reason is the different paths to the sources on the filesystem. There is an open issue on Github here golang/go#18827.
alegen added a commit to emersyx/irc2telegram that referenced this issue Mar 3, 2018
It is currently not possible to have API interfaces in the "vendor" folder. If two or more projects share the same interfaces via their own vendor folders, the go compiler will not treat the interfaces as equal. The reason is the different paths to the sources on the filesystem. There is an open issue on Github here golang/go#18827.
alegen added a commit to emersyx/telegram that referenced this issue Mar 3, 2018
It is currently not possible to have API interfaces in the "vendor" folder. If two or more projects share the same interfaces via their own vendor folders, the go compiler will not treat the interfaces as equal. The reason is the different paths to the sources on the filesystem. There is an open issue on Github here golang/go#18827.
alegen added a commit to emersyx/irc that referenced this issue Mar 3, 2018
It is currently not possible to have API interfaces in the "vendor" folder. If two or more projects share the same interfaces via their own vendor folders, the go compiler will not treat the interfaces as equal. The reason is the different paths to the sources on the filesystem. There is an open issue on Github here golang/go#18827.
alegen added a commit to emersyx/emersyx_core that referenced this issue Mar 3, 2018
It is currently not possible to have API interfaces in the "vendor" folder. If two or more projects share the same interfaces via their own vendor folders, the go compiler will not treat the interfaces as equal. The reason is the different paths to the sources on the filesystem. There is an open issue on Github here golang/go#18827.
alegen added a commit to emersyx/emersyx_core that referenced this issue Mar 3, 2018
It is currently not possible to have API interfaces in the "vendor" folder. If two or more projects share the same interfaces via their own vendor folders, the go compiler will not treat the interfaces as equal. The reason is the different paths to the sources on the filesystem. There is an open issue on Github here golang/go#18827.
alegen added a commit to emersyx/emersyx_core that referenced this issue Mar 3, 2018
It is currently not possible to have API interfaces in the "vendor" folder. If two or more projects share the same interfaces via their own vendor folders, the go compiler will not treat the interfaces as equal. The reason is the different paths to the sources on the filesystem. There is an open issue on Github here golang/go#18827.
kemokemo added a commit to kemokemo/kuronan-dash that referenced this issue May 6, 2018
@ianlancetaylor ianlancetaylor modified the milestones: Go1.11, Unplanned Jun 27, 2018
@e-nikolov

This comment has been minimized.

Copy link

@e-nikolov e-nikolov commented Jul 27, 2018

I am not sure if this behaviour is a bug or a feature. On one hand it prevents a base application and a plugin from passing shared types to each other, but on the other hand it allows two plugins to vendor different versions of the same dependency and still be loaded. Without this behaviour, the plugin loader will complain that the second plugin was built with a different version of the shared dependency.

The issue with a shared type being returned by a plugin function to the base could be worked around by either pulling that specific dependency out of vendor/ into the GOPATH or by using a combination of interfaces, type assertion and reflection to convert the two types.

Also should the discussion on this issue happen here or in #20481?

@ash2k

This comment has been minimized.

Copy link

@ash2k ash2k commented Jul 27, 2018

I tried using plugins a while ago and gave up. I had 2+ plugins (and the main application too) that use kubernetes libraries and kubernetes uses glog. So, 2+ glog copies, each trying to add a flag to the flag package. A flag cannot be added more than once -> glog/stdlib panics. Plugins are completely unuseable in such situation.
I think shared mutable state should be removed from standard library in Go 2.

@e-nikolov

This comment has been minimized.

Copy link

@e-nikolov e-nikolov commented Jul 27, 2018

A similar thing happens for plugins that import x/net/trace which in its init() adds some http handlers to the http.DefaultServeMux. So if you load 2 plugins that have vendored x/net/trace, the init() will panic because it gets executed twice and you can't have two handlers registered at the same route.

The only way to "resolve" this right now seems to be to reassign http.DefaultServeMux before loading each plugin.

@graup

This comment has been minimized.

Copy link

@graup graup commented Nov 27, 2018

Are there any updates on this? I agree with the commenters above that this renders plugins useless for a lot of use cases. There should at least be a warning at the top of the documentation of the plugins package that when working with vendored code you cannot share types between different packages.

@juhwany

This comment has been minimized.

Copy link

@juhwany juhwany commented Sep 9, 2019

Is there any progress? It's still very difficult to use plugins

@edaniels

This comment has been minimized.

Copy link
Contributor

@edaniels edaniels commented Sep 9, 2019

It seems like plugins are effectively dead, support wise. They've received little mention since their release.

@huxiangdong

This comment has been minimized.

Copy link

@huxiangdong huxiangdong commented Sep 9, 2019

then are there any other alternative we can leverage?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
You can’t perform that action at this time.