Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

plugin: gopath type != vendor type #18827

Closed
fsenart opened this issue Jan 28, 2017 · 24 comments
Closed

plugin: gopath type != vendor type #18827

fsenart opened this issue Jan 28, 2017 · 24 comments
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. FrozenDueToAge help wanted NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Milestone

Comments

@fsenart
Copy link

fsenart commented Jan 28, 2017

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go version go1.8rc3 linux/amd64

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/fsenart/projects"
GORACE=""
GOROOT="/home/fsenart/tools/go1.8rc3"
GOTOOLDIR="/home/fsenart/tools/go1.8rc3/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build101924921=/tmp/go-build -gno-record-gcc-switches"
CXX="g++"
CGO_ENABLED="1"
PKG_CONFIG="pkg-config"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"

What did you do?

If possible, provide a recipe for reproducing the error.
A complete runnable program is good.
A link on play.golang.org is best.

Scenario:

  1. Build a main program with some dependency located in the GOPATH
  2. Build a plugin program with the same dependency vendored
  3. Load the plugin from main

Result:

  • At runtime, the GOPATH dependency is not the same as the vendored depedency.
// $GOPATH/src/issues/issue_vendor/main .go

package main

import (
	"foo"
	"plugin"
	"reflect"
)

func main() {
	p, err := plugin.Open("plugin.so")
	if err != nil {
		panic(err)
	}
	f, err := p.Lookup("F")
	if err != nil {
		panic(err)
	}

	println(reflect.TypeOf(f).Elem() == reflect.TypeOf((*foo.Bar)(nil)))
}
// $GOPATH/src/issues/issue_vendor/plugin.go

package main

import "C"
import "foo"

var F *foo.Bar
// Copy the file one time in the GOPATH and one time in the vendor directory
// $GOPATH/src/foo/bar.go
// $GOPATH/src/issues/issue_vendor/vendor/foo/bar.go

package foo

type Bar struct{}
# $GOPATH/src/issues/issue_vendor/Makefile

test1: build1
	@./main

test2: build2
	@./main

build1: clean
	@go build -o main main.go
	@go build -buildmode=plugin -o plugin.so plugin.go

build2: clean
	@mv vendor _vendor
	@go build -o main main.go
	@mv _vendor vendor
	@go build -buildmode=plugin -o plugin.so plugin.go

clean:
	@rm -rf main

What did you expect to see?

  • make test1 should print true
  • make test2 should print true

What did you see instead?

  • make test1 prints true
  • make test2 prints false
@fsenart
Copy link
Author

fsenart commented Feb 20, 2017

@ianlancetaylor is there any chance to add this issue to the 1.8.1 milestone as it makes the usage of vendoring nearly impossible in the context of plugins?

@ianlancetaylor
Copy link
Contributor

Adding the 1.8.1 milestone to this issue won't fix the problem. When and if we have a fix on tip, then if the fix is simple and safe, we can consider adding it to 1.8.1.

@edaniels
Copy link
Contributor

edaniels commented Feb 23, 2017

As mentioned in #19233, this sounds like the behavior described at #12432. Unless this behavior is set to change in 1.9, this seems like a non-issue, albeit annoying in the context of plugins.

@fsenart
Copy link
Author

fsenart commented Feb 23, 2017

@edaniels it really seems like the behavior you mentioned but the nuance resides in the fact that this issue happens on run time. I understand that this is the "plugin" version of the behavior, but this is also far more annoying for end users as it can't be detected at compile time.
Plugins come with a lot of advantages but also open the pandora's box. IMHO, the main advantage is to allow the separation between a core program and a myriad of run time functionalities developed by different folks but this kind of issues constraint both parties to know about each others and conform to a set of common dependencies rules.
Maybe we should come up with a freshen look about vendoring in the context of plugins.

@edaniels
Copy link
Contributor

I totally agree. It's probably a discussion for another issue but it seems like this is an issue for vendoring in general and maybe the go tool chain should be more aware of package identity in the future. To make plugins more useful something will need to be done.

@crawshaw
Copy link
Member

crawshaw commented Sep 3, 2017

If you built this as a single program, there would be two separate packages for foo. There would be two instances of any global variables, and the types you are comparing would be different.

It seems very strange to me that plugins would de-duplicate vendored packages. What if a plugin has vendored a different version of a package because it wants to use a different version? The only thing we could do in plugin.Open is fail in this case, because the packages wouldn't be compatible.

It would also be impossible to build a plugin that vendors two different versions of a package under different subtrees. I can't say I've ever done this, but all of this is possible with the vendor directory in usual programs, so I think it should work in plugins also.

@odeke-em
Copy link
Member

Sorry that we didn't look at this issue in the Go1.10 cycle, but I think we might have to move this to Go1.11. What do you think @crawshaw?

@bradfitz
Copy link
Contributor

@crawshaw is on leave, so that's a yes: moving to Go 1.11.

@bradfitz bradfitz modified the milestones: Go1.10, Go1.11 Dec 13, 2017
@odeke-em
Copy link
Member

Oh, right, I had forgotten. Thanks @bradfitz.

@frankgreco
Copy link

As this issue has been assigned to a milestone, what exactly is proposed to be changed/addressed?

@crawshaw says:

It would also be impossible to build a plugin that vendors two different versions of a package under different subtrees.

If this is the case, I'm struggling to find the relevance in the Plugin package entirely. If plugins are to always be tightly coupled with the dependencies of the project that loads it, why would I not just include that code in the parent project's source.

kemokemo added a commit to kemokemo/kuronan-dash that referenced this issue May 6, 2018
@ianlancetaylor ianlancetaylor added NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. help wanted labels Jun 27, 2018
@ianlancetaylor ianlancetaylor removed this from the Go1.11 milestone Jun 27, 2018
@e-nikolov
Copy link

e-nikolov commented Jul 27, 2018

I am not sure if this behaviour is a bug or a feature. On one hand it prevents a base application and a plugin from passing shared types to each other, but on the other hand it allows two plugins to vendor different versions of the same dependency and still be loaded. Without this behaviour, the plugin loader will complain that the second plugin was built with a different version of the shared dependency.

The issue with a shared type being returned by a plugin function to the base could be worked around by either pulling that specific dependency out of vendor/ into the GOPATH or by using a combination of interfaces, type assertion and reflection to convert the two types.

Also should the discussion on this issue happen here or in #20481?

@ash2k
Copy link
Contributor

ash2k commented Jul 27, 2018

I tried using plugins a while ago and gave up. I had 2+ plugins (and the main application too) that use kubernetes libraries and kubernetes uses glog. So, 2+ glog copies, each trying to add a flag to the flag package. A flag cannot be added more than once -> glog/stdlib panics. Plugins are completely unuseable in such situation.
I think shared mutable state should be removed from standard library in Go 2.

@e-nikolov
Copy link

e-nikolov commented Jul 27, 2018

A similar thing happens for plugins that import x/net/trace which in its init() adds some http handlers to the http.DefaultServeMux. So if you load 2 plugins that have vendored x/net/trace, the init() will panic because it gets executed twice and you can't have two handlers registered at the same route.

The only way to "resolve" this right now seems to be to reassign http.DefaultServeMux before loading each plugin.

@graup
Copy link

graup commented Nov 27, 2018

Are there any updates on this? I agree with the commenters above that this renders plugins useless for a lot of use cases. There should at least be a warning at the top of the documentation of the plugins package that when working with vendored code you cannot share types between different packages.

@juhwany
Copy link

juhwany commented Sep 9, 2019

Is there any progress? It's still very difficult to use plugins

@edaniels
Copy link
Contributor

edaniels commented Sep 9, 2019

It seems like plugins are effectively dead, support wise. They've received little mention since their release.

@huxiangdong
Copy link

then are there any other alternative we can leverage?

@seankhliao
Copy link
Member

Is there anything to do here?
With modules, all dependencies have a singular identity, and workspaces makes it relatively easy to keep dependency versions in sync across modules

@seankhliao seankhliao added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Jun 15, 2022
@gopherbot gopherbot added the compiler/runtime Issues related to the Go compiler and/or runtime. label Jul 7, 2022
@gopherbot
Copy link
Contributor

Timed out in state WaitingForInfo. Closing.

(I am just a bot, though. Please speak up if this is a mistake or you have the requested information.)

@Fire-Dragon-DoL
Copy link

Fire-Dragon-DoL commented Jan 20, 2023

This is not resolved, it's still a problem. Effectively Go can't be used to build something similar to Wordpress which loads plugins from a marketplace, even though it would be amazing

@svengreb
Copy link

@Fire-Dragon-DoL Responding in a closed issue might not help and this topic should only be revisited when there are new ideas or information.
Anyway, you can build plugins, but not via the standard library. Like in many languages this use case is something the community should resolve and many larger projects did so. The best example is Traefik that leverages Yaegi, one of the most advanced Go interpreters, that was also built by the Traefik team. Traefik‘s plugin system uses Yaegi to execute code on the fly rather than trying to rely on pre-compiled and linked plugins that were built with Go‘s plugin package. This is a perfect and elegant solution with full flexibility and does not require official SDK support at all. The Traefik GitHub organization has many “official“ plugins as well as demo plugins like traefik/pluginproviderdemo and traefik/plugindemo which you can use to get an idea and some inspiration how it works. Also check out Yaegi before to roughly understand how plugins work under-the-hood.

@Fire-Dragon-DoL
Copy link

Fire-Dragon-DoL commented Jan 21, 2023

@Fire-Dragon-DoL Responding in a closed issue might not help and this topic should only be revisited when there are new ideas or information.
Anyway, you can build plugins, but not via the standard library. Like in many languages this use case is something the community should resolve and many larger projects did so. The best example is Traefik that leverages Yaegi, one of the most advanced Go interpreters, that was also built by the Traefik team. Traefik‘s plugin system uses Yaegi to execute code on the fly rather than trying to rely on pre-compiled and linked plugins that were built with Go‘s plugin package. This is a perfect and elegant solution with full flexibility and does not require official SDK support at all. The Traefik GitHub organization has many “official“ plugins as well as demo plugins like traefik/pluginproviderdemo and traefik/plugindemo which you can use to get an idea and some inspiration how it works. Also check out Yaegi before to roughly understand how plugins work under-the-hood.

Thank you, I was not aware of Yaegi, it can definitely help the problem!

It seems like there is still a big performance penalty, sadly, but there might be no other solutions.

Edit: there seems to be some serious limitations to yaegi to importable packages unfortunately.

@yurishkuro
Copy link

@Fire-Dragon-DoL have you looked into using WASM?

@edaniels
Copy link
Contributor

It seems like at this point the feature may as well be removed to avoid confusion.

@golang golang locked and limited conversation to collaborators Jan 21, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. FrozenDueToAge help wanted NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Projects
None yet
Development

No branches or pull requests