New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

plugin: add Windows support #19282

Open
QuestionPython opened this Issue Feb 24, 2017 · 22 comments

Comments

Projects
None yet
8 participants
@QuestionPython

QuestionPython commented Feb 24, 2017

hi
Plugin Pkg Work for Windows!?
i want use this for mac,linux,win,... os.
when(what time) fix this?

https://golang.org/pkg/plugin/

@bradfitz bradfitz changed the title from Plugin Pkg for Windows! to plugin: add Windows support Feb 24, 2017

@bradfitz bradfitz added the OS-Windows label Feb 24, 2017

@bradfitz bradfitz added this to the Unplanned milestone Feb 24, 2017

@bradfitz

This comment has been minimized.

Show comment
Hide comment
@bradfitz

bradfitz Feb 24, 2017

Member

There is currently nobody working on it, as far as I know.

/cc @alexbrainman @crawshaw

Member

bradfitz commented Feb 24, 2017

There is currently nobody working on it, as far as I know.

/cc @alexbrainman @crawshaw

@QuestionPython

This comment has been minimized.

Show comment
Hide comment
@QuestionPython

QuestionPython Feb 24, 2017

mean in go 1.8 , plugin pkg work for apple-mac,windows and more ?

QuestionPython commented Feb 24, 2017

mean in go 1.8 , plugin pkg work for apple-mac,windows and more ?

@bradfitz

This comment has been minimized.

Show comment
Hide comment
@bradfitz

bradfitz Feb 24, 2017

Member

@QuestionPython, yes, it's even documented in multiple places:

https://golang.org/pkg/plugin/

Currently plugins only work on Linux.

https://golang.org/doc/go1.8#plugin

Plugin support is currently only available on Linux.

Member

bradfitz commented Feb 24, 2017

@QuestionPython, yes, it's even documented in multiple places:

https://golang.org/pkg/plugin/

Currently plugins only work on Linux.

https://golang.org/doc/go1.8#plugin

Plugin support is currently only available on Linux.

@alexbrainman

This comment has been minimized.

Show comment
Hide comment
@alexbrainman

alexbrainman Feb 24, 2017

Member

There is currently nobody working on it, as far as I know.

I am not working on it. Sorry.

Alex

Member

alexbrainman commented Feb 24, 2017

There is currently nobody working on it, as far as I know.

I am not working on it. Sorry.

Alex

@bradfitz

This comment has been minimized.

Show comment
Hide comment
@bradfitz

bradfitz Feb 26, 2017

Member

We delete all "me too" voting comments per https://golang.org/wiki/NoMeToo. Vote with emoji reactions at top instead.

Member

bradfitz commented Feb 26, 2017

We delete all "me too" voting comments per https://golang.org/wiki/NoMeToo. Vote with emoji reactions at top instead.

@golang golang locked and limited conversation to collaborators Mar 3, 2017

@bradfitz

This comment has been minimized.

Show comment
Hide comment
@bradfitz

bradfitz Mar 3, 2017

Member

The "me too" comments wouldn't stop, so this issue is now locked. If there are updates, they will be made here.

Member

bradfitz commented Mar 3, 2017

The "me too" comments wouldn't stop, so this issue is now locked. If there are updates, they will be made here.

@0xdevalias

This comment has been minimized.

Show comment
Hide comment
@0xdevalias

0xdevalias Apr 20, 2018

Out of curiosity, do we know how much effort it would take to implement windows support? Or if there are any blockers to it (and what they are?)

0xdevalias commented Apr 20, 2018

Out of curiosity, do we know how much effort it would take to implement windows support? Or if there are any blockers to it (and what they are?)

@akavel

This comment has been minimized.

Show comment
Hide comment
@akavel

akavel Apr 23, 2018

Contributor

Notably, the recently published Go kernel for Jupyter notebooks is using buildmode=shared, and thus doesn't currently support Windows natively. This is a very cool use case, adding a REPL-like live coding feature to the Go ecosystem, thus it would be really awesome if someone tried to start work on buildmode=shared on Windows to support this use case.

Similar to @0xdevalias , I'm quite interested in some hints as to what is missing for this to work on Windows? I'm especially curious what extra work is needed given that c-shared is already implemented on Windows?

Contributor

akavel commented Apr 23, 2018

Notably, the recently published Go kernel for Jupyter notebooks is using buildmode=shared, and thus doesn't currently support Windows natively. This is a very cool use case, adding a REPL-like live coding feature to the Go ecosystem, thus it would be really awesome if someone tried to start work on buildmode=shared on Windows to support this use case.

Similar to @0xdevalias , I'm quite interested in some hints as to what is missing for this to work on Windows? I'm especially curious what extra work is needed given that c-shared is already implemented on Windows?

@alexbrainman

This comment has been minimized.

Show comment
Hide comment
@alexbrainman

alexbrainman Apr 23, 2018

Member

@0xdevalias and @akavel I don't have any effort estimation or any hints as to what missing here. I have not actually looked at what is involved. I am so much behind at fixing old issues ...

Alex

Member

alexbrainman commented Apr 23, 2018

@0xdevalias and @akavel I don't have any effort estimation or any hints as to what missing here. I have not actually looked at what is involved. I am so much behind at fixing old issues ...

Alex

@akavel

This comment has been minimized.

Show comment
Hide comment
@akavel

akavel Apr 23, 2018

Contributor

@alexbrainman Thanks! I'll ask on golang-dev then, maybe someone else can shed some light (edit: link to the thread)

Contributor

akavel commented Apr 23, 2018

@alexbrainman Thanks! I'll ask on golang-dev then, maybe someone else can shed some light (edit: link to the thread)

@0xdevalias

This comment has been minimized.

Show comment
Hide comment
@0xdevalias

0xdevalias Apr 23, 2018

There doesn't seem to be a huge amount to it in the src: https://github.com/golang/go/tree/master/src/plugin

My completely naive guess would be figuring the windows equivalents to the C-bindings in plugin_dlopen.go.

The main functions I can see there are:

  • https://linux.die.net/man/3/dlopen : The function dlopen() loads the dynamic library file named by the null-terminated string filename and returns an opaque "handle" for the dynamic library.
  • https://linux.die.net/man/3/dlsym : The function dlsym() takes a "handle" of a dynamic library returned by dlopen() and the null-terminated symbol name, returning the address where that symbol is loaded into memory.

Googling for "dlopen equivalent windows" led me to the following:

And "dlsym equivalent windows":

So from that, it sounds like we have the following premise to work from:

  • dlopen in *nix roughly maps to LoadLibrary in windows
  • dlsym in *nix roughly maps to GetProcAddress in windows

The main definitions in FlexDLL don't look too complex.. but there is quite a bit of extra code around those that may be required too:

Hopefully this helps scope out what will be required/start pointing in the right direction :)

0xdevalias commented Apr 23, 2018

There doesn't seem to be a huge amount to it in the src: https://github.com/golang/go/tree/master/src/plugin

My completely naive guess would be figuring the windows equivalents to the C-bindings in plugin_dlopen.go.

The main functions I can see there are:

  • https://linux.die.net/man/3/dlopen : The function dlopen() loads the dynamic library file named by the null-terminated string filename and returns an opaque "handle" for the dynamic library.
  • https://linux.die.net/man/3/dlsym : The function dlsym() takes a "handle" of a dynamic library returned by dlopen() and the null-terminated symbol name, returning the address where that symbol is loaded into memory.

Googling for "dlopen equivalent windows" led me to the following:

And "dlsym equivalent windows":

So from that, it sounds like we have the following premise to work from:

  • dlopen in *nix roughly maps to LoadLibrary in windows
  • dlsym in *nix roughly maps to GetProcAddress in windows

The main definitions in FlexDLL don't look too complex.. but there is quite a bit of extra code around those that may be required too:

Hopefully this helps scope out what will be required/start pointing in the right direction :)

@jclc

This comment has been minimized.

Show comment
Hide comment
@jclc

jclc Aug 10, 2018

All the posts seem to be concerned with the loading of symbols, but does the compiler support producing a plugin (presumably DLL) on Windows?

jclc commented Aug 10, 2018

All the posts seem to be concerned with the loading of symbols, but does the compiler support producing a plugin (presumably DLL) on Windows?

@alexbrainman

This comment has been minimized.

Show comment
Hide comment
@alexbrainman

alexbrainman Aug 11, 2018

Member

does the compiler support producing a plugin (presumably DLL) on Windows?

It is possible to build Windows DLL from your Go code. You want -buildmode=c-shared 'go build' flag for that. See #26714 for an example. 'go build' command uses gcc under covers to build DLL.

Alex

Member

alexbrainman commented Aug 11, 2018

does the compiler support producing a plugin (presumably DLL) on Windows?

It is possible to build Windows DLL from your Go code. You want -buildmode=c-shared 'go build' flag for that. See #26714 for an example. 'go build' command uses gcc under covers to build DLL.

Alex

@jclc

This comment has been minimized.

Show comment
Hide comment
@jclc

jclc Aug 12, 2018

I've been hacking on this issue for a while and it seems to be going well for now. I've managed to load a dll built with -buildmode=c-shared and call its init function. The only limitation of this is that the plugin needs to have a main function or it won't compile. I'm developing on Linux using GCC and Wine. Just a few questions if anyone could clarify:

What exactly is going on in this function? The dlopen implementation calls this function and apparently returns the symbols; it doesn't work with Windows's shared objects.
https://github.com/golang/go/blob/master/src/runtime/plugin.go#L10

Secondly, I couldn't find any consistent guidelines for using wide strings with CGO so I ended up depending on unicode/utf16 and unicode/utf8. However, go/build/deps_test.go has pretty strict restrictions on which packages you can import. Is this a problem?

Edit: I guess this isn't so straightforward as I thought. -buildmode=plugin adds some metadata that is needed to find the exported symbols. Reading the PE data (using pev's readpe) doesn't show any of the functions that the plugin is meant to export, only init and main. When loading it, the init function is run implicitly.

jclc commented Aug 12, 2018

I've been hacking on this issue for a while and it seems to be going well for now. I've managed to load a dll built with -buildmode=c-shared and call its init function. The only limitation of this is that the plugin needs to have a main function or it won't compile. I'm developing on Linux using GCC and Wine. Just a few questions if anyone could clarify:

What exactly is going on in this function? The dlopen implementation calls this function and apparently returns the symbols; it doesn't work with Windows's shared objects.
https://github.com/golang/go/blob/master/src/runtime/plugin.go#L10

Secondly, I couldn't find any consistent guidelines for using wide strings with CGO so I ended up depending on unicode/utf16 and unicode/utf8. However, go/build/deps_test.go has pretty strict restrictions on which packages you can import. Is this a problem?

Edit: I guess this isn't so straightforward as I thought. -buildmode=plugin adds some metadata that is needed to find the exported symbols. Reading the PE data (using pev's readpe) doesn't show any of the functions that the plugin is meant to export, only init and main. When loading it, the init function is run implicitly.

@bradfitz

This comment has been minimized.

Show comment
Hide comment
@bradfitz

bradfitz Aug 13, 2018

Member

@jclc, deps_test.go is largely there to force us to think about dependencies and decide whether they're acceptable. An edge from plugin to unicode/utf* seems fine, especially since syscall can already use those, and plugin => to syscall is already permitted.

Member

bradfitz commented Aug 13, 2018

@jclc, deps_test.go is largely there to force us to think about dependencies and decide whether they're acceptable. An edge from plugin to unicode/utf* seems fine, especially since syscall can already use those, and plugin => to syscall is already permitted.

@heaths

This comment has been minimized.

Show comment
Hide comment
@heaths

heaths Aug 29, 2018

@jclc several toolchains will simply add a DllMain for you automatically. I recommend a similar model here where the compiler (with modifications as necessary) does that automatically. There should be no need for a plugin developer to define this function or make modifications to it (it's pretty rare to even in production).

As this is a blocking issue to something I'm trying to get into Moby, I'd be willing to help if you need it.

heaths commented Aug 29, 2018

@jclc several toolchains will simply add a DllMain for you automatically. I recommend a similar model here where the compiler (with modifications as necessary) does that automatically. There should be no need for a plugin developer to define this function or make modifications to it (it's pretty rare to even in production).

As this is a blocking issue to something I'm trying to get into Moby, I'd be willing to help if you need it.

@ImVexed

This comment has been minimized.

Show comment
Hide comment
@ImVexed

ImVexed Sep 9, 2018

I've been hacking on this issue as well as far as getting https://github.com/golang/go/blob/master/src/plugin/plugin_dlopen.go functioning on windows. As @0xdevalias gave insight into, LoadLibrary & GetProcAddress are the two equivalent we're looking for.

As far as @heaths suggestion to automatically generate a DllMain, I think this may be our best option. While we could use something similar to how https://github.com/alainfrisch/flexdll/blob/master/flexdll.c#L83-L117 does it by using either DONT_RESOLVE_DLL_REFERENCES or LOAD_LIBRARY_AS_DATAFILE in LoadLibraryEx I think it may just cause more work as on top of not invoking DllMain it also does not load any references to dependencies and would be quite tedious to implement manually.

ImVexed commented Sep 9, 2018

I've been hacking on this issue as well as far as getting https://github.com/golang/go/blob/master/src/plugin/plugin_dlopen.go functioning on windows. As @0xdevalias gave insight into, LoadLibrary & GetProcAddress are the two equivalent we're looking for.

As far as @heaths suggestion to automatically generate a DllMain, I think this may be our best option. While we could use something similar to how https://github.com/alainfrisch/flexdll/blob/master/flexdll.c#L83-L117 does it by using either DONT_RESOLVE_DLL_REFERENCES or LOAD_LIBRARY_AS_DATAFILE in LoadLibraryEx I think it may just cause more work as on top of not invoking DllMain it also does not load any references to dependencies and would be quite tedious to implement manually.

@jclc

This comment has been minimized.

Show comment
Hide comment
@jclc

jclc Sep 9, 2018

There shouldn't be a need to use cgo, since package syscall has functions for loading symbols on Windows, NewLazyDLL() and (*LazyDLL).Handle(). I missed these initially since they don't show up on godoc. There's also helper functions for Windows wide strings.

jclc commented Sep 9, 2018

There shouldn't be a need to use cgo, since package syscall has functions for loading symbols on Windows, NewLazyDLL() and (*LazyDLL).Handle(). I missed these initially since they don't show up on godoc. There's also helper functions for Windows wide strings.

@heaths

This comment has been minimized.

Show comment
Hide comment
@heaths

heaths Sep 9, 2018

@ImVexed LOAD_LIBRARY_AS_DATAFILE also doesn't do other things like rebase addresses as necessary. As the documentation reads,

Nothing is done to execute or prepare to execute the mapped file. Therefore, you cannot call functions like GetModuleFileName, GetModuleHandle or GetProcAddress with this DLL.

heaths commented Sep 9, 2018

@ImVexed LOAD_LIBRARY_AS_DATAFILE also doesn't do other things like rebase addresses as necessary. As the documentation reads,

Nothing is done to execute or prepare to execute the mapped file. Therefore, you cannot call functions like GetModuleFileName, GetModuleHandle or GetProcAddress with this DLL.

@ImVexed

This comment has been minimized.

Show comment
Hide comment
@ImVexed

ImVexed Sep 10, 2018

@jclc Interesting, do we know if there are any reasons as to why NewLazyDLL isn't used currently for plugins?

ImVexed commented Sep 10, 2018

@jclc Interesting, do we know if there are any reasons as to why NewLazyDLL isn't used currently for plugins?

@jclc

This comment has been minimized.

Show comment
Hide comment
@jclc

jclc Sep 10, 2018

I'm not an expert, but I believe c-shared Go libraries ship their own Go runtime along with garbage collection. I don't think you'd want that if you're loading it for use in a Go program. Also, see my earlier post. plugin_dlopen calls https://github.com/golang/go/blob/master/src/runtime/plugin.go#L10 to hook up with the module somehow.

jclc commented Sep 10, 2018

I'm not an expert, but I believe c-shared Go libraries ship their own Go runtime along with garbage collection. I don't think you'd want that if you're loading it for use in a Go program. Also, see my earlier post. plugin_dlopen calls https://github.com/golang/go/blob/master/src/runtime/plugin.go#L10 to hook up with the module somehow.

@alexbrainman

This comment has been minimized.

Show comment
Hide comment
@alexbrainman

alexbrainman Sep 10, 2018

Member

do we know if there are any reasons as to why NewLazyDLL isn't used currently for plugins?

Just a guess. Plugins where developed by Linux and Darwin developers. There is no syscall.NewLazyDLL on Linux or Darwin.

Alex

Member

alexbrainman commented Sep 10, 2018

do we know if there are any reasons as to why NewLazyDLL isn't used currently for plugins?

Just a guess. Plugins where developed by Linux and Darwin developers. There is no syscall.NewLazyDLL on Linux or Darwin.

Alex

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment