plugin: add Windows support

[QuestionPython]

hi
Plugin Pkg Work for Windows!?
i want use this for mac,linux,win,... os.
when(what time) fix this?

https://golang.org/pkg/plugin/

[bradfitz]

There is currently nobody working on it, as far as I know.

/cc @alexbrainman @crawshaw

[bradfitz]

There is currently nobody working on it, as far as I know.

/cc @alexbrainman @crawshaw

[QuestionPython]

mean in go 1.8 , plugin pkg work for apple-mac,windows and more ?

[QuestionPython]

mean in go 1.8 , plugin pkg work for apple-mac,windows and more ?

[bradfitz]

@QuestionPython, yes, it's even documented in multiple places:

https://golang.org/pkg/plugin/

Currently plugins only work on Linux.

https://golang.org/doc/go1.8#plugin

Plugin support is currently only available on Linux.

[bradfitz]

@QuestionPython, yes, it's even documented in multiple places:

https://golang.org/pkg/plugin/

Currently plugins only work on Linux.

https://golang.org/doc/go1.8#plugin

Plugin support is currently only available on Linux.

[alexbrainman]

There is currently nobody working on it, as far as I know.

I am not working on it. Sorry.

Alex

[alexbrainman]

There is currently nobody working on it, as far as I know.

I am not working on it. Sorry.

Alex

[bradfitz]

We delete all "me too" voting comments per https://golang.org/wiki/NoMeToo. Vote with emoji reactions at top instead.

[bradfitz]

We delete all "me too" voting comments per https://golang.org/wiki/NoMeToo. Vote with emoji reactions at top instead.

[bradfitz]

The "me too" comments wouldn't stop, so this issue is now locked. If there are updates, they will be made here.

[bradfitz]

The "me too" comments wouldn't stop, so this issue is now locked. If there are updates, they will be made here.

[0xdevalias]

Out of curiosity, do we know how much effort it would take to implement windows support? Or if there are any blockers to it (and what they are?)

[0xdevalias]

Out of curiosity, do we know how much effort it would take to implement windows support? Or if there are any blockers to it (and what they are?)

[akavel]

Notably, the recently published Go kernel for Jupyter notebooks is using buildmode=shared, and thus doesn't currently support Windows natively. This is a very cool use case, adding a REPL-like live coding feature to the Go ecosystem, thus it would be really awesome if someone tried to start work on buildmode=shared on Windows to support this use case.

Similar to @0xdevalias , I'm quite interested in some hints as to what is missing for this to work on Windows? I'm especially curious what extra work is needed given that c-shared is already implemented on Windows?

[akavel]

Notably, the recently published Go kernel for Jupyter notebooks is using buildmode=shared, and thus doesn't currently support Windows natively. This is a very cool use case, adding a REPL-like live coding feature to the Go ecosystem, thus it would be really awesome if someone tried to start work on buildmode=shared on Windows to support this use case.

Similar to @0xdevalias , I'm quite interested in some hints as to what is missing for this to work on Windows? I'm especially curious what extra work is needed given that c-shared is already implemented on Windows?

[alexbrainman]

@0xdevalias and @akavel I don't have any effort estimation or any hints as to what missing here. I have not actually looked at what is involved. I am so much behind at fixing old issues ...

Alex

[alexbrainman]

@0xdevalias and @akavel I don't have any effort estimation or any hints as to what missing here. I have not actually looked at what is involved. I am so much behind at fixing old issues ...

Alex

[akavel]

@alexbrainman Thanks! I'll ask on golang-dev then, maybe someone else can shed some light (edit: link to the thread)

[akavel]

@alexbrainman Thanks! I'll ask on golang-dev then, maybe someone else can shed some light (edit: link to the thread)

[0xdevalias]

There doesn't seem to be a huge amount to it in the src: https://github.com/golang/go/tree/master/src/plugin

My completely naive guess would be figuring the windows equivalents to the C-bindings in plugin_dlopen.go.

The main functions I can see there are:

• https://linux.die.net/man/3/dlopen : The function dlopen() loads the dynamic library file named by the null-terminated string filename and returns an opaque "handle" for the dynamic library.
• https://linux.die.net/man/3/dlsym : The function dlsym() takes a "handle" of a dynamic library returned by dlopen() and the null-terminated symbol name, returning the address where that symbol is loaded into memory.

Googling for "dlopen equivalent windows" led me to the following:

• https://github.com/alainfrisch/flexdll : an implementation of a dlopen-like API for Windows
  • "FlexDLL implements mostly the usual dlopen POSIX API, without trying to be fully conformant though (e.g. it does not respect the official priority ordering for symbol resolution). This should make it easy to port applications developped for Unix."
• LoadLibrary: https://msdn.microsoft.com/en-us/library/windows/desktop/ms684175(v=vs.85).aspx

And "dlsym equivalent windows":

• https://stackoverflow.com/questions/4184017/dlopen-dlsym-on-the-main-executable-how-portable-is-it
  • "Windows uses LoadLibrary() and GetProcAddress() instead."
• GetProcAddress: https://msdn.microsoft.com/en-us/library/windows/desktop/ms683212(v=vs.85).aspx
• https://www.symantec.com/connect/articles/dynamic-linking-linux-and-windows-part-two
  • "Both Linux and Windows provide routines (such as dlopen() and dlsym() in Linux, and LoadLibrary() ,GetProcAddress() in Windows)"
• https://www.experts-exchange.com/questions/22762973/Want-to-create-a-wrapper-class-for-dlopen-dlsym-dlclose-to-work-on-both-UNIX-and-windows-platform.html
  • "On windows, I believe LoadLibrary() == dlopen(), GetProcAddress() == dlsym(), and FreeLibrary() == dlclose()"

So from that, it sounds like we have the following premise to work from:

• dlopen in *nix roughly maps to LoadLibrary in windows
• dlsym in *nix roughly maps to GetProcAddress in windows

The main definitions in FlexDLL don't look too complex.. but there is quite a bit of extra code around those that may be required too:

• https://github.com/alainfrisch/flexdll/blob/master/flexdll.c#L83-L117

Hopefully this helps scope out what will be required/start pointing in the right direction :)

[0xdevalias]

There doesn't seem to be a huge amount to it in the src: https://github.com/golang/go/tree/master/src/plugin

My completely naive guess would be figuring the windows equivalents to the C-bindings in plugin_dlopen.go.

The main functions I can see there are:

• https://linux.die.net/man/3/dlopen : The function dlopen() loads the dynamic library file named by the null-terminated string filename and returns an opaque "handle" for the dynamic library.
• https://linux.die.net/man/3/dlsym : The function dlsym() takes a "handle" of a dynamic library returned by dlopen() and the null-terminated symbol name, returning the address where that symbol is loaded into memory.

Googling for "dlopen equivalent windows" led me to the following:

• https://github.com/alainfrisch/flexdll : an implementation of a dlopen-like API for Windows
  • "FlexDLL implements mostly the usual dlopen POSIX API, without trying to be fully conformant though (e.g. it does not respect the official priority ordering for symbol resolution). This should make it easy to port applications developped for Unix."
• LoadLibrary: https://msdn.microsoft.com/en-us/library/windows/desktop/ms684175(v=vs.85).aspx

And "dlsym equivalent windows":

• https://stackoverflow.com/questions/4184017/dlopen-dlsym-on-the-main-executable-how-portable-is-it
  • "Windows uses LoadLibrary() and GetProcAddress() instead."
• GetProcAddress: https://msdn.microsoft.com/en-us/library/windows/desktop/ms683212(v=vs.85).aspx
• https://www.symantec.com/connect/articles/dynamic-linking-linux-and-windows-part-two
  • "Both Linux and Windows provide routines (such as dlopen() and dlsym() in Linux, and LoadLibrary() ,GetProcAddress() in Windows)"
• https://www.experts-exchange.com/questions/22762973/Want-to-create-a-wrapper-class-for-dlopen-dlsym-dlclose-to-work-on-both-UNIX-and-windows-platform.html
  • "On windows, I believe LoadLibrary() == dlopen(), GetProcAddress() == dlsym(), and FreeLibrary() == dlclose()"

So from that, it sounds like we have the following premise to work from:

• dlopen in *nix roughly maps to LoadLibrary in windows
• dlsym in *nix roughly maps to GetProcAddress in windows

The main definitions in FlexDLL don't look too complex.. but there is quite a bit of extra code around those that may be required too:

• https://github.com/alainfrisch/flexdll/blob/master/flexdll.c#L83-L117

Hopefully this helps scope out what will be required/start pointing in the right direction :)

[jclc]

All the posts seem to be concerned with the loading of symbols, but does the compiler support producing a plugin (presumably DLL) on Windows?

[jclc]

All the posts seem to be concerned with the loading of symbols, but does the compiler support producing a plugin (presumably DLL) on Windows?

[alexbrainman]

does the compiler support producing a plugin (presumably DLL) on Windows?

It is possible to build Windows DLL from your Go code. You want -buildmode=c-shared 'go build' flag for that. See #26714 for an example. 'go build' command uses gcc under covers to build DLL.

Alex

[alexbrainman]

does the compiler support producing a plugin (presumably DLL) on Windows?

It is possible to build Windows DLL from your Go code. You want -buildmode=c-shared 'go build' flag for that. See #26714 for an example. 'go build' command uses gcc under covers to build DLL.

Alex

[jclc]

I've been hacking on this issue for a while and it seems to be going well for now. I've managed to load a dll built with -buildmode=c-shared and call its init function. The only limitation of this is that the plugin needs to have a main function or it won't compile. I'm developing on Linux using GCC and Wine. Just a few questions if anyone could clarify:

What exactly is going on in this function? The dlopen implementation calls this function and apparently returns the symbols; it doesn't work with Windows's shared objects.
https://github.com/golang/go/blob/master/src/runtime/plugin.go#L10

Secondly, I couldn't find any consistent guidelines for using wide strings with CGO so I ended up depending on unicode/utf16 and unicode/utf8. However, go/build/deps_test.go has pretty strict restrictions on which packages you can import. Is this a problem?

Edit: I guess this isn't so straightforward as I thought. -buildmode=plugin adds some metadata that is needed to find the exported symbols. Reading the PE data (using pev's readpe) doesn't show any of the functions that the plugin is meant to export, only init and main. When loading it, the init function is run implicitly.

[jclc]

I've been hacking on this issue for a while and it seems to be going well for now. I've managed to load a dll built with -buildmode=c-shared and call its init function. The only limitation of this is that the plugin needs to have a main function or it won't compile. I'm developing on Linux using GCC and Wine. Just a few questions if anyone could clarify:

What exactly is going on in this function? The dlopen implementation calls this function and apparently returns the symbols; it doesn't work with Windows's shared objects.
https://github.com/golang/go/blob/master/src/runtime/plugin.go#L10

Secondly, I couldn't find any consistent guidelines for using wide strings with CGO so I ended up depending on unicode/utf16 and unicode/utf8. However, go/build/deps_test.go has pretty strict restrictions on which packages you can import. Is this a problem?

Edit: I guess this isn't so straightforward as I thought. -buildmode=plugin adds some metadata that is needed to find the exported symbols. Reading the PE data (using pev's readpe) doesn't show any of the functions that the plugin is meant to export, only init and main. When loading it, the init function is run implicitly.

[bradfitz]

@jclc, deps_test.go is largely there to force us to think about dependencies and decide whether they're acceptable. An edge from plugin to unicode/utf* seems fine, especially since syscall can already use those, and plugin => to syscall is already permitted.

[bradfitz]

@jclc, deps_test.go is largely there to force us to think about dependencies and decide whether they're acceptable. An edge from plugin to unicode/utf* seems fine, especially since syscall can already use those, and plugin => to syscall is already permitted.

[heaths]

@jclc several toolchains will simply add a DllMain for you automatically. I recommend a similar model here where the compiler (with modifications as necessary) does that automatically. There should be no need for a plugin developer to define this function or make modifications to it (it's pretty rare to even in production).

As this is a blocking issue to something I'm trying to get into Moby, I'd be willing to help if you need it.

[heaths]

@jclc several toolchains will simply add a DllMain for you automatically. I recommend a similar model here where the compiler (with modifications as necessary) does that automatically. There should be no need for a plugin developer to define this function or make modifications to it (it's pretty rare to even in production).

As this is a blocking issue to something I'm trying to get into Moby, I'd be willing to help if you need it.

[ImVexed]

I've been hacking on this issue as well as far as getting https://github.com/golang/go/blob/master/src/plugin/plugin_dlopen.go functioning on windows. As @0xdevalias gave insight into, LoadLibrary & GetProcAddress are the two equivalent we're looking for.

As far as @heaths suggestion to automatically generate a DllMain, I think this may be our best option. While we could use something similar to how https://github.com/alainfrisch/flexdll/blob/master/flexdll.c#L83-L117 does it by using either DONT_RESOLVE_DLL_REFERENCES or LOAD_LIBRARY_AS_DATAFILE in LoadLibraryEx I think it may just cause more work as on top of not invoking DllMain it also does not load any references to dependencies and would be quite tedious to implement manually.

[ImVexed]

I've been hacking on this issue as well as far as getting https://github.com/golang/go/blob/master/src/plugin/plugin_dlopen.go functioning on windows. As @0xdevalias gave insight into, LoadLibrary & GetProcAddress are the two equivalent we're looking for.

As far as @heaths suggestion to automatically generate a DllMain, I think this may be our best option. While we could use something similar to how https://github.com/alainfrisch/flexdll/blob/master/flexdll.c#L83-L117 does it by using either DONT_RESOLVE_DLL_REFERENCES or LOAD_LIBRARY_AS_DATAFILE in LoadLibraryEx I think it may just cause more work as on top of not invoking DllMain it also does not load any references to dependencies and would be quite tedious to implement manually.

[jclc]

There shouldn't be a need to use cgo, since package syscall has functions for loading symbols on Windows, NewLazyDLL() and (*LazyDLL).Handle(). I missed these initially since they don't show up on godoc. There's also helper functions for Windows wide strings.

[jclc]

There shouldn't be a need to use cgo, since package syscall has functions for loading symbols on Windows, NewLazyDLL() and (*LazyDLL).Handle(). I missed these initially since they don't show up on godoc. There's also helper functions for Windows wide strings.

[heaths]

@ImVexed LOAD_LIBRARY_AS_DATAFILE also doesn't do other things like rebase addresses as necessary. As the documentation reads,

Nothing is done to execute or prepare to execute the mapped file. Therefore, you cannot call functions like GetModuleFileName, GetModuleHandle or GetProcAddress with this DLL.

[heaths]

@ImVexed LOAD_LIBRARY_AS_DATAFILE also doesn't do other things like rebase addresses as necessary. As the documentation reads,

Nothing is done to execute or prepare to execute the mapped file. Therefore, you cannot call functions like GetModuleFileName, GetModuleHandle or GetProcAddress with this DLL.

[ImVexed]

@jclc Interesting, do we know if there are any reasons as to why NewLazyDLL isn't used currently for plugins?

[ImVexed]

@jclc Interesting, do we know if there are any reasons as to why NewLazyDLL isn't used currently for plugins?

[jclc]

I'm not an expert, but I believe c-shared Go libraries ship their own Go runtime along with garbage collection. I don't think you'd want that if you're loading it for use in a Go program. Also, see my earlier post. plugin_dlopen calls https://github.com/golang/go/blob/master/src/runtime/plugin.go#L10 to hook up with the module somehow.

[jclc]

I'm not an expert, but I believe c-shared Go libraries ship their own Go runtime along with garbage collection. I don't think you'd want that if you're loading it for use in a Go program. Also, see my earlier post. plugin_dlopen calls https://github.com/golang/go/blob/master/src/runtime/plugin.go#L10 to hook up with the module somehow.

[alexbrainman]

do we know if there are any reasons as to why NewLazyDLL isn't used currently for plugins?

Just a guess. Plugins where developed by Linux and Darwin developers. There is no syscall.NewLazyDLL on Linux or Darwin.

Alex

[alexbrainman]

do we know if there are any reasons as to why NewLazyDLL isn't used currently for plugins?

Just a guess. Plugins where developed by Linux and Darwin developers. There is no syscall.NewLazyDLL on Linux or Darwin.

Alex