plugin: add Windows support

[QuestionPython]

hi
Plugin Pkg Work for Windows!?
i want use this for mac,linux,win,... os.
when(what time) fix this?

https://golang.org/pkg/plugin/

[bradfitz]

There is currently nobody working on it, as far as I know.

/cc @alexbrainman @crawshaw

[QuestionPython]

mean in go 1.8 , plugin pkg work for apple-mac,windows and more ?

[bradfitz]

@QuestionPython, yes, it's even documented in multiple places:

https://golang.org/pkg/plugin/

Currently plugins only work on Linux.

https://golang.org/doc/go1.8#plugin

Plugin support is currently only available on Linux.

[alexbrainman]

There is currently nobody working on it, as far as I know.

I am not working on it. Sorry.

Alex

[bradfitz]

We delete all "me too" voting comments per https://golang.org/wiki/NoMeToo. Vote with emoji reactions at top instead.

[bradfitz]

The "me too" comments wouldn't stop, so this issue is now locked. If there are updates, they will be made here.

[0xdevalias]

Out of curiosity, do we know how much effort it would take to implement windows support? Or if there are any blockers to it (and what they are?)

[akavel]

Notably, the recently published Go kernel for Jupyter notebooks is using buildmode=shared, and thus doesn't currently support Windows natively. This is a very cool use case, adding a REPL-like live coding feature to the Go ecosystem, thus it would be really awesome if someone tried to start work on buildmode=shared on Windows to support this use case.

Similar to @0xdevalias , I'm quite interested in some hints as to what is missing for this to work on Windows? I'm especially curious what extra work is needed given that c-shared is already implemented on Windows?

[alexbrainman]

@0xdevalias and @akavel I don't have any effort estimation or any hints as to what missing here. I have not actually looked at what is involved. I am so much behind at fixing old issues ...

Alex

[akavel]

@alexbrainman Thanks! I'll ask on golang-dev then, maybe someone else can shed some light (edit: link to the thread)

[0xdevalias]

There doesn't seem to be a huge amount to it in the src: https://github.com/golang/go/tree/master/src/plugin

My completely naive guess would be figuring the windows equivalents to the C-bindings in plugin_dlopen.go.

The main functions I can see there are:

• https://linux.die.net/man/3/dlopen : The function dlopen() loads the dynamic library file named by the null-terminated string filename and returns an opaque "handle" for the dynamic library.
• https://linux.die.net/man/3/dlsym : The function dlsym() takes a "handle" of a dynamic library returned by dlopen() and the null-terminated symbol name, returning the address where that symbol is loaded into memory.

Googling for "dlopen equivalent windows" led me to the following:

• https://github.com/alainfrisch/flexdll : an implementation of a dlopen-like API for Windows
  • "FlexDLL implements mostly the usual dlopen POSIX API, without trying to be fully conformant though (e.g. it does not respect the official priority ordering for symbol resolution). This should make it easy to port applications developped for Unix."
• LoadLibrary: https://msdn.microsoft.com/en-us/library/windows/desktop/ms684175(v=vs.85).aspx

And "dlsym equivalent windows":

• https://stackoverflow.com/questions/4184017/dlopen-dlsym-on-the-main-executable-how-portable-is-it
  • "Windows uses LoadLibrary() and GetProcAddress() instead."
• GetProcAddress: https://msdn.microsoft.com/en-us/library/windows/desktop/ms683212(v=vs.85).aspx
• https://www.symantec.com/connect/articles/dynamic-linking-linux-and-windows-part-two
  • "Both Linux and Windows provide routines (such as dlopen() and dlsym() in Linux, and LoadLibrary() ,GetProcAddress() in Windows)"
• https://www.experts-exchange.com/questions/22762973/Want-to-create-a-wrapper-class-for-dlopen-dlsym-dlclose-to-work-on-both-UNIX-and-windows-platform.html
  • "On windows, I believe LoadLibrary() == dlopen(), GetProcAddress() == dlsym(), and FreeLibrary() == dlclose()"

So from that, it sounds like we have the following premise to work from:

• dlopen in *nix roughly maps to LoadLibrary in windows
• dlsym in *nix roughly maps to GetProcAddress in windows

The main definitions in FlexDLL don't look too complex.. but there is quite a bit of extra code around those that may be required too:

• https://github.com/alainfrisch/flexdll/blob/master/flexdll.c#L83-L117

Hopefully this helps scope out what will be required/start pointing in the right direction :)

[jclc]

All the posts seem to be concerned with the loading of symbols, but does the compiler support producing a plugin (presumably DLL) on Windows?

[alexbrainman]

does the compiler support producing a plugin (presumably DLL) on Windows?

It is possible to build Windows DLL from your Go code. You want -buildmode=c-shared 'go build' flag for that. See #26714 for an example. 'go build' command uses gcc under covers to build DLL.

Alex

[jclc]

I've been hacking on this issue for a while and it seems to be going well for now. I've managed to load a dll built with -buildmode=c-shared and call its init function. The only limitation of this is that the plugin needs to have a main function or it won't compile. I'm developing on Linux using GCC and Wine. Just a few questions if anyone could clarify:

What exactly is going on in this function? The dlopen implementation calls this function and apparently returns the symbols; it doesn't work with Windows's shared objects.
https://github.com/golang/go/blob/master/src/runtime/plugin.go#L10

Secondly, I couldn't find any consistent guidelines for using wide strings with CGO so I ended up depending on unicode/utf16 and unicode/utf8. However, go/build/deps_test.go has pretty strict restrictions on which packages you can import. Is this a problem?

Edit: I guess this isn't so straightforward as I thought. -buildmode=plugin adds some metadata that is needed to find the exported symbols. Reading the PE data (using pev's readpe) doesn't show any of the functions that the plugin is meant to export, only init and main. When loading it, the init function is run implicitly.

[alexbrainman]

There are some symbols local.moduledata that are supposed to end up in .noptrdata. At somepoint (I think maybe gcc is doing it, but it could be pe.go) the stuff is supposed to be in .noptrdata is getting stuffed into .data

Are you talking about pe sections named .noptrdata and .data ? Because I do not remember that Go linker generates .noptrdata pe section in the object file to be used for external linker. As far as I remember whatever symbol named as .noptrdata endup in either .text, .rdata or .data pe section. But, regardless, relocations should be correct, because they apply to the whole pe section. Maybe Go linker does not align some symbols or sections properly.

It is hard for me to advise here. If you have some repro for me to play with, I might be able to help.

Alex

[cchamplin]

As far as I remember whatever symbol named as .noptrdata endup in either .text, .rdata or .data pe section.

Okay that's good to know, I wasn't sure if it was gcc doing it or the go linker

It is hard for me to advise here. If you have some repro for me to play with, I might be able to help.

Understood, let me see if I get pull out a minimum set of changes needed to get underway and get them in a repo

[alexbrainman]

let me see if I get pull out a minimum set of changes needed to get underway and get them in a repo

SGTM. Thank you.

Alex

[cchamplin]

@alexbrainman
Branch with minimal set of changes can be found here https://github.com/cchamplin/go/tree/plugins_windows
Test Project Here
https://github.com/cchamplin/plugin_test

[cchamplin]

Okay some additional follow up, it actually looks like the output assembly and symbols are fine there may not be corruption, I think what I was seeing was slight differences in how the memory is represented in ELF vs PE, but doing more thorough checking of the memory at runtime things might be okay.

What appears to be happening is that in a functional example (*nix)
runtime.addmoduledata loads lastmoduledatap which I guess should have a pointer to firstmoduledata.

Whats happening on windows is that lastmoduledatap has a pointer to local.moduledata which is apparently not correct.

I'm not sure where lastmoduledatap is getting set to firstmoduledata as I can't seem to find it in code. The code I've found shows that it should have the address of local.moduledata. symtab.go:680

[mattn]

As far as I can see, most of C code in plugin_windows.go can be replaced to Go.

[cchamplin]

@mattn This is correct, it's only the way it is to try and get other parts working.

[alexbrainman]

What appears to be happening is that in a functional example (*nix)
runtime.addmoduledata loads lastmoduledatap which I guess should have a pointer to firstmoduledata.

I see about the same. I could not stop gdb in the DLL loading code - I do not know how to do it. I tried breakpoint and rwatch. But I used objdump -d, and I can see this

000000006fdf7f40 <runtime.addmoduledata>:
    6fdf7f40:	41 57                	push   %r15
    6fdf7f42:	4c 8b 3d af 21 00 00 	mov    0x21af(%rip),%r15        # 6fdfa0f8 <runtime.lastmoduledatap>
    6fdf7f49:	49 8b 07             	mov    (%r15),%rax
    6fdf7f4c:	48 89 b8 c0 01 00 00 	mov    %rdi,0x1c0(%rax)
    6fdf7f53:	4c 8b 3d 9e 21 00 00 	mov    0x219e(%rip),%r15        # 6fdfa0f8 <runtime.lastmoduledatap>
    6fdf7f5a:	49 89 3f             	mov    %rdi,(%r15)
    6fdf7f5d:	41 5f                	pop    %r15
    6fdf7f5f:	c3                   	retq   

000000006fdf7f60 <go.link.addmoduledata>:
    6fdf7f60:	48 8d 3d b9 71 00 00 	lea    0x71b9(%rip),%rdi        # 6fdff120 <local.moduledata>
    6fdf7f67:	e8 d4 ff ff ff       	callq  6fdf7f40 <runtime.addmoduledata>
    6fdf7f6c:	c3                   	retq   
    6fdf7f6d:	cc                   	int3   
    6fdf7f6e:	cc                   	int3   
    6fdf7f6f:	cc                   	int3   

but there is nothing at 0x6fdfa0f8 address in objdump output.

I'm not sure where lastmoduledatap is getting set to firstmoduledata as I can't seem to find it in code. The code I've found shows that it should have the address of local.moduledata. symtab.go:680

I am not familiar with this code. Maybe @ianlancetaylor can help.

But, regardless of what you achieve here, you still have a problem (see #22192 for details) where you cannot use Go DLL from Go executable. How you are going to overcome that?

Alex

[cchamplin]

Thanks @alexbrainman

So my suspicion has been that what we are seeing is there is a runtime.* in the plugin dll and a runtime.* in the host executable. Which would explain why the memory for lastmoduledatap is not correct from the perspective of the plugin. I had forgot about #22192 but that ticket also pretty much confirms that suspicion.

It looks like what we need to replicate is the -rdynamic gcc flag. Which I don't believe gcc on windows supports.

So where do we go from here... What I'm trying to work on is having the host executable (though really we just need the base go internals) export all of its symbols (-Wl,--export-all-symbols) then using those via dlltool to build a library file that plugins link against (this is where I am now). Once we're linking against an external library for the "shared" go runtime I'm hoping we'll see some forward movement. I haven't gotten it to work quiet yet and I suspect it's because I need to stuff all of the go runtime symbols into .idata (windows import table) vs where it normally lives (.data,.text) for the plugin dll. Right now the whole thing is a lot of manual process and it may require that the plugin host be externally linked on windows for plugins to work (not the end of the world, I suspect that when building a host executable we can check for usage of the plugin symbols and switch to external linking automatically)

My bigger concern is whether or not we can link against a general go runtime in the plugin vs linking against the go runtime provided by the plugin host executable. I suspect we can, I don't really see any reason this shouldn't work, but I fear the plugin host may also have to be linked against the same general runtime...unsure though.

[cchamplin]

I see about the same. I could not stop gdb in the DLL loading code - I do not know how to do it. I tried breakpoint and rwatch. But I used objdump -d, and I can see this

What I'm doing is:
b go.link.addmoduledata then si from there to step forward until I know where the segfault occurs. If this is not working you may try b link.addmoduledata which is what the symbol is showing up for me as on *nix. I'm not sure why there are different.

[alexbrainman]

So my suspicion has been that what we are seeing is there is a runtime.* in the plugin dll and a runtime.* in the host executable. Which would explain why the memory for lastmoduledatap is not correct from the perspective of the plugin. I had forgot about #22192 but that ticket also pretty much confirms that suspicion.

I am not sure what you are saying here. As far as I understand runtime.* variables in executable and dll do not share the same addresses - they are independent of each other. The problem described in #22192 is different - both executable and dll use the same TLS register ( #22192 (comment) ).

It looks like what we need to replicate is the -rdynamic gcc flag. Which I don't believe gcc on windows supports.

I am not familiar with gcc enough to comment. But from what I can gather, -rdynamic gcc flag provides support for dlopen. But all Windows DLLs can be loaded "dynamically" using LoadLibrary Windows API. So I am not sure we need -rdynamic gcc flag.

I haven't gotten it to work quiet yet and I suspect it's because I need to stuff all of the go runtime symbols into .idata (windows import table) vs where it normally lives (.data,.text) for the plugin dll.

.idata always contains very specific data - it is effectively relocations that Windows uses when it loads / initialises executable or DLL. .idata lists all DLL names / function names used by executable or DLL and some refs for Windows to write these function addresses during load.

I would not be surprised if existing Go DLLs already has all functions exported - so you could call them via LoadLibrary / GetProcAddress already. You just need to pass parameters correctly to them. And that is what I expect plugin mode achieves. But I could be wrong.

b go.link.addmoduledata then si from there to step forward until I know where the segfault occurs. If this is not working you may try b link.addmoduledata which is what the symbol is showing up for me as on *nix. I'm not sure why there are different.

This is what I do

c:\Users\Alex\dev\src\github.com\cchamplin\plugin_test>gdb plugin_test.exe
GNU gdb (GDB) 7.8
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-w64-mingw32".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from plugin_test.exe...done.
(gdb) b go.link.addmoduledata
Function "go.link.addmoduledata" not defined.
Make breakpoint pending on future shared library load? (y or [n]) y

Breakpoint 1 (go.link.addmoduledata) pending.
(gdb) b link.addmoduledata
Function "link.addmoduledata" not defined.
Make breakpoint pending on future shared library load? (y or [n]) y

Breakpoint 2 (link.addmoduledata) pending.
(gdb) r
Starting program: c:\Users\Alex\dev\src\github.com\cchamplin\plugin_test\plugin_test.exe
[New Thread 6200.0xe2c]
warning: Can not parse XML library list; XML support was disabled at compile time
[New Thread 6200.0x858]
[New Thread 6200.0x1f8c]
[New Thread 6200.0x25d8]
[New Thread 6200.0x420]
[New Thread 6200.0x2be4]
[New Thread 6200.0x24a0]
Starting
Module Path: c:\Users\Alex\dev\src\github.com\cchamplin\plugin_test\eng\eng.so

Program received signal SIGSEGV, Segmentation fault.
0x000000006fdf7f4c in ?? ()
(gdb) bt
#0  0x000000006fdf7f4c in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
(gdb) disas
No function contains program counter for selected frame.
(gdb)

I do not see how I can use si command. Should I break on 'main.main' first, and then use si? That would take forever.

And, like @mattn said, you should replace your C code with single syscall.LoadLibrary call. For the purpose of your debugging, it does the same thing.

Alex

[cchamplin]

I am not sure what you are saying here. As far as I understand runtime.* variables in executable and dll do not share the same addresses - they are independent of each other. The problem described in #22192 is different - both executable and dll use the same TLS register ( #22192 (comment) ).

This is not what I am seeing on *nix. Here is a GDB run-through that unless I am mistaken shows that at a minimum runtime.lastmoduledata and runtime.firstmoduledata are at the same address in both contexts https://gist.github.com/cchamplin/abb9468e9b89e94d14cc0e2c4bafdaaa

IN main.main()

(gdb) x 0x0080a300
0x80a300 <runtime.lastmoduledatap>:     **0x0080f460**
(gdb) x *0x0080a300
0x80f460 <runtime.firstmoduledata>:     **0x00575fc0**

IN plugin->runtime.addmoduledata

(gdb) info registers
rax            **0x80f460** 8451168
rbx            0x7ffffa089800   140737388255232
rcx            0x836560 8611168
rdx            0x7ffffffee108   140737488281864
rsi            0x7ffffffee0f8   140737488281848
rdi            0x7ffffa16f200   140737389195776
rbp            0x1      0x1
rsp            0x7ffffffeda78   0x7ffffffeda78
r8             0x7fffff7e09d8   140737479838168
r9             0x7fffff7e09d8   140737479838168
r10            0x0      0
r11            0x0      0
r12            0x7ffffffee0f8   140737488281848
r13            0x7ffffffee108   140737488281864
r14            0x7ffffa089808   140737388255240
r15            **0x80a300** 8430336
rip            0x7ffff9e700cc   0x7ffff9e700cc <runtime.addmoduledata+12>
(gdb) x 0x0080a300
0x80a300 <runtime.lastmoduledatap>:     **0x0080f460**
(gdb) x *0x0080a300
0x80f460 <runtime.firstmoduledata>:     **0x00575fc0**

If you look at the values of R15 and RAX you see the addresses match.

I am not familiar with gcc enough to comment. But from what I can gather, -rdynamic gcc flag provides support for dlopen. But all Windows DLLs can be loaded "dynamically" using LoadLibrary Windows API. So I am not sure we need -rdynamic gcc flag.

My understanding of rdyanmic is that it is exporting symbols from the host in such a way that the plugin is able to use those symbols. Again I could be mistaken but that's my understanding

-rdynamic
    Pass the flag -export-dynamic to the ELF linker, on targets that support it. This instructs the
    linker to add all symbols, not only used ones, to the dynamic symbol table. This option is needed for
    some uses of "dlopen" or to allow obtaining backtraces from within a program.

.idata always contains very specific data - it is effectively relocations that Windows uses when it loads / initialises executable or DLL. .idata lists all DLL names / function names used by executable or DLL and some refs for Windows to write these function addresses during load.

So going off the assumption of the plugin needing to access loader/host symbols I believe idata would be the appropriate place for those symbols (inside the plugin). This is what I see when creating a C/C++ plugin

# common.h:
extern int testThing;

# loader.cpp
// gcc loader.cpp -o loader.exe 
// dlltool --export-all-symbols loader.exe -z loader.def
// dlltool -d loader.def -l libloader.a -D loader.exe
#include  "common.h"
 __declspec(dllexport) int testThing;
...
hinstLib = LoadLibrary(TEXT("plugin.dll")); 
...

# plugin.c
// gcc -c -o plugin.o plugin.c
// gcc -o plugin.dll -s -shared plugin.o -Wl,--subsystem,windows -LI:\development\\dynamicdll_gcc\ -lloader
#include  "common.h"

loader.exe EXPORTS (.edata)
testThing

plugin.dll IMPORTS (.idata)
testThing

I do not see how I can use si command. Should I break on 'main.main' first, and then use si? That would take forever.

I'm not sure why your experience is different than mine. I've provided a gist the debugging on windows if that helps here: https://gist.github.com/cchamplin/2f930b60551aece7de4118f168390093

[alexbrainman]

This is not what I am seeing on *nix. Here is a GDB run-through ...

I am, probably, wrong, but imagine you have runtime.addmoduledata function in your Go program.

When you compile Go executable, the function code endup somewhere in .text section of PE executable.

When executable runs, Windows loads executable file starting at some fixed address (that is hardcoded in the Go linker). So .text section will always starts at the same address (.text section is first section), and runtime.addmoduledata will starts somewhere after that address.

Similarly when Go DLL is built, Go linker puts runtime.addmoduledata somewhere in .text section of an object file and passes it to GCC. GCC copies runtime.addmoduledata code into .text section of DLL file. When .DLL file is loaded by any executable, Windows loads file at some random address - it has to be random, otherwise different DLLs will conflict with each other. So, similarly to executable, .DLL file .text section will endup at fixed offset after that random .DLL address. And runtime.addmoduledata will apears some offset after that.

So if Go executable loads Go DLL, we will have 2 copies of runtime.addmoduledata loaded at different addresses.

But, I don't see how this is relevant to our conversation.

If you look at the values of R15 and RAX you see the addresses match.

I think the difference you see between Linux and Windows, is that R15 in runtime·addmoduledata is set to 0. See this comment about R15

go/src/runtime/asm_amd64.s

Lines 1342 to 1349 in ba2e8a6

	// This is called from .init_array and follows the platform, not Go, ABI.
	TEXT runtime·addmoduledata(SB),NOSPLIT,$0-0
	PUSHQ R15 // The access to global variables below implicitly uses R15, which is callee-save
	MOVQ runtime·lastmoduledatap(SB), AX
	MOVQ DI, moduledata_next(AX)
	MOVQ DI, runtime·lastmoduledatap(SB)
	POPQ R15
	RET

I do not know how R15 is used to access global variables. And who sets R15. Maybe others will help.

I'm not sure why your experience is different than mine. I've provided a gist the debugging on windows if that helps here: https://gist.github.com/cchamplin/2f930b60551aece7de4118f168390093

It does help. I downloaded later gdb version, and it works similarly to yours. Thank you very much.

Alex

[cchamplin]

@alexbrainman

So if Go executable loads Go DLL, we will have 2 copies of runtime.addmoduledata loaded at different addresses.

But, I don't see how this is relevant to our conversation.

It's not so much runtime.addmoduledata but runtime.lastmoduledatap and runtime.firstmoduledata. When we have two versions of these it's causing it to blow up, since in the plugin lastmoduledatap isn't initialized (this is handled by the plugin host)

Anyways I've made real progress and think that the path I'm on is workable here's where we're at, if anyone notices anything that sounds wrong please let me know:

• These comments speak for Windows - Linux may be different in many respects
• Technically the host and the plugin don't both need copies of the go runtime in their binary
• The plugin ideally should not include the whole Go runtime but instead should utilize the runtime included in the host
• The PE format is different in that there isn't really a GOT which appears to be how plugins on *nix are accessing host memory
• If we can make the runtime.* symbols from the host available to the plugin during linking and execution most of the existing plugin code should just work

Current technical steps:

1. build the host

go build -ldflags=all="-linkmode=external -extldflags=-Wl,--export-all-symbols" greeter.go

2. Create symbol definition (this might be temporary, there may be ways to get gcc to produce the shared library)

dlltool --export-all-symbols greeter.exe -z runtime.defs

3. Cleanup runtime.defs (Basically remove non runtime.* symbols)
4. Create library

dlltool -d runtime.defs -l libruntime.a -D greeter.exe

5. Adjust golang linker to produce object files that gcc will correctly link up for buildmode=plugin
   a. Mark every symbol as extern
   b. Set the value for the runtime.* symbols to 0x0 instead of their normal locations
   c. Prefix runtime.* symbols with __imp_ (gcc uses this to correctly build the idata/import table)
   d. Set the section for all runtime.* symbols to idx 0 (should be .text I think)
6. Build the plugin

go build -ldflags=all="\"-extldflags=-LI:\\path\\to\\dir\\ -lruntime\"" -buildmode=plugin .\eng

Where we are now:

So things build and if you run the host (greeter.exe) we are able to get past runtime.addmoduledata. It doesn't directly segfault but we panic when trying to read the pkghash data. Haven't had time to investigate yet.

[rocket049]

I found sombody have a new idea:
"github.com/dearplain/goloader"
it seems better.

[blaubaer]

Ok, what we can do to move that issue forward?

[bradfitz]

@blaubaer, it depends on whether you know how to fix it I guess.

If so, see https://golang.org/doc/contribute.html
If not, see http://golang.org/wiki/NoPlusOne

Really, this is waiting on somebody who knows the subject area and wants to send some code.