Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/link: support msvc object files #20982

Open
ghost opened this issue Jul 11, 2017 · 222 comments
Open

cmd/link: support msvc object files #20982

ghost opened this issue Jul 11, 2017 · 222 comments

Comments

@ghost
Copy link

@ghost ghost commented Jul 11, 2017

I understand that the go linker cannot currently link msvc object files and also recognize that this issue is likely to be a low priority. However, it would be nice to support this because it would somewhat simplify windows workflow. This issue is mainly to understand how much effort this would be and/or what would be required.

@bradfitz bradfitz changed the title cgo/ support msvc object files using go linker cmd/link: support msvc object files Jul 11, 2017
@bradfitz bradfitz added the OS-Windows label Jul 11, 2017
@bradfitz
Copy link
Contributor

@bradfitz bradfitz commented Jul 11, 2017

@alexbrainman
Copy link
Member

@alexbrainman alexbrainman commented Jul 12, 2017

@xoviat what is the problem that you are having? I need to be able to reproduce it here. So, please, provide all steps I will need to follow to reproduce this.

Thank you

Alex

PS: I won't have computer until end of July. I will have a look at this then.

@ghost
Copy link
Author

@ghost ghost commented Jul 12, 2017

what is the problem that you are having?

I haven't tried it yet, but I would like to specifically call c functions in msvc object files by linking them as .syso with the go linker. Everything that I have read indicates that this is not possible but I will create a procedure to reproduce the specific error that occurs.

@alexbrainman
Copy link
Member

@alexbrainman alexbrainman commented Jul 12, 2017

specifically call c functions in msvc object files by linking them as .syso with the go linker

Have you tried building these into a DLL, and use them from inside of DLL?

I will create a procedure to reproduce the specific error that occurs.

Please, do. Thank you.

Alex

@ghost
Copy link
Author

@ghost ghost commented Jul 12, 2017

Have you tried building these into a DLL, and use them from inside of DLL?

That was actually my original plan. I am using swig so it is not so convenient to compile the generated c code separately and then rewrite the functions as DLL exports. It's not difficult, but it is annoying when the workflow with gcc is just go generate; go build.

@ghost
Copy link
Author

@ghost ghost commented Jul 13, 2017

Alright, I have a procedure. Start with these files:

hello.go:

package main

/*
	extern void hello();
*/
import "C"

func main() {
	C.hello()
}

hello.c:

#include <stdio.h>

extern void hello()
{
    printf("Hello World from C");
}

then run these commands (with msvc on path):

cl /c hello.c
mv hello.obj hello.syso
mv hello.c hello.c.bak
go build

Result:
Warning: corrupt .drectve at end of def file

When running the produced file:

Exception 0xc0000005 0x8 0x13 0x13
PC=0x13
signal arrived during external code execution

main._Cfunc_hello()
        _//_obj/_cgo_gotypes.go:41 +
main.main()
        C://hello.go:9 +0x27

goroutine 17 [syscall, locked to thread]:
runtime.goexit()
        C:/Program Files/Go/src/runtime/asm_amd64.s:2197 +0x1
rax     0x4a5960
rbx     0xc042045f78
rcx     0x4a9a20
rdi     0xc042045f78
rsi     0x4adc60
rbp     0xc042045f38
rsp     0x6dfd68
r8      0xc042016340
r9      0x0
r10     0xc04204faa0
r11     0x4783c2
r12     0x0
r13     0x6
r14     0x0
r15     0xf1
rip     0x13
rflags  0x10216
cs      0x33
fs      0x53
gs      0x2b
@alexbrainman
Copy link
Member

@alexbrainman alexbrainman commented Jul 13, 2017

I do not have cl command installed on my pc. How do I install msvc?

Thank you.

Alex

@ghost
Copy link
Author

@ghost ghost commented Jul 13, 2017

You need the "build tools 2017" here. Microsoft actually now allows anyone to use visual studio for free as long as it's not for commercial development. If it's too much trouble, I can just give you the object file if you want.

@ghost
Copy link
Author

@ghost ghost commented Jul 13, 2017

If only c++ was not a thing, I wouldn't have to worry about this. But it is, so the pain continues...

@bradfitz bradfitz added this to the Go1.10 milestone Jul 13, 2017
@alexbrainman
Copy link
Member

@alexbrainman alexbrainman commented Jul 13, 2017

You need the "build tools 2017" here.

Got it. Thank you.

If it's too much trouble, I can just give you the object file if you want.

Yes, please, post hello.obj somewhere.

Thinking about this some more. You are actually using gcc to link object file compiled with msvc compiler. Can you try and do your exercise, but replacing "go build" step with gcc linking hello.obj to a C program? Perhaps it has been done before. I suspect, if we know how to do that, we might be able to do similar with Go.

Alex

@ghost
Copy link
Author

@ghost ghost commented Jul 13, 2017

AFAIK lld https://github.com/llvm-mirror/lld supports msvc object files.

Object file is here: https://github.com/xoviat/msvcgo/blob/master/hello.syso

@alexbrainman
Copy link
Member

@alexbrainman alexbrainman commented Jul 13, 2017

lld https://github.com/llvm-mirror/lld supports msvc object files

Go uses gcc linker (not lld) on Windows.

Object file is here: https://github.com/xoviat/msvcgo/blob/master/hello.syso

I will try it when I get home in August. thank you.

Alex

@ghost
Copy link
Author

@ghost ghost commented Jul 13, 2017

Go uses gcc linker (not lld) on Windows.

I know. But lld is the best open source documentation of msvc object format.

@ghost
Copy link
Author

@ghost ghost commented Jul 13, 2017

I actually get same error from ld so the error is definitely coming from ld.

@alexbrainman
Copy link
Member

@alexbrainman alexbrainman commented Jul 13, 2017

I actually get same error from ld so the error is definitely coming from ld.

Yes. We need to work out how to build C program by compiling part with msvc, and linking with gcc.

Alex

@ghost
Copy link
Author

@ghost ghost commented Jul 13, 2017

Forgive my ignorance, but what exactly gcc linking? Is it linking output of the go linker?

@ghost
Copy link
Author

@ghost ghost commented Jul 13, 2017

Running objconv on the object files to convert to elf:

objconv -felf hello.obj hello.syso

We now have these errors:

hello.syso: In function `__local_stdio_printf_options':
(.text$mn+0x3): undefined reference to `?_OptionsStorage@?1??__local_stdio_printf_options@@9@9'
hello.syso: In function `_vfprintf_l':
(.text$mn+0x3a): undefined reference to `__stdio_common_vfprintf'
hello.syso: In function `printf':
(.text$mn+0x28): undefined reference to `__acrt_iob_func'
collect2.exe: error: ld returned 1 exit status
@ghost
Copy link
Author

@ghost ghost commented Jul 13, 2017

Possibly stdio is off limits?

@alexbrainman
Copy link
Member

@alexbrainman alexbrainman commented Jul 26, 2017

Forgive my ignorance, but what exactly gcc linking? Is it linking output of the go linker?

You use 2 programs to build your Go program:

  • compiler converts your .go files (1 package at the time) into an object file stored under %GOPATH%/pkg directory;
  • linker that builds final .exe file from object files from under %GOPATH%/pkg.

Sometimes (when one of your packages uses Cgo), the Go linker calls external linker to find all bits that are implemented in C. Imagine you call printf from your C code. The C printf compiled code needs to be part of Go executable, but Go linker does not know where to get it. So Go linker calls external linker to include that code.

Current Go uses gcc compiler/linker to compile and link C code (we use mingw gcc). If you are going to compile your C code with different compiler (by Microsoft), you would have to use correspondent linker (by Microsoft) to find all external C code that the compiler created.

So, I guess, I was wrong about suggesting to use gcc linker. For general scenario, you would have to use both Microsoft compiler and linker. We would have to figure out what Microsoft linker requires as input and match that.

You might get away without MC linker, if your C code does not have any external code. Please try some really simple C program (like the one that adds 2 integers or something). That might just work as you have described above.

Alex

@alexbrainman
Copy link
Member

@alexbrainman alexbrainman commented Jul 26, 2017

Possibly stdio is off limits?

I suspect you need to call Microsoft linker to find that code.

Alex

@mattn
Copy link
Member

@mattn mattn commented Jul 26, 2017

I don't make sure but

objconv -felf hello.obj hello.syso

Maybe, you should try to do coff or omf instead of elf?

@alexbrainman
Copy link
Member

@alexbrainman alexbrainman commented Jul 26, 2017

Maybe, you should try to do coff or omf instead of elf?

@xoviat yes, you should not convert .obj files to elf, Windows version of gcc generates pe/coff files just like any other Windows compiler.

Alex

@ghost
Copy link
Author

@ghost ghost commented Jul 26, 2017

the Go linker calls external linker to find all bits that are implemented in C.

What is the specific command used? If I know the mingw command, then perhaps I can proceed down a file comparison path to try to get msvc to match what mingw is putting out.

@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Jul 26, 2017

You can see the exact comment by running go build -ldflags=-v.

@ghost
Copy link
Author

@ghost ghost commented Aug 5, 2017

Okay, so from what I can tell:

ld (link -> go.obj) + (gcc -> obj files) ==> a.out.exe

Go appears to create a temporary directory with these files. Is there any way to preserve the temporary directory so that I can inspect its contents?

@mattn
Copy link
Member

@mattn mattn commented Aug 5, 2017

Try go build -work

WORK=C:\Users\mattn\AppData\Local\Temp\go-build566171254

Object files are remaining in this directory.

@alexbrainman
Copy link
Member

@alexbrainman alexbrainman commented Jun 2, 2020

Hi, no one ever responded to my request for more information, ...

Hello @manbearian

Which request is that? What information do you need?

Alex

@manbearian
Copy link

@manbearian manbearian commented Jun 3, 2020

I was tagged earlier in this thread for information, but i don't know what folks are looking for and have been unable to help. If no help is needed, that's great. Just please let me know.

@alexbrainman
Copy link
Member

@alexbrainman alexbrainman commented Jun 4, 2020

Just please let me know.

@manbearian

Thanks for your offer. I don't know who tagged you. If they still need help, they will tag you again.

Alex

@cchamplin
Copy link

@cchamplin cchamplin commented Jun 4, 2020

If not yet released, is there any idea on when it will be released ?
Eagerly waiting for the MSVC support.

@dhinesherode91

The code is there and complete; I believe the only thing really holding this back is people testing the changes and providing feedback so they can then be properly reviewed and approved. I am hesitant to rebase the changes on top of current master though. I've done it 5-6 times and it's extremely time consuming, and then I get no testing/feedback. Unless additional parties are willing to participate in testing this, the issue might be dead in the water.

@qmuntal
Copy link
Contributor

@qmuntal qmuntal commented Jun 4, 2020

Unless additional parties are willing to participate in testing this, the issue might be dead in the water.

@cchamplin count on me for testing/debugging your changes. I don't have experience with to go toolchain but I've reviewed your commits and they seem to be pretty well structured and documented.

@dhinesherode91
Copy link

@dhinesherode91 dhinesherode91 commented Jun 5, 2020

@cchamplin Great to hear that the implementation part is completed.
I am interested to involve in the testing part, but I am a beginner in GoLang and also a good learner in windows apis(started few weeks before). I was planning to work with windows api in MS C++ as it looks like lot of official resources are available for it. But then I started looking for an language (for my system programming) where I could use it for my multi platform system programming and I am impressed with GoLang. Always ready for the contribution from my end and is in much need of guidance.

@ericlagergren
Copy link
Contributor

@ericlagergren ericlagergren commented Jun 23, 2020

@cchamplin if you rebase I’ll try building on top of it. I’d love to drop the mingw64 requirement from some of my projects at work. My build tooling is all set up for MSVC, so I wouldn’t have to do anything other than use your patch.

@ericlagergren
Copy link
Contributor

@ericlagergren ericlagergren commented Jun 23, 2020

@cchamplin also, I looked through your CLs and didn’t see any for the cgo tool so I started in on my own patch. Is there any prior work?

@cchamplin
Copy link

@cchamplin cchamplin commented Jun 24, 2020

@ericlagergren

@cchamplin also, I looked through your CLs and didn’t see any for the cgo tool so I started in on my own patch. Is there any prior work?

Not sure if I understand, at https://go-review.googlesource.com/c/go/+/133946/6 many of the related commits directly touch the cgo portion of Go in some capacity.

@ericlagergren
Copy link
Contributor

@ericlagergren ericlagergren commented Jun 24, 2020

@ccahoon okay, I must’ve missed them. If the cgo tool hadn’t been updated I was going to offer to do that.

@alexbrainman
Copy link
Member

@alexbrainman alexbrainman commented Jun 27, 2020

@ericlagergren

I’d love to drop the mingw64 requirement from some of my projects at work

Just FYI. The

https://go-review.googlesource.com/c/go/+/133946/6

change still needs Mingw to work. You will need both Mingw and MSVC tools installed to use the @cchamplin change.

Alex

@cglong
Copy link

@cglong cglong commented Jun 27, 2020

@cchamplin, why is Mingw still needed? What would be needed to remove this dependency?

@ericlagergren
Copy link
Contributor

@ericlagergren ericlagergren commented Jun 27, 2020

@alexbrainman I misspoke. Right now we have to use MSYS2 (which includes mingw64) because the non-MSVC build process requires autotools. Dropping that would be fantastic. But also like @cglong said, I'm not sure why this would also require mingw64.

@cchamplin
Copy link

@cchamplin cchamplin commented Jun 27, 2020

@ericlagergren @cglong CGO needs to gather and have available type information about the C code it's being compiled with. GCC has a flag that will output type data in a conveniently parsable format. MSVC as far as I know does not have the functionality to provide the same information. Building the tooling in go would not only require code to lex/parse C and C++ code but also to understand and evaluate pre-processor directives. It would be a fairly large undertaking to build such a thing in Go. Perhaps there is a library already available somewhere though that we could figure out how to adopt?

@ericlagergren
Copy link
Contributor

@ericlagergren ericlagergren commented Jun 27, 2020

@cchamplin gotcha, that's what I figured. I know of third party libraries like libclang and cznic/cc. But unfortunately libclang is gigantic and cznic/cc hasn't been tested on Windows or macOS. Does PDB not provide enough information?

@cglong
Copy link

@cglong cglong commented Jun 27, 2020

CGO needs to gather and have available type information about the C code it's being compiled with. GCC has a flag that will output type data in a conveniently parsable format. MSVC as far as I know does not have the functionality to provide the same information.

Perhaps @manbearian might be able to provide some pointers here?

@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Jun 27, 2020

I wouldn't say that GCC emits output type data in a special format. It emits DWARF debugging data intended for debuggers, and cgo reads that data using the debug/dwarf package. MSVC presumably also has a debug format. If that format is documented anywhere, we could read it using cgo just as we do the DWARF data.

@ericlagergren
Copy link
Contributor

@ericlagergren ericlagergren commented Jun 27, 2020

@ericlagergren
Copy link
Contributor

@ericlagergren ericlagergren commented Jun 27, 2020

FWIW, reading MSVC’s debug output is what I was talking about when I mentioned the cgo tool in #20982 (comment)

@cchamplin
Copy link

@cchamplin cchamplin commented Jun 27, 2020

I wouldn't say that GCC emits output type data in a special format. It emits DWARF debugging data intended for debuggers, and cgo reads that data using the debug/dwarf package. MSVC presumably also has a debug format. If that format is documented anywhere, we could read it using cgo just as we do the DWARF data.

@ericlagergren @ianlancetaylor
So we do need the DWARF data, which is a big part of it. However, we could theoretically get the same information out of PDB data.

What I was specifically referring to, though, was not just the DWARF data that GCC produces but the output of gcc -E -dM which outputs all the #define's after preprocessing has taken place. It is possible that PDB on its own could provide the same information...what I'm skeptical of is whether or not we'd be able to coerce MSVC into dumping debug information/type information in any capacity without it needing to actually compile the code. Right now CGO is doing just that with GCC .

@ericlagergren
Copy link
Contributor

@ericlagergren ericlagergren commented Jun 27, 2020

From memory, /EP /d1PP should emit something similar to -E -dM.

@cchamplin
Copy link

@cchamplin cchamplin commented Jun 28, 2020

From memory, /EP /d1PP should emit something similar to -E -dM.

If so that'd be excellent, I assume /d1PP is an undocumented flag? I've never seen it before. That would just leave getting equivalent type information out ahead of time.

@ericlagergren
Copy link
Contributor

@ericlagergren ericlagergren commented Jun 28, 2020

@cchamplin yeah, it’s undocumented. A lot of the /dXXX flags are, apparently. Using undocumented flags is unfortunate, but (IMO) it doesn’t seem to be that bad of a sin on Windows, at least. I saw MSVC STL devs talking about it on a reddit thread, but it’s also all over the Internet, too.

@TopperDEL
Copy link

@TopperDEL TopperDEL commented Nov 18, 2020

Can someone give me a short summary of the current state of the MSVC-support? I do not understand very much of all this low-level-compiler-stuff. It's just that I have a go library I want to use from C#/UWP - and that is already working with go build and the buildmode c-shared. BUT: that app won't ever get accepted by the Windows Store as the generated DLL from go misses some compiler/linker-flags that (I assume) can only be set by the MSVC-toolchain.

So: how could I test this here? Any advice is appreciated! Thank you!

@qmuntal
Copy link
Contributor

@qmuntal qmuntal commented Nov 18, 2020

that app won't ever get accepted by the Windows Store as the generated DLL from go misses some compiler/linker-flags that (I assume) can only be set by the MSVC-toolchain.

@TopperDEL does Windows Store tell you which compiler flags are missing? If mingw-64 support them you can always use -extldflags with the right flags when build the dll.

@TopperDEL
Copy link

@TopperDEL TopperDEL commented Nov 18, 2020

@qmuntal Yes, they do:
"Apply the required linker options - SAFESEH, DYNAMICBASE, NXCOMPAT, and APPCONTAINER - when you link the app."

I already tried
go build -ldflags="-s -w '-extldflags=-Wl,--dynamicbase,--high-entropy-va'" -o storj_uplink.dll -buildmode c-shared

But that did not change anything regarding the store-submittion. And from my understanding at least SAFESEH is a MSVC-specific-thing, right?

@qmuntal
Copy link
Contributor

@qmuntal qmuntal commented Nov 18, 2020

"Apply the required linker options - SAFESEH, DYNAMICBASE, NXCOMPAT, and APPCONTAINER - when you link the app."

I don't know any easy way to enable SAFESEH nor APPCONTAINER flags in Mingw-w64, maybe this project https://github.com/status-im/mingw-windows10-uwp can give you some guidance.

On the other hand DYNAMICBASE and NXCOMPAT will be enabled by default in go 1.16 (see #41421) but can already be used with -extldflags just as mentioned, in fact I'm doing so for some projects.

@TopperDEL
Copy link

@TopperDEL TopperDEL commented Nov 18, 2020

@qmuntal Thanks! So at least my extldflags above should be right?
I'll have a look at the mingw for windows 10-uwp.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
You can’t perform that action at this time.