New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/link: support msvc object files #20982

Open
ghost opened this Issue Jul 11, 2017 · 154 comments

Comments

Projects
None yet
@ghost

ghost commented Jul 11, 2017

I understand that the go linker cannot currently link msvc object files and also recognize that this issue is likely to be a low priority. However, it would be nice to support this because it would somewhat simplify windows workflow. This issue is mainly to understand how much effort this would be and/or what would be required.

@bradfitz bradfitz changed the title from cgo/ support msvc object files using go linker to cmd/link: support msvc object files Jul 11, 2017

@bradfitz bradfitz added the OS-Windows label Jul 11, 2017

@bradfitz

This comment has been minimized.

Member

bradfitz commented Jul 11, 2017

@alexbrainman

This comment has been minimized.

Member

alexbrainman commented Jul 12, 2017

@xoviat what is the problem that you are having? I need to be able to reproduce it here. So, please, provide all steps I will need to follow to reproduce this.

Thank you

Alex

PS: I won't have computer until end of July. I will have a look at this then.

@ghost

This comment has been minimized.

ghost commented Jul 12, 2017

what is the problem that you are having?

I haven't tried it yet, but I would like to specifically call c functions in msvc object files by linking them as .syso with the go linker. Everything that I have read indicates that this is not possible but I will create a procedure to reproduce the specific error that occurs.

@alexbrainman

This comment has been minimized.

Member

alexbrainman commented Jul 12, 2017

specifically call c functions in msvc object files by linking them as .syso with the go linker

Have you tried building these into a DLL, and use them from inside of DLL?

I will create a procedure to reproduce the specific error that occurs.

Please, do. Thank you.

Alex

@ghost

This comment has been minimized.

ghost commented Jul 12, 2017

Have you tried building these into a DLL, and use them from inside of DLL?

That was actually my original plan. I am using swig so it is not so convenient to compile the generated c code separately and then rewrite the functions as DLL exports. It's not difficult, but it is annoying when the workflow with gcc is just go generate; go build.

@ghost

This comment has been minimized.

ghost commented Jul 13, 2017

Alright, I have a procedure. Start with these files:

hello.go:

package main

/*
	extern void hello();
*/
import "C"

func main() {
	C.hello()
}

hello.c:

#include <stdio.h>

extern void hello()
{
    printf("Hello World from C");
}

then run these commands (with msvc on path):

cl /c hello.c
mv hello.obj hello.syso
mv hello.c hello.c.bak
go build

Result:
Warning: corrupt .drectve at end of def file

When running the produced file:

Exception 0xc0000005 0x8 0x13 0x13
PC=0x13
signal arrived during external code execution

main._Cfunc_hello()
        _//_obj/_cgo_gotypes.go:41 +
main.main()
        C://hello.go:9 +0x27

goroutine 17 [syscall, locked to thread]:
runtime.goexit()
        C:/Program Files/Go/src/runtime/asm_amd64.s:2197 +0x1
rax     0x4a5960
rbx     0xc042045f78
rcx     0x4a9a20
rdi     0xc042045f78
rsi     0x4adc60
rbp     0xc042045f38
rsp     0x6dfd68
r8      0xc042016340
r9      0x0
r10     0xc04204faa0
r11     0x4783c2
r12     0x0
r13     0x6
r14     0x0
r15     0xf1
rip     0x13
rflags  0x10216
cs      0x33
fs      0x53
gs      0x2b
@alexbrainman

This comment has been minimized.

Member

alexbrainman commented Jul 13, 2017

I do not have cl command installed on my pc. How do I install msvc?

Thank you.

Alex

@ghost

This comment has been minimized.

ghost commented Jul 13, 2017

You need the "build tools 2017" here. Microsoft actually now allows anyone to use visual studio for free as long as it's not for commercial development. If it's too much trouble, I can just give you the object file if you want.

@ghost

This comment has been minimized.

ghost commented Jul 13, 2017

If only c++ was not a thing, I wouldn't have to worry about this. But it is, so the pain continues...

@bradfitz bradfitz added this to the Go1.10 milestone Jul 13, 2017

@alexbrainman

This comment has been minimized.

Member

alexbrainman commented Jul 13, 2017

You need the "build tools 2017" here.

Got it. Thank you.

If it's too much trouble, I can just give you the object file if you want.

Yes, please, post hello.obj somewhere.

Thinking about this some more. You are actually using gcc to link object file compiled with msvc compiler. Can you try and do your exercise, but replacing "go build" step with gcc linking hello.obj to a C program? Perhaps it has been done before. I suspect, if we know how to do that, we might be able to do similar with Go.

Alex

@ghost

This comment has been minimized.

ghost commented Jul 13, 2017

AFAIK lld https://github.com/llvm-mirror/lld supports msvc object files.

Object file is here: https://github.com/xoviat/msvcgo/blob/master/hello.syso

@alexbrainman

This comment has been minimized.

Member

alexbrainman commented Jul 13, 2017

lld https://github.com/llvm-mirror/lld supports msvc object files

Go uses gcc linker (not lld) on Windows.

Object file is here: https://github.com/xoviat/msvcgo/blob/master/hello.syso

I will try it when I get home in August. thank you.

Alex

@ghost

This comment has been minimized.

ghost commented Jul 13, 2017

Go uses gcc linker (not lld) on Windows.

I know. But lld is the best open source documentation of msvc object format.

@ghost

This comment has been minimized.

ghost commented Jul 13, 2017

I actually get same error from ld so the error is definitely coming from ld.

@alexbrainman

This comment has been minimized.

Member

alexbrainman commented Jul 13, 2017

I actually get same error from ld so the error is definitely coming from ld.

Yes. We need to work out how to build C program by compiling part with msvc, and linking with gcc.

Alex

@ghost

This comment has been minimized.

ghost commented Jul 13, 2017

Forgive my ignorance, but what exactly gcc linking? Is it linking output of the go linker?

@ghost

This comment has been minimized.

ghost commented Jul 13, 2017

Running objconv on the object files to convert to elf:

objconv -felf hello.obj hello.syso

We now have these errors:

hello.syso: In function `__local_stdio_printf_options':
(.text$mn+0x3): undefined reference to `?_OptionsStorage@?1??__local_stdio_printf_options@@9@9'
hello.syso: In function `_vfprintf_l':
(.text$mn+0x3a): undefined reference to `__stdio_common_vfprintf'
hello.syso: In function `printf':
(.text$mn+0x28): undefined reference to `__acrt_iob_func'
collect2.exe: error: ld returned 1 exit status
@ghost

This comment has been minimized.

ghost commented Jul 13, 2017

Possibly stdio is off limits?

@alexbrainman

This comment has been minimized.

Member

alexbrainman commented Jul 26, 2017

Forgive my ignorance, but what exactly gcc linking? Is it linking output of the go linker?

You use 2 programs to build your Go program:

  • compiler converts your .go files (1 package at the time) into an object file stored under %GOPATH%/pkg directory;
  • linker that builds final .exe file from object files from under %GOPATH%/pkg.

Sometimes (when one of your packages uses Cgo), the Go linker calls external linker to find all bits that are implemented in C. Imagine you call printf from your C code. The C printf compiled code needs to be part of Go executable, but Go linker does not know where to get it. So Go linker calls external linker to include that code.

Current Go uses gcc compiler/linker to compile and link C code (we use mingw gcc). If you are going to compile your C code with different compiler (by Microsoft), you would have to use correspondent linker (by Microsoft) to find all external C code that the compiler created.

So, I guess, I was wrong about suggesting to use gcc linker. For general scenario, you would have to use both Microsoft compiler and linker. We would have to figure out what Microsoft linker requires as input and match that.

You might get away without MC linker, if your C code does not have any external code. Please try some really simple C program (like the one that adds 2 integers or something). That might just work as you have described above.

Alex

@alexbrainman

This comment has been minimized.

Member

alexbrainman commented Jul 26, 2017

Possibly stdio is off limits?

I suspect you need to call Microsoft linker to find that code.

Alex

@mattn

This comment has been minimized.

Member

mattn commented Jul 26, 2017

I don't make sure but

objconv -felf hello.obj hello.syso

Maybe, you should try to do coff or omf instead of elf?

@alexbrainman

This comment has been minimized.

Member

alexbrainman commented Jul 26, 2017

Maybe, you should try to do coff or omf instead of elf?

@xoviat yes, you should not convert .obj files to elf, Windows version of gcc generates pe/coff files just like any other Windows compiler.

Alex

@ghost

This comment has been minimized.

ghost commented Jul 26, 2017

the Go linker calls external linker to find all bits that are implemented in C.

What is the specific command used? If I know the mingw command, then perhaps I can proceed down a file comparison path to try to get msvc to match what mingw is putting out.

@ianlancetaylor

This comment has been minimized.

Contributor

ianlancetaylor commented Jul 26, 2017

You can see the exact comment by running go build -ldflags=-v.

@ghost

This comment has been minimized.

ghost commented Aug 5, 2017

Okay, so from what I can tell:

ld (link -> go.obj) + (gcc -> obj files) ==> a.out.exe

Go appears to create a temporary directory with these files. Is there any way to preserve the temporary directory so that I can inspect its contents?

@mattn

This comment has been minimized.

Member

mattn commented Aug 5, 2017

Try go build -work

WORK=C:\Users\mattn\AppData\Local\Temp\go-build566171254

Object files are remaining in this directory.

@deadprogram

This comment has been minimized.

deadprogram commented May 16, 2018

Discovered this Github issue recently, and this work is very exciting to me. I am hoping this lands somewhere in the near future.

My intention is to use CGo on Windows as wrappers to some 3rd-party code that has been built using MSVC, that I cannot just rebuild from sources using mingw-w64. I have some relatively serious test-cases to run it thru, so can validate it pretty well.

Anyhow, thanks to @cchamplin and whoever else is working on it, and please let me know anyway I can help.

gopherbot pushed a commit to golang/build that referenced this issue May 17, 2018

env/windows: add visual studio tools to image
This change downloads/extracts the Visual Studio Build Tools to C:\godep\vs

Verified by running C:\godep\vs\VC\Tools\MSVC\14.14.26428\bin\Hostx64\x86\cl.exe

Related: golang/go#20982

Change-Id: I9dcb4afc0971317ff39381f089867a08c402974a
Reviewed-on: https://go-review.googlesource.com/112036
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>

@bradfitz bradfitz modified the milestones: Go1.11, Go1.12 Jun 20, 2018

@mxplusb

This comment has been minimized.

mxplusb commented Sep 6, 2018

@cchamplin I know enough to be dangerous about the compiler toolchains in Windows and cgo that I might be able to assist. Would you be able to catch me up on some of what you've been working on? I'll see what I can do to assist. I know the use case @deadprogram has so I have a good advanced test bed when we're ready to test.

@cchamplin

This comment has been minimized.

cchamplin commented Sep 6, 2018

Hey @mxplusb, the patch is actually more or less ready to go. If I can find the time I'll submit the changes to gerrit tonight or tomorrow which will get us underway for getting everything reviewed, updated, and approved.

@gopherbot

This comment has been minimized.

gopherbot commented Sep 6, 2018

Change https://golang.org/cl/133937 mentions this issue: cmd/link: Add flag rlocbss to relocate .bss data to .data

@gopherbot

This comment has been minimized.

gopherbot commented Sep 6, 2018

Change https://golang.org/cl/133938 mentions this issue: runtime/cgo: MSVC toolchain support in cgo native code

@gopherbot

This comment has been minimized.

gopherbot commented Sep 6, 2018

Change https://golang.org/cl/133939 mentions this issue: cmd/cgo: Add toolchain flag to cgo command for MSVC support

@gopherbot

This comment has been minimized.

gopherbot commented Sep 6, 2018

Change https://golang.org/cl/133946 mentions this issue: cmd/compile: Add support for MSVC toolchain to go build

@cchamplin

This comment has been minimized.

cchamplin commented Sep 6, 2018

It looks like I typoed the issue in some of the commits. I'll let the review process get underway and determine the best way to fix.

@gopherbot

This comment has been minimized.

gopherbot commented Sep 7, 2018

Change https://golang.org/cl/133943 mentions this issue: cmd/cgo: Add support for CC_FOR_CGO environment variable

@gopherbot

This comment has been minimized.

gopherbot commented Sep 7, 2018

Change https://golang.org/cl/133942 mentions this issue: tests: Update various tests to prepare for MSVC compiler toolchain

@gopherbot

This comment has been minimized.

gopherbot commented Sep 7, 2018

Change https://golang.org/cl/133940 mentions this issue: misc/cgo: Adjust tests to be compatible with MSVC toolchain support

@gopherbot

This comment has been minimized.

gopherbot commented Sep 7, 2018

Change https://golang.org/cl/133941 mentions this issue: runtime: Add runtime.CompilerType to denote between host compiler type

@gopherbot

This comment has been minimized.

gopherbot commented Sep 7, 2018

Change https://golang.org/cl/133945 mentions this issue: cmd/link: Add external toolchain support for MSVC

@gopherbot

This comment has been minimized.

gopherbot commented Sep 7, 2018

Change https://golang.org/cl/133944 mentions this issue: cmd/compile, cgo: Add support for MSVC flags

@rasky

This comment has been minimized.

Member

rasky commented Sep 7, 2018

@cchamplin can you describe what was the final decision? How does MSVC support look like? What command line flags will be added? What an user will have to do to their environment in addition to downloading MSVC? Thanks

@blizzardplus

This comment has been minimized.

blizzardplus commented Sep 7, 2018

Also, how can we try this patch? I couldn't find your commits on the repo.

@cchamplin

This comment has been minimized.

cchamplin commented Sep 8, 2018

@rasky It's still in a bit of flux as the changes go through review. There will likely not be any specific command line flags that need to be set, simply building cgo applications where the CC environment variable is pointed to MSVC or no CC is set an cl.exe is on the path but gcc is not.

@blizzardplus Similar to above things are in a lot of flux right now with the review, so it might be a bit early to start trying it out (up to you though). All the related changes were posted back to this issue by gopherbot, you can find them there.

@blizzardplus

This comment has been minimized.

blizzardplus commented Sep 9, 2018

@cchamplin is there support for cross compiling using either gcc or clang? If so, are they tested?

@kshelton

This comment has been minimized.

kshelton commented Oct 19, 2018

@cchamplin I don't see any activity on the CLs since 9/11, is there forward progress on this or is it in need of additional resources?

@kshelton

This comment has been minimized.

kshelton commented Oct 30, 2018

@cchamplin for this to work for c-archive the ctors section should be changed to .CRT$XCU (https://msdn.microsoft.com/en-us/library/bb918180.aspx) - If this is not done then the golang runtime will not be initialized. Also, it turns out, if you set it to this it works fine for gcc as well - gcc stil ltakes those functions, moves them to a text section, and modifies main to call them before the real main. Visual Studio can understand .a archive files and, as the interface of the pre-compiled/assembled files is C, there should be no issue just having this in the archive. There is a warning about 2 .text sections but simple cases seem to function fine (needs more testing for confirmation there are no issues).

Also, for this to work for c-shared the functions to be exported must be decorated __declspec(dllexport), but I can't seem to find where this expansion of //export occurs.

@cchamplin

This comment has been minimized.

cchamplin commented Oct 30, 2018

@blizzardplus There is a followup patch to this to support clang-MSVC. I'm not sure what you mean by cross compiling in this case? You will not be able to use the MSVC toolchain on non-windows systems. Other cross compiling functionality should not change.

@kshelton Unfortunately this is a very busy time of year for me as I have other projects in the works and conferences I'm presenting at. I may not be able to jump back into getting the required refactoring into the patch until December.

@kshelton Thanks. I haven't double checked the patch, I assume that I developed it with the assumption that c-archive would not work on MSVC, or specifically disabled it? I went down the path of trying to properly decorate exports when I was trying to get plugin/shared working on windows. There are other issues I believe, but I could be conflating two issues. There are real problems with how go places it's shared code into the PLT/GOT for ELF vs Windows .edata,idata and the EAT. I believe most of the issues I faced were around relocations and being unable to get them to work. If anyone has insight in this that would be wonderful. See #19282

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment