Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/cgo: support clang on Windows #17014

Open
nadiasvertex opened this issue Sep 7, 2016 · 47 comments
Open

cmd/cgo: support clang on Windows #17014

nadiasvertex opened this issue Sep 7, 2016 · 47 comments
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. help wanted NeedsFix The path to resolution is known, but the work has not been done. OS-Windows
Milestone

Comments

@nadiasvertex
Copy link
Contributor

What version of Go are you using (go version)?

go version devel +0cff219 Wed Sep 7 10:43:13 2016 +0000 windows/amd64

What operating system and processor architecture are you using (go env)?

Windows 10 / AMD64

What did you do?

I used CGO with Clang for Windows 3.8.1 (http://llvm.org/releases/3.8.1/LLVM-3.8.1-win64.exe) like this:

set CC=clang
set CGO_CFLAGS="-m64 -Ipath/to/my/includes"
set CGO_LDFLAGS = "-v -Xlinker -libpath:path/to/libs -lwldap32 -ladvapi32 -lws2_32 -ldbghelp -luser32"

go build

What did you expect to see?

Expected to see nothing (ie, successful compile)

What did you see instead?

# editor/dom
clang version 3.8.1 (branches/release_38)
Target: x86_64-pc-windows-msvc
Thread model: posix
InstalledDir: C:\Program Files\LLVM\bin
 "C:\\Program Files (x86)\\Microsoft Visual Studio 14.0\\VC\\bin\\amd64\\link.exe" "-out:C:\\Users\\CHRIST~1\\AppData\\Local\\Temp\\go-build406860094\\editor\\dom\\_obj\\_cgo_.o" -defaultlib:libcmt "-libpath:C:\\Program Files (x86)\\Microsoft Visual Studio 14.0\\VC\\lib\\amd64" "-libpath:C:\\Program Files (x86)\\Windows Kits\\10\\Lib\\10.0.10240.0\\ucrt\\x64" "-libpath:C:\\Program Files (x86)\\Windows Kits\\8.1\\Lib\\winv6.3\\um\\x64" -nologo "C:\\Users\\CHRIST~1\\AppData\\Local\\Temp\\go-build406860094\\editor\\dom\\_obj\\_cgo_main.o" "C:\\Users\\CHRIST~1\\AppData\\Local\\Temp\\go-build406860094\\editor\\dom\\_obj\\_cgo_export.o" "C:\\Users\\CHRIST~1\\AppData\\Local\\Temp\\go-build406860094\\editor\\dom\\_obj\\binding.cgo2.o" "-libpath:c:\\temp\\go\\src\\editor\\.mm\\windows\\amd64\\release\\lib" core-static.lib ssl-static.lib crypto-static.lib ssh2-static.lib c-ares-static.lib user32.lib ws2_32.lib wldap32.lib advapi32.lib dbghelp.lib
   Creating library $WORK\editor\dom\_obj\_cgo_.lib and object $WORK\editor\dom\_obj\_cgo_.exp
core-static.lib(node.obj) : warning LNK4049: locally defined symbol xmlFree imported
core-static.lib(xml.obj) : warning LNK4049: locally defined symbol xmlFree imported
# editor/dom
clang version 3.8.1 (branches/release_38)
Target: x86_64-pc-windows-msvc
Thread model: posix
InstalledDir: C:\Program Files\LLVM\bin
 "C:\\Program Files (x86)\\Microsoft Visual Studio 14.0\\VC\\bin\\amd64\\link.exe" "-out:C:\\Users\\CHRIST~1\\AppData\\Local\\Temp\\go-build406860094\\editor\\dom\\_obj\\_all.o" "-libpath:C:\\Program Files (x86)\\Microsoft Visual Studio 14.0\\VC\\lib\\amd64" "-libpath:C:\\Program Files (x86)\\Windows Kits\\10\\Lib\\10.0.10240.0\\ucrt\\x64" "-libpath:C:\\Program Files (x86)\\Windows Kits\\8.1\\Lib\\winv6.3\\um\\x64" -nologo "C:\\Users\\CHRIST~1\\AppData\\Local\\Temp\\go-build406860094\\editor\\dom\\_obj\\_cgo_export.o" "C:\\Users\\CHRIST~1\\AppData\\Local\\Temp\\go-build406860094\\editor\\dom\\_obj\\binding.cgo2.o" -Wl,-r
LINK : warning LNK4044: unrecognized option '/Wl,-r'; ignored
LINK : fatal error LNK1561: entry point must be defined
clang.exe: error: linker command failed with exit code 1561 (use -v to see invocation)
@bradfitz
Copy link
Contributor

bradfitz commented Sep 7, 2016

I don't believe clang is supported yet. Our installation docs only mention mingw.

That said, we should probably support clang on Windows at some point.

/cc @ianlancetaylor @alexbrainman @crawshaw

@quentinmit quentinmit added this to the Go1.8Maybe milestone Sep 7, 2016
@alexbrainman
Copy link
Member

I have nothing against clang support. I have never had it installed on my computer. I will try install clang when I have some time.

Alex

@quentinmit quentinmit added the NeedsFix The path to resolution is known, but the work has not been done. label Oct 10, 2016
@rsc rsc modified the milestones: Go1.9, Go1.8Maybe Oct 20, 2016
@rsc rsc changed the title build: cgo does not supply an entry point correctly when compiling with clang for windows cmd/cgo: support clang on Windows Oct 20, 2016
@DmitriyMV
Copy link
Contributor

DmitriyMV commented Nov 3, 2016

I did some digging around and found out that with -Xlinker /subsystem:console it actually gets past entry point must be defined. But now I'm getting this error:

# runtime/cgo
clang version 3.9.0 (branches/release_39)
Target: x86_64-pc-windows-msvc
Thread model: posix
InstalledDir: C:\Dmitry\LLVM_\bin
 "C:\\Program Files (x86)\\Microsoft Visual Studio\\VS15Preview\\Common7\\IDE\\VC\\bin\\amd64\\link.exe" "-out:C:\\Users\\Dmitry\\AppData\\Local\\Temp\\go-build586540150\\runtime\\cgo\\_obj\\_cgo_.o" -defaultlib:libcmt -nologo "C:\\Users\\Dmitry\\AppData\\Local\\Temp\\go-build586540150\\runtime\\cgo\\_obj\\_cgo_main.o" "C:\\Users\\Dmitry\\AppData\\Local\\Temp\\go-build586540150\\runtime\\cgo\\_obj\\_cgo_export.o" "C:\\Users\\Dmitry\\AppData\\Local\\Temp\\go-build586540150\\runtime\\cgo\\_obj\\cgo.cgo2.o" "C:\\Users\\Dmitry\\AppData\\Local\\Temp\\go-build586540150\\runtime\\cgo\\_obj\\gcc_context.o" "C:\\Users\\Dmitry\\AppData\\Local\\Temp\\go-build586540150\\runtime\\cgo\\_obj\\gcc_libinit_windows.o" "C:\\Users\\Dmitry\\AppData\\Local\\Temp\\go-build586540150\\runtime\\cgo\\_obj\\gcc_util.o" "C:\\Users\\Dmitry\\AppData\\Local\\Temp\\go-build586540150\\runtime\\cgo\\_obj\\gcc_windows_amd64.o" "
subsystem:console wldap32.lib advapi32.lib ws2_32.lib dbghelp.lib user32.lib kernel32.lib winmm.lib ntdll.lib
cmd/go
# runtime/cgo
clang version 3.9.0 (branches/release_39)
Target: x86_64-pc-windows-msvc
Thread model: posix
InstalledDir: C:\Dmitry\LLVM_\bin
 "C:\\Program Files (x86)\\Microsoft Visual Studio\\VS15Preview\\Common7\\IDE\\VC\\bin\\amd64\\link.exe" "-out:C:\\Users\\Dmitry\\AppData\\Local\\Temp\\go-build586540150\\runtime\\cgo\\_obj\\_all.o" -nologo "C:\\Users\\Dmitry\\AppData\\Local\\Temp\\go-build586540150\\runtime\\cgo\\_obj\\_cgo_export.o" "C:\\Users\\Dmitry\\AppData\\Local\\Temp\\go-build586540150\\runtime\\cgo\\_obj\\cgo.cgo2.o" "C:\\Users\\Dmitry\\AppData\\Local\\Temp\\go-build586540150\\runtime\\cgo\\_obj\\gcc_context.o" "C:\\Users\\Dmitry\\AppData\\Local\\Temp\\go-build586540150\\runtime\\cgo\\_obj\\gcc_libinit_windows.o" "C:\\Users\\Dmitry\\AppData\\Local\\Temp\\go-build586540150\\runtime\\cgo\\_obj\\gcc_util.o" "C:\\Users\\Dmitry\\AppData\\Local\\Temp\\go-build586540150\\runtime\\cgo\\_obj\\gcc_windows_amd64.o" "C:\\Users\\Dmitry\\AppData\\Local\\Temp\\go-build586540150\\runtime\\cgo\\_obj\\gcc_amd64.o" /subsystem:console -r
LINK : warning LNK4044: unrecognized option '/r'; ignored
gcc_context.o : error LNK2019: unresolved external symbol __stdio_common_vsprintf referenced in function _vsnprintf_l
gcc_libinit_windows.o : error LNK2001: unresolved external symbol __stdio_common_vsprintf
gcc_util.o : error LNK2001: unresolved external symbol __stdio_common_vsprintf
gcc_windows_amd64.o : error LNK2001: unresolved external symbol __stdio_common_vsprintf
gcc_libinit_windows.o : error LNK2019: unresolved external symbol __imp_CreateEventA referenced in function _cgo_preinit_init
gcc_libinit_windows.o : error LNK2019: unresolved external symbol __acrt_iob_func referenced in function _cgo_preinit_init
gcc_util.o : error LNK2001: unresolved external symbol __acrt_iob_func
gcc_windows_amd64.o : error LNK2001: unresolved external symbol __acrt_iob_func
gcc_libinit_windows.o : error LNK2019: unresolved external symbol abort referenced in function _cgo_preinit_init
gcc_util.o : error LNK2001: unresolved external symbol abort
gcc_windows_amd64.o : error LNK2001: unresolved external symbol abort
gcc_libinit_windows.o : error LNK2019: unresolved external symbol __imp_InitializeCriticalSection referenced in function _cgo_preinit_init
gcc_libinit_windows.o : error LNK2019: unresolved external symbol __imp_Sleep referenced in function _cgo_maybe_run_preinit
gcc_libinit_windows.o : error LNK2019: unresolved external symbol _beginthread referenced in function x_cgo_sys_thread_create
gcc_windows_amd64.o : error LNK2001: unresolved external symbol _beginthread
gcc_libinit_windows.o : error LNK2019: unresolved external symbol _errno referenced in function x_cgo_sys_thread_create
gcc_windows_amd64.o : error LNK2001: unresolved external symbol _errno
gcc_libinit_windows.o : error LNK2019: unresolved external symbol __imp_EnterCriticalSection referenced in function _cgo_is_runtime_initialized
gcc_libinit_windows.o : error LNK2019: unresolved external symbol __imp_LeaveCriticalSection referenced in function _cgo_is_runtime_initialized
gcc_libinit_windows.o : error LNK2019: unresolved external symbol __imp_WaitForSingleObject referenced in function _cgo_wait_runtime_init_done
gcc_libinit_windows.o : error LNK2019: unresolved external symbol __imp_SetEvent referenced in function x_cgo_notify_runtime_init_done
gcc_libinit_windows.o : error LNK2019: unresolved external symbol __stdio_common_vfprintf referenced in function _vfprintf_l
gcc_util.o : error LNK2001: unresolved external symbol __stdio_common_vfprintf
gcc_windows_amd64.o : error LNK2001: unresolved external symbol __stdio_common_vfprintf
gcc_util.o : error LNK2019: unresolved external symbol malloc referenced in function x_cgo_thread_start
gcc_windows_amd64.o : error LNK2019: unresolved external symbol free referenced in function threadentry
LINK : error LNK2001: unresolved external symbol mainCRTStartup
C:\Users\Dmitry\AppData\Local\Temp\go-build586540150\runtime\cgo\_obj\_all.o : fatal error LNK1120: 16 unresolved externals
clang.exe: error: linker command failed with exit code 1120 (use -v to see invocation)

With:
set CGO_LDFLAGS=-v -Xlinker /subsystem:console -lwldap32 -ladvapi32 -lws2_32 -ldbghelp -luser32 -lkernel32 -lwinmm -lntdll

I guess, it cannot find the C libs, but then when I tried to add -libpath:"C:\Program Files (x86)\Microsoft Visual Studio\VS15Preview\Common7\IDE\VC\lib\" I'm sure I did something wrong because it started to produce this:

# runtime/cgo
clang version 3.9.0 (branches/release_39)
Target: x86_64-pc-windows-msvc
Thread model: posix
InstalledDir: C:\Dmitry\LLVM_\bin
clang.exe: error: no such file or directory: 'Files'
clang.exe: error: no such file or directory: '(x86)/Microsoft'
clang.exe: error: no such file or directory: 'Visual'
clang.exe: error: no such file or directory: 'Studio/VS15Preview/Common7/IDE/VC/lib"'

@nadiasvertex I would really appreciate an advice on proper libpath setting.

@mattn
Copy link
Member

mattn commented Nov 3, 2016

Maybe it's related on #16455 ?

@DmitriyMV
Copy link
Contributor

DmitriyMV commented Nov 3, 2016

Yes - I created symlink because of that. It solved the second part of the problem, but the first(and main) remains.

@nadiasvertex
Copy link
Contributor Author

@DmitriyMV I don't have much to tell you. The fundamental problem is that clang for Windows uses Microsoft's LIB as its backend. You do not need to tell LIB where to find the C library, it already knows that. I believe that the problem is that Go does not currently pass paths to clang in a way that LIB knows how to consume. I have not had success getting clang to compile with Go by playing with the commands. I think that the Go backend needs to be modified to send different values in order to make this work.

Sorry. I haven't had much time to work on this, but it is something that I would like to get working, or see someone else get working. :-)

@minux
Copy link
Member

minux commented Nov 8, 2016 via email

@nadiasvertex
Copy link
Contributor Author

I prefer Clang's emulation. It makes the compiler part simpler since MSVC
is notoriously slow to get up to date with the standard, among other
frustrations. It would be really nice to be able to create MSVC-compatible
binaries without having to use MSVC. Not that I'm against also supporting
MSVC. We support a lot of C++ code that must compile under MSVC for now,
but we are very interested in dropping MSVC as soon as it is feasible.

On Tue, Nov 8, 2016, 2:48 AM Minux Ma notifications@github.com wrote:

I think core issue is that the target for clang is: x86_64-pc-windows-msvc
i.e. it's meant to be MSVC compatible.

However, Go on windows rely on mingw for cgo.

If we want to do anything here, it's probably to support msvc directly,
rather than clang's emulation of it.


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#17014 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AB8JvBeiKJxeAx5mwRnS6oOx7G8k00lBks5q8Ck1gaJpZM4J2_91
.

@sbhushan87
Copy link

will this feature be available in GO1.8 release as I saw this issue has been linked to GO1.8Maybe issue list?

Any update will be helpful. thanks.

@minux
Copy link
Member

minux commented Jan 12, 2017 via email

@bradfitz bradfitz modified the milestones: Go1.10, Go1.9 Jun 7, 2017
@blizzardplus
Copy link

I'd like to get an update on this? I might be able to push this forward, if there's interest.

@alexbrainman
Copy link
Member

I'd like to get an update on this?

Nothing new to report here.

I might be able to push this forward, if there's interest.

Go ahead if you know what to do. Everyone will help, if we can.
Maybe related #20982.

Alex

@as
Copy link
Contributor

as commented Aug 30, 2017

I guess, it cannot find the C libs, but then when I tried to add -libpath:"C:\Program Files (x86)\Microsoft Visual Studio\VS15Preview\Common7\IDE\VC\lib" I'm sure I did something wrong because it started to produce this:

# runtime/cgo
clang version 3.9.0 (branches/release_39)
Target: x86_64-pc-windows-msvc
Thread model: posix
InstalledDir: C:\Dmitry\LLVM_\bin
clang.exe: error: no such file or directory: 'Files'
clang.exe: error: no such file or directory: '(x86)/Microsoft'
clang.exe: error: no such file or directory: 'Visual'
clang.exe: error: no such file or directory: 'Studio/VS15Preview/Common7/IDE/VC/lib"'

@DmitriyMV
It's splitting the path due to spaces, maybe try setting -libpath to a legacy 8.3 path to the same directory

-libpath C:\progra~2\microsoft\visual~1\VS15Preview\Common7\IDE\VC\lib

@blizzardplus
Copy link

blizzardplus commented Sep 5, 2017

I found a bunch of issues. If you could tell me where in the code these things are processed, I might be able to fix them, unless someone else wants to take care of them:

  1. The Lib path in LDFLAGS should be passed as -L"path/to/libs"
  2. If there is space in the lib path, it does not compile for me. I don't know how spaces are handled by cgo, but when I copied my lib folder to "C:", it had no issue finding it.
  3. For some reason, the clang linker command is missing the _cgo_main.o object when linking objects.
  4. For me, the linker command is called with "-nostdlib" option and it causes this error: "LINK : error LNK2001: unresolved external symbol mainCRTStartup"

Note: I used the go build -x -work command to see the clang commands and keep the temp working directory.

@alexbrainman
Copy link
Member

The Lib path in LDFLAGS should be passed as -L"path/to/libs"

OK. What is your question?

If there is space in the lib path, it does not compile for me. I don't know how spaces are handled by cgo, but when I copied my lib folder to "C:", it had no issue finding it.

I would copy all code into directories without spaces at the start. Once that works, we can deal with paths with spaces.

For some reason, the clang linker command is missing the _cgo_main.o object when linking objects.

According to $GOROOT/src/cmd/cgo/doc.go file:

the build process generates an object file using dynamic
linkage to the desired libraries. The main function is provided by
_cgo_main.c:
	int main() { return 0; }
	void crosscall2(void(*fn)(void*, int, uintptr_t), void *a, int c, uintptr_t ctxt) { }
	uintptr_t _cgo_wait_runtime_init_done() { return 0; }
	void _cgo_release_context(uintptr_t ctxt) { }
	char* _cgo_topofstack(void) { return (char*)0; }
	void _cgo_allocate(void *a, int c) { }
	void _cgo_panic(void *a, int c) { }
	void _cgo_reginit(void) { }
The extra functions here are stubs to satisfy the references in the C
code generated for gcc. The build process links this stub, along with
_cgo_export.c and *.cgo2.c, into a dynamic executable and then lets
cgo examine the executable. 

So it seems to me Go creates _cgo_main.c file and then uses gcc to compile it into _cgo_main.o. Does _cgo_main.c gets created? It should be compiled into _cgo_main.o by invoking gcc, but you do not have gcc. Do you use clang to compile _cgo_main.c? Does compilation succeeds or fails?

For me, the linker command is called with "-nostdlib" option and it causes this error: "LINK : error LNK2001: unresolved external symbol mainCRTStartup"

Searching for "-nostdlib", I find many instances in $GOROOT/src/cmd/go/internal/work/build.go. If clang does not accepts "-nostdlib", maybe try and adjust there?

Note: I used the go build -x -work command to see the clang commands and keep the temp working directory.

The "go build -x -work" only controls go command itself. You, probably, want to do the same for the cmd/link as well. See #20982 (comment) for how to access cmd/link temp files. You might also find -v cmd/link flag useful (or "-v -v").

I hope it helps.

Alex

@ghost
Copy link

ghost commented Sep 6, 2017

The main issue that I received when using the MSVC linker is a segmentation fault that occurred while attempting to bind stdout. I can help to resolve the unresolved symbols but the segmentation fault issue is a bit trickier. My plan for that is to basically manually link everything with MinGW and then use DLL linking to factor out each object at a time until I receive the segmentation fault. I think then I will need to compare assembly to see what the problem is, although maybe there is someone more talented than I am that has a better idea? I am probably stretching the limits of my capability so I am open to as much help as I can get.

@alexbrainman
Copy link
Member

The main issue that I received when using the MSVC linker is a segmentation fault that occurred while attempting to bind stdout.

@xoviat are you taking about issue #20982? Did you manage to build executable successfully? But the executable crashes when it runs? Perhaps if you can provide the steps, I will try and debug it too.

Maybe reply on #20982 so we don't confuse things.

Thank you.

Alex

@blizzardplus
Copy link

blizzardplus commented Sep 6, 2017

Searching for "-nostdlib", I find many instances in $GOROOT/src/cmd/go/internal/work/build.go. If clang does not accepts "-nostdlib", maybe try and adjust there?

I tried to remove the -nostdlib option from the source file and build Go from source, but having some linking issues when building the source code, so stuck there! The problem is that the build.go seems to be appending the option at the end of LDFLAGS, so no matter what LDFLAG options user provides, -nostdlib will be appended. I'm wondering if there is a way to completely override the flags?

Note: I used the go build -x -work command to see the clang commands and keep the temp working directory.
The "go build -x -work" only controls go command itself. You, probably, want to do the same for the cmd/link as well. See #20982 (comment) for how to access cmd/link temp files. You might also find -v cmd/link flag useful (or "-v -v").

I can link the files, by manually adding missing object files and modifying linker options, but don't know the next step the build process takes, so I can't move further.

@ghost
Copy link

ghost commented Sep 6, 2017

@blizzardplus Do you have libgo?

image

@neonphog
Copy link

I'm trying use a go library through rust ffi tx5-go-pion-sys

A bunch of rust deps will only build on windows through msvc, and go will only build with mingw. So right now we're using dynamic linking, but it'd be nice if go supported either msvc directly, or clang and i could use clang-for-msvc so I could statically link the library.

@vault-thirteen
Copy link

vault-thirteen commented Apr 9, 2023

Golang positions itself as a "cross-platform language".
Why is there still no support for a cross-platform technology, the LLVM, in a cross-platform language Go ?

Why are the Go wrapper libraries which use CGO store its source code as on Unix or Linux O.S. ?
Why not store the results as on MS-DOS ? Is MS-DOS bad ?

Some libraries written in C language have portability issues when imported as Unix-Linux source code and compiled on Windows O.S. This fact does not allow me to say that Golang is a truly cross-platfrom language. At least its CGO feature is not truly cross-platform.

Moreover, the GCC compiler is not a best solution for compiling code for Windows O.S.
GCC was created for GNU/Linux and BSD and has issues when used on Windows.

@neptoess
Copy link

neptoess commented Apr 9, 2023

Moreover, the GCC compiler is not a best solution for compiling code for Windows O.S. GCC was created for GNU/Linux and BSD and has issues when used on Windows.

The rest is a bit philosophical, but I’m going to disagree on this point. There are certainly codebases where other compilers will produce “better” output on Windows, but I’ve shipped production Go code using MinGW gcc for the Windows cgo dependencies, and it’s been quite reliable, given me performant code, etc.

I’m not even positive that LLVM on Windows doesn’t work with cgo, because it certainly does on macOS.

@ianlancetaylor
Copy link
Contributor

@vault-thirteen Thanks for the questions. Go is an open source project. As such, the answer to these kinds of questions tends to be "because nobody has fixed it yet." This issue is tagged "help wanted". Would you like to fix it?

@vault-thirteen
Copy link

vault-thirteen commented Apr 11, 2023

Hello @ianlancetaylor.

There are several moments about the Go language.

  1. I saw several merge requests (they are called pull requests on GitHub) about fixing missing C compilers for CGO. These MRs (PRs) were intended to change source code of Go language related to CLang and MSVC. These requests were closed without merging. So, to me it looks not like a question of open-source-ness.

Open-source means literally source which is published to people.
Open-source does not mean that it allows anybody to add anything to the product.

  1. Go language is controlled by Google corporation. Open-sourceness does not affect features of the language if Google does not want to allow feature X or feature Y for some reason.

Would you like to fix it ?

Yes, I would like to fix this, but I am not a creator of this language.

@gophun
Copy link

gophun commented Apr 11, 2023

@vault-thirteen The Go project does not do GitHub pull requests, it uses Gerrit, as has been pointed out in the PR you seem to be referring to. Also, it was the author who closed it.

@vault-thirteen
Copy link

vault-thirteen commented Apr 11, 2023

@gophun thank you for correcting me.
So, does that mean that we have chances to see alternative C compilers in CGO ? :-)

@gophun
Copy link

gophun commented Apr 11, 2023

I don't see why not, if someone steps up and pulls through the whole contribution process and does not abandon their change halfway through.

@vault-thirteen
Copy link

Does CGO provide a way to configure the process of compiling and linking ?
I think it is not giving a user such an opportunity. Correct me if it is not true.
I think the main reason is there.

@gophun
Copy link

gophun commented Apr 11, 2023

Yes, it does. The environment variables are called CC, CXX, CGO_CFLAGS, CGO_CPPFLAGS, CGO_CXXFLAGS, CGO_FFLAGS, CGO_LDFLAGS. https://pkg.go.dev/cmd/cgo#hdr-Using_cgo_with_the_go_command

@vault-thirteen
Copy link

Thank you, @gophun.
I will try to study all this when I have more time.

@vault-thirteen
Copy link

vault-thirteen commented Apr 12, 2023

Where can I find the source code of the CGO mechanism ?

I have found these parts so far:

Are there any other parts available ?

@vault-thirteen
Copy link

vault-thirteen commented Apr 12, 2023

Why are there a lot of unresolved references in Go's source code ? How do you build it ?
I will have to open a new issue ...

@TBBle
Copy link

TBBle commented Apr 12, 2023

How do you build it ?

https://go.dev/doc/install/source appears to be the instructions for building Go from source.

@ianlancetaylor
Copy link
Contributor

Where can I find the source code of the CGO mechanism ?

What are you looking for more precisely? As you've discovered, cmd/cgo is the tool that takes code that transform Go code that uses import "C" into Go code that does not. The generated Go code relies on support in cmd/compile and in runtime, as described in the long comment at the end of cmd/cgo/doc.go.

Why are there a lot of unresolved references in Go's source code ?

There aren't. For questions like this we don't use the issue tracker. Please see https://go.dev/wiki/Questions. Thanks.

@voltagex
Copy link

voltagex commented Jun 4, 2023

I have tried the patch from Gerrit to enable MSVC's clang to be used, unfortunately the following program crashes

package main

/*
#include <stdio.h>
#include <stdlib.h>

void myprint() {
 printf("Hello world\n");
}
*/
import "C"

func main() {
 C.myprint()
}
PS C:\git\cgo-test> go run hello.go
# command-line-arguments
   Creating library $WORK\b001\exe\hello.lib and object $WORK\b001\exe\hello.exp
minpc= 0x7ff70dad5079 min= 0x7ff70dad9ee0 maxpc= 0x7ff70db39b2a max= 0x7ff70db39b2a
fatal error: minpc or maxpc invalid
runtime: panic before malloc heap initialized

runtime stack:
runtime.stopTheWorldWithSema(0x40)
        c:/git/goroot/src/runtime/proc.go:1321 fp=0xba6acffa08 sp=0xba6acffa00 pc=0x7ff70db07b25
runtime: g 0: unexpected return pc for runtime.stopTheWorldWithSema called from 0xba6acffa08
stack: frame={sp:0xba6acffa00, fp:0xba6acffa08} stack=[0xba6acefb90,0xba6acffb90)
0x000000ba6acff900:  0x000000ba6acff970  0x00007ff70db0837e <runtime.forEachP+0xffffffffffffb35e>
0x000000ba6acff910:  0x00007ff70dbddc9b  0x000000ba6acff948
0x000000ba6acff920:  0x00007ff70db0bad1 <runtime.schedule+0xffffffffffffb1f1>  0x00007ff70dc78080
0x000000ba6acff930:  0x0000000000000000  0x01007ff70db0bb00
0x000000ba6acff940:  0x0000000000000008  0x000000ba6acff970
0x000000ba6acff950:  0x00007ff70db08192 <runtime.mexit+0xffffffffffffb3d2>  0x00007ff70dbe232d
0x000000ba6acff960:  0x00007ff70db07b25 <runtime.stopTheWorldWithSema+0xffffffffffffb3a5>  0x000000ba6acffa00
0x000000ba6acff970:  0x000000ba6acff9b0  0x00007ff70db07ea5 <runtime.mstart1+0xffffffffffffb225>
0x000000ba6acff980:  0x00007ff70dc78080  0x00007ff70dbf51e8
0x000000ba6acff990:  0x0000000000000001  0x000000ba6acffa00
0x000000ba6acff9a0:  0x00007ff70db07b25 <runtime.stopTheWorldWithSema+0xffffffffffffb3a5>  0x00007ff70dc78080
0x000000ba6acff9b0:  0x000000ba6acff9f0  0x00007ff70db07e25 <runtime.startTheWorldWithSema+0xffffffffffffb405>
0x000000ba6acff9c0:  0x000000ba6acff9d0  0x00007ff70dc78080
0x000000ba6acff9d0:  0x00007ff70db07e60 <runtime.startTheWorldWithSema+0xffffffffffffb440>  0x00007ff70dc78080
0x000000ba6acff9e0:  0x00007ff70db07b25 <runtime.stopTheWorldWithSema+0xffffffffffffb3a5>  0x000000ba6acffa00
0x000000ba6acff9f0:  0x000000ba6acffa20  0x00007ff70db07b25 <runtime.stopTheWorldWithSema+0xffffffffffffb3a5>
0x000000ba6acffa00: <0x000000ba6acffa08 >0x00007ff70db07b40 <runtime.stopTheWorldWithSema+0xffffffffffffb3c0>
0x000000ba6acffa10:  0x00007ff70dbdee1d  0x0000000000000016
0x000000ba6acffa20:  0x000000ba6acffb40  0x00007ff70db1f0a5 <runtime.(*traceStackTable).put.func1+0xffffffffffffb1c5>
0x000000ba6acffa30:  0x00007ff70dbdee1d  0x000000ba6acffa58
0x000000ba6acffa40:  0x0000000000000000  0xfffffff100000000
0x000000ba6acffa50:  0x000000ba6acffaa0  0x000000ba6acffa90
0x000000ba6acffa60:  0x0000000000000489  0x00007ff70dad9ee0 <runtime.dumpregs+0xffffffffffffb680>
0x000000ba6acffa70:  0x00007ff70db39b2a  0x000000ba6acffa70
0x000000ba6acffa80:  0x000000ba6acffad0  0x00007ff70db2fcca <runtime.mstart0+0xffffffffffffb1ea>
0x000000ba6acffa90:  0x000000ba6acffaa8  0x00007ff70db05156 <runtime.hexdumpWords+0xffffffffffffb2f6>
0x000000ba6acffaa0:  0x0000000000000001  0x000000ba6acffb20
0x000000ba6acffab0:  0x00007ff70db39b2a  0x00007ff70dad5079 fatal error: unexpected signal during runtime execution
runtime: panic before malloc heap initialized
panic during panic
[signal 0x00000000c0000005 code=0x0000000000000000 addr=0x00007ff70eff6a40 pc=0x00007ff70db1f5db]

runtime stack:
runtime.stopTheWorldWithSema(0x0000000000000040)
        c:/git/goroot/src/runtime/proc.go:1321 fp=0x000000ba6acfefe0 sp=0x000000ba6acfefd8 pc=0x00007ff70db07b25
runtime: g 0: unexpected return pc for runtime.stopTheWorldWithSema called from 0x000000ba6acfefe0
stack: frame={sp:0x000000ba6acfefd8, fp:0x000000ba6acfefe0} stack=[0x000000ba6acefb90,0x000000ba6acffb90)
0x000000ba6acfeed8:  0x000000ba6acfef48  0x00007ff70db0837e <runtime.forEachP+0xffffffffffffb35e>
0x000000ba6acfeee8:  0x00007ff70dbddc9b  0x000000ba6acfef28
0x000000ba6acfeef8:  0x00007ff70dbde4dd  0x00007ff70dc78080
0x000000ba6acfef08:  0x00007ff70dbde4dd  0x0100000000000013
0x000000ba6acfef18:  0xc000000500000008  0x00007ff70db1f5db <runtime.traceGCSweepSpan+0xffffffffffffb25b>
0x000000ba6acfef28:  0x00007ff70eff6a40  0x0000000000000000
0x000000ba6acfef38:  0x00007ff70db07b25 <runtime.stopTheWorldWithSema+0xffffffffffffb3a5>  0x000000ba6acfefd8
0x000000ba6acfef48:  0x000000ba6acfef88  0x00007ff70db07ea5 <runtime.mstart1+0xffffffffffffb225>
0x000000ba6acfef58:  0x00007ff70dc78080  0x00007ff70dbf51e8
0x000000ba6acfef68:  0x0000000000000001  0x000000ba6acfefd8
0x000000ba6acfef78:  0x00007ff70db07b25 <runtime.stopTheWorldWithSema+0xffffffffffffb3a5>  0x00007ff70dc78080
0x000000ba6acfef88:  0x000000ba6acfefc8  0x00007ff70db07e25 <runtime.startTheWorldWithSema+0xffffffffffffb405>
0x000000ba6acfef98:  0x000000ba6acfefa8  0x00007ff70dc78080
0x000000ba6acfefa8:  0x00007ff70db07e60 <runtime.startTheWorldWithSema+0xffffffffffffb440>  0x00007ff70dc78080
0x000000ba6acfefb8:  0x00007ff70db07b25 <runtime.stopTheWorldWithSema+0xffffffffffffb3a5>  0x000000ba6acfefd8
0x000000ba6acfefc8:  0x000000ba6acfeff8  0x00007ff70db07b25 <runtime.stopTheWorldWithSema+0xffffffffffffb3a5>
0x000000ba6acfefd8: <0x000000ba6acfefe0 >0x00007ff70db07b40 <runtime.stopTheWorldWithSema+0xffffffffffffb3c0>
0x000000ba6acfefe8:  0x00007ff70dbe1cc5  0x000000000000002a
0x000000ba6acfeff8:  0x000000ba6acff040  0x00007ff70db19d85 <runtime.moduledataverify1+0xffffffffffffb225>
0x000000ba6acff008:  0x00007ff70dbe1cc5  0x0000000000000000
0x000000ba6acff018:  0x00007ff70dbdc319  0x00007ff70dc78080
0x000000ba6acff028:  0x00007ff70dbdc319  0x0000000000000001
0x000000ba6acff038:  0x0000000000000001  0x000000ba6acff060
0x000000ba6acff048:  0x00007ff70db1f5db <runtime.traceGCSweepSpan+0xffffffffffffb25b>  0x00007ff70dbdc319
0x000000ba6acff058:  0x3937303564616430  0x000000ba6acff0d0
0x000000ba6acff068:  0x00007ff70db09fa5 <runtime.gcstopm+0xffffffffffffb205>  0x000000ba6acff08f
0x000000ba6acff078:  0x0000000000000001  0x000000ba6acff0d0
0x000000ba6acff088:  0x20007ff70db09d54  0x0000000000000014
0x000000ba6acff098:  0x00007ff70db09e60 <runtime.stoplockedm+0xffffffffffffb400>  0x00007ff70dad5079 fatal error: unexpected signal during runtime execution
runtime: panic before malloc heap initialized
stack trace unavailable
exit status 4

@davy-ikv
Copy link

davy-ikv commented Apr 16, 2024

I've tried cross-compile from macOS to Windows with llvm-mingw (using ucrt) and it works well. Running the output binary on Windows successfully.

@voltagex
Copy link

voltagex commented Apr 20, 2024

I can confirm the patch in #54811 works against go 1.22.2.

Because I didn't document my last experiment properly

FROM debian:sid
WORKDIR /src
RUN apt update && apt full-upgrade -y
RUN apt install --no-install-recommends -y ca-certificates git golang-go
RUN git clone --single-branch --branch go1.22.2 https://github.com/golang/go

WORKDIR /src/go
COPY *.patch .
RUN git apply *.patch
WORKDIR /src/go/src
RUN GOOS=windows GOARCH=amd64 ./make.bash
  • I could not work out how to pull a complete toolchain out of what I had built. Therefore I installed Go 1.22.2 on Windows, and replaced go.exe and link.exe from the patched source above.
  • I installed a full Visual Studio 2022 with additional clang component from the installer. I suspect you'd be able to get away with the Build Tools and clang.
  • You need to set CGO_ENABLED=1 and CC=clang
  • You need to start the x64 Native Tools command prompt
  • I was then able to build a simple binary with SQLite linked in.

HOWEVER:
It was then immediately detected as malicious by Windows Defender. This is of course, ridiculous.

https://www.microsoft.com/en-us/wdsi/submission/89af1110-6167-44d0-918d-d1c440a81b6f
https://www.microsoft.com/en-us/wdsi/submission/1b794e3a-f8df-4fbd-bf75-e799296f6a85
https://www.virustotal.com/gui/file/c97d9c5a02811bae97048042244a7cee0d44b0459b8f6687fb0cd7f924f7c60b
https://www.virustotal.com/gui/file/7b1e31ad7922c2b0573c11ce4157ab6f366d9d9d2c5687edac5219d212b8ffb9
(although my test.exe wasn't actually built with CGO_ENABLED)

If anyone is interested in getting this merged I am going to need help with the Gerrit process, and I'm going to need to find someone from Microsoft interested in fixing whatever's making Defender angry.

I believe link.exe built from source is also being removed!
image

@voltagex
Copy link

voltagex commented Jul 24, 2024

@dagood / @gdams sorry to ping you directly for this, but maybe this would be something in your wheelhouse

@dagood
Copy link
Contributor

dagood commented Jul 24, 2024

Assuming the ping is about Defender specifically, it looks like https://www.microsoft.com/en-us/wdsi/submission/89af1110-6167-44d0-918d-d1c440a81b6f and https://www.microsoft.com/en-us/wdsi/submission/1b794e3a-f8df-4fbd-bf75-e799296f6a85 have expired. If you upload those again, we may be able to dig deeper into this with the Defender team. (Sorry if it's just a permissions issue and they're still visible on your end, let me know if that's the case.)

Windows Defender is well known to have issues with Go programs in general (https://go.dev/doc/faq#virus, microsoft/go#1255), and it seems understandable to me if switching to a not-before-seen cgo compiler is making Defender act up more than usual. I'm a little confused about your comment about not having CGO_ENABLED enabled, though.

More broadly, I do think clang (and other) C compiler support for cgo on Windows is something that we'd be interested in helping with, but I can't promise we'll have time for it soon.

(For context on the thread: @gdams and I work on the Microsoft fork of Go and try to help out with OS-Windows issues.)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. help wanted NeedsFix The path to resolution is known, but the work has not been done. OS-Windows
Projects
None yet
Development

No branches or pull requests