Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: load two go so/DLLs in one c++ executable on Windows #36628

Closed
foxhlchen opened this issue Jan 17, 2020 · 6 comments
Closed

runtime: load two go so/DLLs in one c++ executable on Windows #36628

foxhlchen opened this issue Jan 17, 2020 · 6 comments

Comments

@foxhlchen
Copy link

@foxhlchen foxhlchen commented Jan 17, 2020

What version of Go are you using (go version)?

$ go version
go version go1.12.6

Does this issue reproduce with the latest release?

not sure

What operating system and processor architecture are you using (go env)?

Windows 10 1903
CentOS 7.2

What did you do?

I wrote two different components in golang and compiled them into C dynamic libraries.
Then built a c++ executable with these two dynamic libraries.

Both these libraries have a running goroutine receiving msg from net and call back c++ functions(not the same one though) to notify events.

In the test environment with a light workload, everything works all right.
But when it comes to heavy load production env, it crashes randomly!

Sometimes I can get exception infomation from stderr saying runtime: unknown pc ,
other times it just simply crashes.

I've carefully checked c++ layer and didn't see any suspicious part (c++ layer is quite simple).
And I also notice that windows is more vulnerable. Rarely see it crashes on Linux.

So I wonder if the problem is coming from the go part.
Can two go shared libraries really coexist together in one process???

If not, How can I solve this?? I don't wanna rewrite everything!

What did you expect to see?

Running normally.

What did you see instead?

randomly crashes
sometimes get exception report like following one.

Exception 0xc0000005 0x0 0x0 0x1d1c26b2
PC=0x1d1c26b2

runtime: unknown pc 0x1d1c26b2
stack: frame={sp:0x791f300, fp:0x0} stack=[0x0,0x791f970)
000000000791f200:  0000000077263128  0000000000000000 
000000000791f210:  0000000000000010  0000000000440000 
000000000791f220:  000000000791f660  0000000000000008 
000000000791f230:  0000000000000000  0000000000000000 
000000000791f240:  0000000000000000  000000001d1c24fb 
000000000791f250:  0000000000000006  0000000000000001 
000000000791f260:  0000000000000030  0000000000000001 
000000000791f270:  0000000000000000  000007fed5d61be6 
000000000791f280:  00000000094ed420  000000001d1c0f3c 
000000000791f290:  0000000000000006  000007fed5d61be0 
000000000791f2a0:  0000000000000000  0000000000398b60 
000000000791f2b0:  000000000791f330  000007fef7585935 
000000000791f2c0:  000000000791f330  000000000791f4b0 
000000000791f2d0:  0000000000000001  0000000000000010 
000000000791f2e0:  0000000000000004  0000000000000001 
000000000791f2f0:  0000000000000000  00000000094ed450 
000000000791f300: <0000007f00000001  0000000000000006 
000000000791f310:  0000000000000006  0000000000000006 
000000000791f320:  0000000000000000  0000000000398b80 
000000000791f330:  0000000002f3a570  000000001d1dbe96 
000000000791f340:  00000000088cbd12  0000000000000000 
000000000791f350:  0000000000000001  0000000000000000 
000000000791f360:  000000000791f3d8  000007fed5d50ddf 
000000000791f370:  00000000094ed420  000000000791f4b0 
000000000791f380:  0000000002bebc10  0000000000000000 
000000000791f390:  fffffffffffffffe  0000000002bebc10 
000000000791f3a0:  000000000791f660  000007fed5d45ccc 
000000000791f3b0:  000007fed5d61be6  00000000094ed420 
000000000791f3c0:  00000000094ed450  0000000000000037 
000000000791f3d0:  0000000000000001  00000000094e6828 
000000000791f3e0:  00000000094ed420  0000000002bebc10 
000000000791f3f0:  00000000094ed420  0000000000000000 
runtime: unknown pc 0x1d1c26b2
stack: frame={sp:0x791f300, fp:0x0} stack=[0x0,0x791f970)
000000000791f200:  0000000077263128  0000000000000000 
000000000791f210:  0000000000000010  0000000000440000 
000000000791f220:  000000000791f660  0000000000000008 
000000000791f230:  0000000000000000  0000000000000000 
000000000791f240:  0000000000000000  000000001d1c24fb 
000000000791f250:  0000000000000006  0000000000000001 
000000000791f260:  0000000000000030  0000000000000001 
000000000791f270:  0000000000000000  000007fed5d61be6 
000000000791f280:  00000000094ed420  000000001d1c0f3c 
000000000791f290:  0000000000000006  000007fed5d61be0 
000000000791f2a0:  0000000000000000  0000000000398b60 
000000000791f2b0:  000000000791f330  000007fef7585935 
000000000791f2c0:  000000000791f330  000000000791f4b0 
000000000791f2d0:  0000000000000001  0000000000000010 
000000000791f2e0:  0000000000000004  0000000000000001 
000000000791f2f0:  0000000000000000  00000000094ed450 
000000000791f300: <0000007f00000001  0000000000000006 
000000000791f310:  0000000000000006  0000000000000006 
000000000791f320:  0000000000000000  0000000000398b80 
000000000791f330:  0000000002f3a570  000000001d1dbe96 
000000000791f340:  00000000088cbd12  0000000000000000 
000000000791f350:  0000000000000001  0000000000000000 
000000000791f360:  000000000791f3d8  000007fed5d50ddf 
000000000791f370:  00000000094ed420  000000000791f4b0 
000000000791f380:  0000000002bebc10  0000000000000000 
000000000791f390:  fffffffffffffffe  0000000002bebc10 
000000000791f3a0:  000000000791f660  000007fed5d45ccc 
000000000791f3b0:  000007fed5d61be6  00000000094ed420 
000000000791f3c0:  00000000094ed450  0000000000000037 
000000000791f3d0:  0000000000000001  00000000094e6828 
000000000791f3e0:  00000000094ed420  0000000002bebc10 
000000000791f3f0:  00000000094ed420  0000000000000000 

goroutine 54 [syscall]:
main._Cfunc_OnRspOrderInsert(0x88cbd12, 0x398b60, 0x398b80, 0xee000, 0xee000, 0x2, 0x402c8f5c28f5c28f, 0x401828f500000064)
	_cgo_gotypes.go:78 +0x4c
main.RecvRun()
	D:/workspace_go/src/ccfinvest/libitg/main.go:232 +0xeff
created by main.LibITG_ReqUserLogin
	D:/workspace_go/src/ccfinvest/libitg/main.go:299 +0x53a

goroutine 51 [IO wait]:
internal/poll.runtime_pollWait(0x38b9198, 0x72, 0x6d553a20)
	C:/Go/src/runtime/netpoll.go:182 +0x5d
internal/poll.(*pollDesc).wait(0xc00010c1c8, 0x72, 0x6d483100, 0x0, 0x0)
	C:/Go/src/internal/poll/fd_poll_runtime.go:87 +0xa2
internal/poll.(*ioSrv).ExecIO(0x6d6931b0, 0xc00010c018, 0x6d51ed30, 0x2, 0xc00051a2d0, 0x9)
	C:/Go/src/internal/poll/fd_windows.go:228 +0x124
internal/poll.(*FD).Read(0xc00010c000, 0xc00051a548, 0x4, 0x4, 0x0, 0x0, 0x0)
	C:/Go/src/internal/poll/fd_windows.go:502 +0x26b
net.(*netFD).Read(0xc00010c000, 0xc00051a548, 0x4, 0x4, 0x10000c00008df78, 0x0, 0x4)
	C:/Go/src/net/fd_windows.go:152 +0x56
net.(*conn).Read(0xc00009c098, 0xc00051a548, 0x4, 0x4, 0x0, 0x0, 0x0)
	C:/Go/src/net/net.go:177 +0x70
ccfinvest/libitg/itg._recvUntilFull(0x6d5573c0, 0xc00009c098, 0xc00051a548, 0x4, 0x4, 0xc0001d9e00, 0x0)
	D:/workspace_go/src/ccfinvest/libitg/itg/itg.go:271 +0xc0
ccfinvest/libitg/itg.(*Itg).recvUntilFull(...)
	D:/workspace_go/src/ccfinvest/libitg/itg/itg.go:288
ccfinvest/libitg/itg.(*Itg).recvRun(0xc0000b83f0)
	D:/workspace_go/src/ccfinvest/libitg/itg/itg.go:294 +0xaf
created by ccfinvest/libitg/itg.(*Itg).Start
	D:/workspace_go/src/ccfinvest/libitg/itg/itg.go:97 +0x1db

goroutine 21 [select]:
github.com/globalsign/mgo.newcoarseTimeProvider.func1(0x17d7840, 0xc000086a50)
	D:/workspace_go/src/github.com/globalsign/mgo/coarse_time.go:51 +0x130
created by github.com/globalsign/mgo.newcoarseTimeProvider
	D:/workspace_go/src/github.com/globalsign/mgo/coarse_time.go:48 +0x109

goroutine 52 [chan receive]:
ccfinvest/libitg/itg.(*Itg).sendRun(0xc0000b83f0)
	D:/workspace_go/src/ccfinvest/libitg/itg/itg.go:350 +0x4d
created by ccfinvest/libitg/itg.(*Itg).Start
	D:/workspace_go/src/ccfinvest/libitg/itg/itg.go:98 +0x200

goroutine 53 [sleep]:
runtime.goparkunlock(...)
	C:/Go/src/runtime/proc.go:307
time.Sleep(0x6fc23ac00)
	C:/Go/src/runtime/time.go:105 +0x16e
ccfinvest/libitg/itg.(*Itg).keepAliveRun(0xc0000b83f0)
	D:/workspace_go/src/ccfinvest/libitg/itg/itg.go:380 +0x61
created by ccfinvest/libitg/itg.(*Itg).Start
	D:/workspace_go/src/ccfinvest/libitg/itg/itg.go:99 +0x225
rax     0x1d4897c0
rbx     0x1c
rcx     0x94e1000
rdi     0x0
rsi     0x3
rbp     0x1
rsp     0x791f300
r8      0x1d4897e0
r9      0x1c
r10     0x6
r11     0x7f
r12     0x2f3a570
r13     0x0
r14     0x1d4897f0
r15     0x0
rip     0x1d1c26b2
rflags  0x10203
cs      0x33
fs      0x53
gs      0x2b
@ianlancetaylor ianlancetaylor changed the title Can I load two go so/DLLs in one c++ executable? runtime: load two go so/DLLs in one c++ executable on Windows Jan 17, 2020
@ianlancetaylor ianlancetaylor added this to the Unplanned milestone Jan 17, 2020
@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Jan 17, 2020

As far as I know this should be possible, but I'm not at all surprised that it does not work.

Can you give us a standalone test case that demonstrates the problem? Thanks.

@foxhlchen
Copy link
Author

@foxhlchen foxhlchen commented Jan 17, 2020

Thanks for replying!

As far as I know this should be possible, but I'm not at all surprised that it does not work.

Every go DLL ships with a golang runtime, I'm curious how they work together? Is there any way to strip away one of them?

Can you give us a standalone test case that demonstrates the problem? Thanks.

I don't know if a simple test case can reproduce this problem.

In golang side, it involves net, channel, goroutine, protobuf, log, encoding/json, etc. And golang/c++ have two-way communications(c++ calls golang, golang calls c++ too). Any of these can go wrong. I don't even know how to trigger the crash, quite random.

But I will try.

@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Jan 17, 2020

On Unix systems, we use symbol visibility so that although each shared object has a copy of the Go runtime, the two runtimes are independent. In effect you get two entirely different Go worlds.

I don't know how that kind of symbol visibility works on Windows.

@foxhlchen
Copy link
Author

@foxhlchen foxhlchen commented Jan 17, 2020

Got it.

I think windows has the equivalent concept, just working in the opposite way. If you don't explicitly export symbols, they are confined in your own mem space.

So you are right, it should work here. Looks like I have to dig more information before blaming it on the go part.

@foxhlchen
Copy link
Author

@foxhlchen foxhlchen commented Feb 10, 2020

After weeks of debugging, we have finally found the bug, it is not related to coexisting of two golang libraries. Multiple golang dlls can work properly even on Windows.

@foxhlchen foxhlchen closed this Feb 10, 2020
@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Feb 10, 2020

Thanks for following up.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
2 participants
You can’t perform that action at this time.