Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: deadlock at _cgo_wait_runtime_init_done #42190

Open
tungmangtdh3 opened this issue Oct 24, 2020 · 10 comments
Open

runtime: deadlock at _cgo_wait_runtime_init_done #42190

tungmangtdh3 opened this issue Oct 24, 2020 · 10 comments

Comments

@tungmangtdh3
Copy link

@tungmangtdh3 tungmangtdh3 commented Oct 24, 2020

What version of Go are you using (go version)?

$ go version
go version go1.15.3 windows/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
set GO111MODULE=
set GOARCH=amd64
set GOBIN=
set GOCACHE=C:\Users\nguyenha\AppData\Local\go-build
set GOENV=C:\Users\nguyenha\AppData\Roaming\go\env
set GOEXE=.exe
set GOFLAGS=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOINSECURE=
set GOMODCACHE=C:\Users\nguyenha\go\pkg\mod
set GONOPROXY=
set GONOSUMDB=
set GOOS=windows
set GOPATH=C:\Users\nguyenha\go
set GOPRIVATE=
set GOPROXY=https://proxy.golang.org,direct
set GOROOT=c:\go
set GOSUMDB=sum.golang.org
set GOTMPDIR=
set GOTOOLDIR=c:\go\pkg\tool\windows_amd64
set GCCGO=gccgo
set AR=ar
set CC=gcc
set CXX=g++
set CGO_ENABLED=1
set GOMOD=D:\work\projects\mimic\mimic-core\go.mod
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -mthreads -fmessage-length=0 -fdebug-prefix-map=C:\Users\nguyenha\AppData\Local\Temp\go-build126925810=/tmp/go-build -gno-record-gcc-switches

What did you do?

set CGO_ENABLED=1
set CC=clang
go build -buildmode=c-archive -o bin/libcrawler.lib -v core/export

Go side:
package main
import "C"

//export Init
func Init() {
print("something")
}
// Required by CGO
//
func main() {

}

From C++ side:
#include
#include "libcrawler.h"
#include

using namespace std;

int main() {
cout << "Hello, world1!" << endl;

// x_cgo_notify_runtime_init_done(NULL);
Init();
cout << "Hello, world!" << endl;
return 0;
}

I use clang-cl to compile c++ project

What did you expect to see?

Can call Go function (Init()) from C++

What did you see instead?

A deadlock at _cgo_wait_runtime_init_done. I tried to test by calling x_cgo_notify_runtime_init_done(NULL); then I could pass the loop but crash later.

@ianlancetaylor ianlancetaylor changed the title Cgo: deadlock at _cgo_wait_runtime_init_done runtime: deadlock at _cgo_wait_runtime_init_done Oct 24, 2020
@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Oct 24, 2020

Is the code you are showing us the complete test case? How exactly are you building and running the program? Please try to show us exactly how we can reproduce the problem ourselves. Thanks.

@tungmangtdh3
Copy link
Author

@tungmangtdh3 tungmangtdh3 commented Oct 24, 2020

The build command and export.go and client.cpp are exactly the same as I posted.
From c++, I will create a simple project and send you.

@tungmangtdh3
Copy link
Author

@tungmangtdh3 tungmangtdh3 commented Oct 24, 2020

Test.zip.
Here you go, visual studio project, compile using clang-cl. If you find anything or any hint that I could try from my side, please let me know :). @ianlancetaylor

@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Oct 25, 2020

Thanks, but, sorry, I can't use a Visual Studio project. I'm not a Windows user at all.

Is there a way that you can show me the exact clang commands that you use to build the executable? Thanks.

@tungmangtdh3
Copy link
Author

@tungmangtdh3 tungmangtdh3 commented Oct 26, 2020

The command in windows:
-cd your_project_root
-clang -Xlinker /libpath:D:\work\projects\test\Test\Test\lib -Xlinker libcrawler.a Test.cpp libcrawler.h

It looks like the problem is windows-only, buildmode=c-archive only. I have tried with buildmode=c-shared and it is ok

@dzonerzy
Copy link

@dzonerzy dzonerzy commented Jan 13, 2021

Same issue here, works fine with c-shared but with c-archive _cgo_wait_runtime_init_done just hand due to an infinite loop 'cause runtime_init_done is never 1 and thus the event won't be triggered, still trying to understand the root cause.

@james-li
Copy link

@james-li james-li commented Jan 21, 2021

same issue with c-shared in linux.

I've compiled a so file to use LD_PRELOAD. Deadlock occurs when using the LD_PRELOAD so file

[golib]# LD_PRELOAD=`pwd`/libshellhook.so bash  grepconf.sh -c
<hang here>

When I use gdb to debug this, it's stack show:

(gdb) bt
#0  0x00007fdf8d79a995 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fdf8e313723 in _cgo_wait_runtime_init_done () at gcc_libinit.c:40
#2  0x00007fdf8e313446 in execve (filename=0x1909350 "/usr/libexec/grepconf.sh", argv=0x1908b00, envp=0x1907d10) at _cgo_export.c:26
#3  0x000000000042fc02 in shell_execve ()

(gdb) info threads
  Id   Target Id         Frame 
* 1    Thread 0x7fdf8e856740 (LWP 25485) "bash" 0x00007fdf8d79a995 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

go version

$ go version
go version go1.14.9 linux/amd64

It works fine in fedora 32 and centos 6, but hangs in ubuntu 1804 and centos 7(3.10.0-862.11.6.el7.x86_64.debug )

@james-li
Copy link

@james-li james-li commented Jan 21, 2021

I've found some clues. I put some debug code in /usr/lib/golang/src/runtime/cgo/gcc_libinit.c to trace the x_cgo_notify_runtime_init_done and _cgo_wait_runtime_init_done.

$ diff -u  ./src/runtime/cgo/gcc_libinit.c /usr/lib/golang/src/runtime/cgo/gcc_libinit.c
--- ./src/runtime/cgo/gcc_libinit.c	2021-01-21 13:31:03.192576533 +0800
+++ /usr/lib/golang/src/runtime/cgo/gcc_libinit.c	2021-01-21 16:08:13.426097949 +0800
@@ -11,6 +11,8 @@
 #include <stdlib.h>
 #include <string.h> // strerror
 #include <time.h>
+#include <unistd.h>
+#include <sys/types.h>
 #include "libcgo.h"
 #include "libcgo_unix.h"
 
@@ -28,7 +30,9 @@
 	if (err != 0) {
 		fprintf(stderr, "pthread_create failed: %s", strerror(err));
 		abort();
-	}
+	}else{
+		fprintf(stderr, "pthread_create success: %d:%x\n", getpid(), (unsigned)p);
+        }
 }
 
 uintptr_t
@@ -37,7 +41,15 @@
 
 	pthread_mutex_lock(&runtime_init_mu);
 	while (runtime_init_done == 0) {
-		pthread_cond_wait(&runtime_init_cond, &runtime_init_mu);
+                struct timespec ts;
+                clock_gettime(CLOCK_REALTIME, &ts);
+                ts.tv_sec += 1;
+		if(pthread_cond_timedwait(&runtime_init_cond, &runtime_init_mu, &ts)){
+                        fprintf(stderr, "wait runtime init done at thread %d:%u, value address %p\n", 
+                                (int)getpid(), (unsigned)pthread_self(), &runtime_init_done);
+                        continue;
+                }
+//                pthread_cond_wait(&runtime_init_cond, &runtime_init_mu);
 	}
 
 	// TODO(iant): For the case of a new C thread calling into Go, such
@@ -49,7 +61,10 @@
 	// initialization to be complete anyhow, later, by waiting for
 	// main_init_done to be closed in cgocallbackg1. We should wait here
 	// instead. See also issue #15943.
-	pfn = cgo_context_function;
+        if(runtime_init_done)
+                pfn = cgo_context_function;
+        else
+                pfn = nil;
 
 	pthread_mutex_unlock(&runtime_init_mu);
 	if (pfn != nil) {
@@ -64,10 +79,14 @@
 
 void
 x_cgo_notify_runtime_init_done(void* dummy __attribute__ ((unused))) {
-	pthread_mutex_lock(&runtime_init_mu);
-	runtime_init_done = 1;
-	pthread_cond_broadcast(&runtime_init_cond);
-	pthread_mutex_unlock(&runtime_init_mu);
+        int tries = 0;
+        while(tries ++ < 3){
+                fprintf(stderr, "notify runtime init done at thread %d:%u, value address %p\n", (int)getpid(), (unsigned)pthread_self(), &runtime_init_done);
+                pthread_mutex_lock(&runtime_init_mu);
+                runtime_init_done = 1;
+                pthread_cond_broadcast(&runtime_init_cond);
+                pthread_mutex_unlock(&runtime_init_mu);
+        }
 }
 
 // Sets the context function to call to record the traceback context

Here is the log

notify runtime init done at thread 32626:1276888832, value address 0x7ff74cea5268
wait runtime init done at thread 32630:1292597056, value address 0x7ff74cea5268

It's clear that the notify and wait action occur in two different processes(parent process and child process). So deadlock is unavoidable.

@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Jan 21, 2021

@james-li Using LD_PRELOAD is interesting, but I think it is completely different problem than the rest of this issue, which is about Windows and does not use LD_PRELOAD. I encourage you to take this topic to a different issue, or to a forum. Thanks. (I don't really expect LD_PRELOAD to work, but perhaps there is some way that it could.)

@james-li
Copy link

@james-li james-li commented Jan 22, 2021

@ianlancetaylor Thanks. I post another issue
#43836

@dmitshur dmitshur modified the milestones: Go1.16, Backlog Feb 9, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
5 participants