New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/link: exporting `free` c function from c-shared library hangs executable with that library LD_PRELOADed #26184

Open
tumdum opened this Issue Jul 2, 2018 · 4 comments

Comments

Projects
None yet
3 participants
@tumdum

tumdum commented Jul 2, 2018

What version of Go are you using (go version)?

go version go1.10.2 linux/amd64

Does this issue reproduce with the latest release?

No idea

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOCACHE="/home/tumdum/.cache/go-build"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/tumdum/go"
GORACE=""
GOROOT="/usr/local/go"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build872026307=/tmp/go-build -gno-record-gcc-switches"

What did you do?

I've tried to intercept C function free via LD_PRELOAD.

$ cat main.go 
package main

import "unsafe"
import "C"

//export free
func free(p unsafe.Pointer) {}
func main()                 {}
$ go build -o fake_malloc.so -buildmode=c-shared main.go
$ LD_PRELOAD=./fake_malloc.so yes
# in never prints nor ends

# in other shell:
$ sudo gdb -q -batch -ex "attach $(pidof yes)" -ex "thread apply all bt"
[New LWP 7421]
Loading Go Runtime support.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x00007fa6355d1072 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7fa635ed1488 <runtime_init_cond+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88	../sysdeps/unix/sysv/linux/futex-internal.h: Nie ma takiego pliku ani katalogu.

Thread 2 (Thread 0x7fa6355c2700 (LWP 7421)):
#0  0x00007fa6355d1072 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7fa635ed1488 <runtime_init_cond+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7fa635ed1420 <runtime_init_mu>, cond=0x7fa635ed1460 <runtime_init_cond>) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7fa635ed1460 <runtime_init_cond>, mutex=0x7fa635ed1420 <runtime_init_mu>) at pthread_cond_wait.c:655
#3  0x00007fa635c4a423 in _cgo_wait_runtime_init_done () at gcc_libinit.c:40
#4  0x00007fa635c4a2ae in free (p0=0x0) at _cgo_export.c:19
#5  0x00007fa6355cc26d in __pthread_attr_destroy (attr=<optimized out>) at pthread_attr_destroy.c:40
#6  0x00007fa635c4a690 in x_cgo_init (g=0x7fa635eb55c0 <runtime.g0>, setg=<optimized out>) at gcc_linux_amd64.c:49
#7  0x00007fa635c41407 in runtime.rt0_go () at /usr/local/go/src/runtime/asm_amd64.s:199
#8  0x00007fa6355c2700 in ?? ()
#9  0x0000000000000000 in ?? ()

Thread 1 (Thread 0x7fa6360be740 (LWP 7420)):
#0  0x00007fa6355d1072 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7fa635ed1488 <runtime_init_cond+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7fa635ed1420 <runtime_init_mu>, cond=0x7fa635ed1460 <runtime_init_cond>) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7fa635ed1460 <runtime_init_cond>, mutex=0x7fa635ed1420 <runtime_init_mu>) at pthread_cond_wait.c:655
#3  0x00007fa635c4a423 in _cgo_wait_runtime_init_done () at gcc_libinit.c:40
#4  0x00007fa635c4a2ae in free (p0=p0@entry=0x5615ac7d7390) at _cgo_export.c:19
#5  0x00007fa63580fb4f in _nl_load_locale_from_archive (category=category@entry=12, namep=namep@entry=0x7ffde9675240) at loadarchive.c:190
#6  0x00007fa63580e6c7 in _nl_find_locale (locale_path=0x0, locale_path_len=0, category=category@entry=12, name=name@entry=0x7ffde9675240) at findlocale.c:154
#7  0x00007fa63580ddfc in __GI_setlocale (category=<optimized out>, locale=<optimized out>) at setlocale.c:340
#8  0x00005615ac2e085d in ?? ()
#9  0x00007fa6358031c1 in __libc_start_main (main=0x5615ac2e0830, argc=1, argv=0x7ffde9675428, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffde9675418) at ../csu/libc-start.c:308
#10 0x00005615ac2e0aba in ?? ()

This works fine with equivalent C version:

$ cat main.c 
void free(void* p)
{
}
$ g++ main.c -fPIC -shared -ldl -o fake_malloc.so
$ LD_PRELOAD=./fake_malloc.so yes
y
y
...

It looks like some sort of deadlock in _cgo_wait_runtime_init_done

What did you expect to see?

Stream of 'y'.

What did you see instead?

Nothing.

@tumdum tumdum changed the title from Expoting `free` c function from shared library hungs executable with that library LD_PRELOADed to Exporting `free` c function from shared library hungs executable with that library LD_PRELOADed Jul 2, 2018

@ianlancetaylor ianlancetaylor changed the title from Exporting `free` c function from shared library hungs executable with that library LD_PRELOADed to cmd/link: exporting `free` c function from c-shared library hangs executable with that library LD_PRELOADed Jul 2, 2018

@ianlancetaylor ianlancetaylor added this to the Go1.12 milestone Jul 2, 2018

@timcooijmans

This comment has been minimized.

Contributor

timcooijmans commented Nov 19, 2018

I'm also hitting this issue with go1.11.2 on darwin/arm64 by using a gomobile created c-shared library.

The issue seems to be that the process is deadlocked here in the while-loop:

uintptr_t
_cgo_wait_runtime_init_done() {
void (*pfn)(struct context_arg*);
pthread_mutex_lock(&runtime_init_mu);
while (runtime_init_done == 0) {
pthread_cond_wait(&runtime_init_cond, &runtime_init_mu);
}
// TODO(iant): For the case of a new C thread calling into Go, such
// as when using -buildmode=c-archive, we know that Go runtime
// initialization is complete but we do not know that all Go init
// functions have been run. We should not fetch cgo_context_function
// until they have been, because that is where a call to
// SetCgoTraceback is likely to occur. We are going to wait for Go
// initialization to be complete anyhow, later, by waiting for
// main_init_done to be closed in cgocallbackg1. We should wait here
// instead. See also issue #15943.
pfn = cgo_context_function;
pthread_mutex_unlock(&runtime_init_mu);
if (pfn != nil) {
struct context_arg arg;
arg.Context = 0;
(*pfn)(&arg);
return arg.Context;
}
return 0;
}

Setting runtime_init_done to 1 is not possible, because it is guarded by the same mutex here:
void
x_cgo_notify_runtime_init_done(void* dummy) {
pthread_mutex_lock(&runtime_init_mu);
runtime_init_done = 1;
pthread_cond_broadcast(&runtime_init_cond);
pthread_mutex_unlock(&runtime_init_mu);
}

@ianlancetaylor is there any risk in moving the lock from line 38 to line 42?

@ianlancetaylor

This comment has been minimized.

Contributor

ianlancetaylor commented Nov 19, 2018

Your analysis is incorrect: pthread_cond_wait unlocks the mutex, so the loop does not block x_cgo_notify_runtime_init_done. Something else is happening.

@timcooijmans

This comment has been minimized.

Contributor

timcooijmans commented Nov 19, 2018

@ianlancetaylor

This comment has been minimized.

Contributor

ianlancetaylor commented Dec 7, 2018

When a Go c-shared library exports a symbol, all calls to that symbol are delayed until the c-shared library is initialized. The c-shared library initialization code starts a thread that initializes the Go runtime; calls to exported symbols are blocked until that initialization is successful.

In this case the c-shared library is exporting a symbol named free. That symbol, which is of course a common C function, is called by the C library startup code before the c-shared library initialization code is run. So we have a deadlock: the call to free is waiting for the c-shared library to be initialized, but the library will never be initialized since the startup code is blocked.

I can only think of one way to resolve this deadlock. If an exported Go function is called before the Go runtime is initialized, it could check whether the initialization thread has been started. If it hasn't been, then it could start the thread itself. But I don't know how safe that is. After all, the C library itself has not yet completed initialization; is it safe to call pthread_create at that point?

I suspect that there is no safe way to implement this.

@ianlancetaylor ianlancetaylor modified the milestones: Go1.12, Unplanned Dec 7, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment