cmd/link: exporting free c function from c-shared library hangs executable with that library LD_PRELOADed

[tumdum]

What version of Go are you using (go version)?

go version go1.10.2 linux/amd64

Does this issue reproduce with the latest release?

No idea

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOCACHE="/home/tumdum/.cache/go-build"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/tumdum/go"
GORACE=""
GOROOT="/usr/local/go"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build872026307=/tmp/go-build -gno-record-gcc-switches"

What did you do?

I've tried to intercept C function free via LD_PRELOAD.

$ cat main.go 
package main

import "unsafe"
import "C"

//export free
func free(p unsafe.Pointer) {}
func main()                 {}
$ go build -o fake_malloc.so -buildmode=c-shared main.go
$ LD_PRELOAD=./fake_malloc.so yes
# in never prints nor ends

# in other shell:
$ sudo gdb -q -batch -ex "attach $(pidof yes)" -ex "thread apply all bt"
[New LWP 7421]
Loading Go Runtime support.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x00007fa6355d1072 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7fa635ed1488 <runtime_init_cond+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88	../sysdeps/unix/sysv/linux/futex-internal.h: Nie ma takiego pliku ani katalogu.

Thread 2 (Thread 0x7fa6355c2700 (LWP 7421)):
#0  0x00007fa6355d1072 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7fa635ed1488 <runtime_init_cond+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7fa635ed1420 <runtime_init_mu>, cond=0x7fa635ed1460 <runtime_init_cond>) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7fa635ed1460 <runtime_init_cond>, mutex=0x7fa635ed1420 <runtime_init_mu>) at pthread_cond_wait.c:655
#3  0x00007fa635c4a423 in _cgo_wait_runtime_init_done () at gcc_libinit.c:40
#4  0x00007fa635c4a2ae in free (p0=0x0) at _cgo_export.c:19
#5  0x00007fa6355cc26d in __pthread_attr_destroy (attr=<optimized out>) at pthread_attr_destroy.c:40
#6  0x00007fa635c4a690 in x_cgo_init (g=0x7fa635eb55c0 <runtime.g0>, setg=<optimized out>) at gcc_linux_amd64.c:49
#7  0x00007fa635c41407 in runtime.rt0_go () at /usr/local/go/src/runtime/asm_amd64.s:199
#8  0x00007fa6355c2700 in ?? ()
#9  0x0000000000000000 in ?? ()

Thread 1 (Thread 0x7fa6360be740 (LWP 7420)):
#0  0x00007fa6355d1072 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7fa635ed1488 <runtime_init_cond+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7fa635ed1420 <runtime_init_mu>, cond=0x7fa635ed1460 <runtime_init_cond>) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7fa635ed1460 <runtime_init_cond>, mutex=0x7fa635ed1420 <runtime_init_mu>) at pthread_cond_wait.c:655
#3  0x00007fa635c4a423 in _cgo_wait_runtime_init_done () at gcc_libinit.c:40
#4  0x00007fa635c4a2ae in free (p0=p0@entry=0x5615ac7d7390) at _cgo_export.c:19
#5  0x00007fa63580fb4f in _nl_load_locale_from_archive (category=category@entry=12, namep=namep@entry=0x7ffde9675240) at loadarchive.c:190
#6  0x00007fa63580e6c7 in _nl_find_locale (locale_path=0x0, locale_path_len=0, category=category@entry=12, name=name@entry=0x7ffde9675240) at findlocale.c:154
#7  0x00007fa63580ddfc in __GI_setlocale (category=<optimized out>, locale=<optimized out>) at setlocale.c:340
#8  0x00005615ac2e085d in ?? ()
#9  0x00007fa6358031c1 in __libc_start_main (main=0x5615ac2e0830, argc=1, argv=0x7ffde9675428, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffde9675418) at ../csu/libc-start.c:308
#10 0x00005615ac2e0aba in ?? ()

This works fine with equivalent C version:

$ cat main.c 
void free(void* p)
{
}
$ g++ main.c -fPIC -shared -ldl -o fake_malloc.so
$ LD_PRELOAD=./fake_malloc.so yes
y
y
...

It looks like some sort of deadlock in _cgo_wait_runtime_init_done

What did you expect to see?

Stream of 'y'.

What did you see instead?

Nothing.

[timcooijmans]

I'm also hitting this issue with go1.11.2 on darwin/arm64 by using a gomobile created c-shared library.

The issue seems to be that the process is deadlocked here in the while-loop:

go/src/runtime/cgo/gcc_libinit.c

Lines 34 to 64 in 5929ead

	uintptr_t
	_cgo_wait_runtime_init_done() {
	void (*pfn)(struct context_arg*);
	pthread_mutex_lock(&runtime_init_mu);
	while (runtime_init_done == 0) {
	pthread_cond_wait(&runtime_init_cond, &runtime_init_mu);
	}
	// TODO(iant): For the case of a new C thread calling into Go, such
	// as when using -buildmode=c-archive, we know that Go runtime
	// initialization is complete but we do not know that all Go init
	// functions have been run. We should not fetch cgo_context_function
	// until they have been, because that is where a call to
	// SetCgoTraceback is likely to occur. We are going to wait for Go
	// initialization to be complete anyhow, later, by waiting for
	// main_init_done to be closed in cgocallbackg1. We should wait here
	// instead. See also issue #15943.
	pfn = cgo_context_function;
	pthread_mutex_unlock(&runtime_init_mu);
	if (pfn != nil) {
	struct context_arg arg;
	arg.Context = 0;
	(*pfn)(&arg);
	return arg.Context;
	}
	return 0;
	}

Setting runtime_init_done to 1 is not possible, because it is guarded by the same mutex here:

go/src/runtime/cgo/gcc_libinit.c

Lines 65 to 71 in 5929ead

	void
	x_cgo_notify_runtime_init_done(void* dummy) {
	pthread_mutex_lock(&runtime_init_mu);
	runtime_init_done = 1;
	pthread_cond_broadcast(&runtime_init_cond);
	pthread_mutex_unlock(&runtime_init_mu);
	}

@ianlancetaylor is there any risk in moving the lock from line 38 to line 42?

[ianlancetaylor]

Your analysis is incorrect: pthread_cond_wait unlocks the mutex, so the loop does not block x_cgo_notify_runtime_init_done. Something else is happening.

[timcooijmans]

You are absolutely right, sorry for the confusion. I have an reproduceable deadlock in this while-loop in a gomobile iOS library on first usage of the library but cannot share any code from this. Let's see if I can create a mininal example causing the same deadlock.

…

On Mon, Nov 19, 2018, 22:50 Ian Lance Taylor ***@***.*** wrote: Your analysis is incorrect: pthread_cond_wait unlocks the mutex, so the loop does not block x_cgo_notify_runtime_init_done. Something else is happening. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#26184 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADEcOKIlCiwZdZ4ligNhO_7ybenhGVH_ks5uwyelgaJpZM4U_nyD> .

[ianlancetaylor]

When a Go c-shared library exports a symbol, all calls to that symbol are delayed until the c-shared library is initialized. The c-shared library initialization code starts a thread that initializes the Go runtime; calls to exported symbols are blocked until that initialization is successful.

In this case the c-shared library is exporting a symbol named free. That symbol, which is of course a common C function, is called by the C library startup code before the c-shared library initialization code is run. So we have a deadlock: the call to free is waiting for the c-shared library to be initialized, but the library will never be initialized since the startup code is blocked.

I can only think of one way to resolve this deadlock. If an exported Go function is called before the Go runtime is initialized, it could check whether the initialization thread has been started. If it hasn't been, then it could start the thread itself. But I don't know how safe that is. After all, the C library itself has not yet completed initialization; is it safe to call pthread_create at that point?

I suspect that there is no safe way to implement this.