runtime: SIGSEGV after performing clone(CLONE_PARENT) via C constructor prior to runtime start

[lifubang]

Go version

go version go1.22.0 linux/amd64

Output of go env in your module/workspace:

GO111MODULE=''
GOARCH='amd64'
GOBIN=''
GOCACHE='/root/.cache/go-build'
GOENV='/root/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/root/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/root/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/usr/local/go'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/usr/local/go/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.22.0'
GCCGO='gccgo'
GOAMD64='v1'
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='1'
GOMOD='/dev/null'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build720171492=/tmp/go-build -gno-record-gcc-switches'

What did you do?

The main language of runc is go, but we are using c to enter some linux namespaces. Recently, go 1.22.0 has been released, when we want to bump go version to 1.22.0(opencontainers/runc#4193), the CI is fail, it seems that after we are calling clone(2) in c, the children process can't return to go if the first process exited in c.

The test code is in https://github.com/lifubang/cgoclone2/blob/main/main.go
I think we should see From main! in the last line.

What did you see happen?

root@acmcoder:/home/acmcoder/cgo# go version
go version go1.22.0 linux/amd64
root@acmcoder:/home/acmcoder/cgo# go run main.go
STAGE_PARENT
STAGE_CHILD
STAGE_INIT
This from nsexec

What did you expect to see?

root@acmcoder:/home/acmcoder/cgo# go version
go version go1.21.1 linux/amd64
root@acmcoder:/home/acmcoder/cgo# go run main.go
STAGE_PARENT
STAGE_CHILD
STAGE_INIT
This from nsexec
From main!

[kolyshkin]

I can't repro this locally (Fedora 39, kernel 6.7.3-200.fc39.x86_64).

[lifubang]

I test two kernels:

1. github codespaces:

@lifubang ➜ ~/cgoclone2 (main) $ go version
go version go1.22.0 linux/amd64
@lifubang ➜ ~/cgoclone2 (main) $ go run main.go
STAGE_PARENT
STAGE_CHILD
STAGE_INIT
This from nsexec
@lifubang ➜ ~/cgoclone2 (main) $ uname -a
Linux codespaces-f28aa2 6.2.0-1018-azure #18~22.04.1-Ubuntu SMP Tue Nov 21 19:25:02 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

2. my pc

root@acmcoder:/home/acmcoder# uname -a
Linux acmcoder 5.15.0-92-generic #102~20.04.1-Ubuntu SMP Mon Jan 15 13:09:14 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
root@acmcoder:/home/acmcoder# go version
go version go1.22.0 linux/amd64
root@acmcoder:/home/acmcoder# cd cgo
root@acmcoder:/home/acmcoder/cgo# go run main.go
STAGE_PARENT
STAGE_CHILD
STAGE_INIT
This from nsexec
From main!
root@acmcoder:/home/acmcoder/cgo# go run main.go
STAGE_PARENT
STAGE_CHILD
STAGE_INIT
This from nsexec
root@acmcoder:/home/acmcoder/cgo#

It seems that there is a race condition? But I only can see 1 success correct log. Most of time it's wrong.

[kolyshkin]

1. It appears the bug is only visible when using some older distros/kernels (e.g. Ubuntu 20.04).
2. The repro provided by @lifubang segfaults when compiled by Go 1.22 and run on an old distro/kernel.
3. I've bisected this issue to commit 52e17c2 (CL 519457); see details below.
4. When using go compiled from go1.22.0 source tree plus a revert of (CL 519457), I can't reproduce this.

Bisecting details

For bisecting, I was using code provided by @lifubang (see above), an x86_64 VM with Ubuntu 20.04 installed, and the following script:

#!/bin/bash

set -e -u -o pipefail

# Build go.
(cd src && ./make.bash)

# Build our repro.
./bin/go build -o main-tst ~/cgoclone2/main.go

# Run it; check if "main" is printed.
./main-tst | grep main

The bisect itself (cd to go source repo first) can be run like this:

git checkout go1.22.0
git bisect start
git bisect bad
git bisect good go1.21.0
git bisect run ./script

[prattmic]

Most of the code in CL 519457 is dealing with libc/pthreads. I wonder if this actually related to kernel version or glibc version?

Could you test using a newer version of glibc on the machine with the older kernel version. (e.g., by running in a Docker container using a newer distro which has a newer version of glibc).

Could you also provide strace -f output from a failing run, and a backtrace (e.g., with gdb) if there is a crash (unclear)? Thanks!

[prattmic]

cc @golang/runtime

[kolyshkin]

@prattmic it looks like it is indeed related to glibc:

Here's a backtrace from a failed run:

kir@kir-Standard-PC-Q35-ICH9-2009:~/cgoclone2$ gdb ./main-1.23 core-main-1.23-404682-1707498496
GNU gdb (Ubuntu 9.2-0ubuntu1~20.04.1) 9.2
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./main-1.23...
[New LWP 404682]
[New LWP 404680]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `./main-1.23'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  __GI___libc_free (mem=0x100) at malloc.c:3102
3102	malloc.c: No such file or directory.
[Current thread is 1 (LWP 404682)]
warning: File "/home/kir/git/go/src/runtime/runtime-gdb.py" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load".
To enable execution of this file add
	add-auto-load-safe-path /home/kir/git/go/src/runtime/runtime-gdb.py
line to your configuration file "/home/kir/.gdbinit".
To completely disable this security protection add
	set auto-load safe-path /
line to your configuration file "/home/kir/.gdbinit".
For more information about this security protection see the
"Auto-loading safe path" section in the GDB manual.  E.g., run from the shell:
	info "(gdb)Auto-loading safe path"
(gdb) bt
#0  __GI___libc_free (mem=0x100) at malloc.c:3102
#1  0x00007f24608c6cd1 in __pthread_attr_destroy (attr=attr@entry=0x7ffc5fec0740) at pthread_attr_destroy.c:38
#2  0x0000000000482f0a in x_cgo_getstackbound (bounds=bounds@entry=0xe1d6b0) at gcc_stack_unix.c:36
#3  0x0000000000482982 in _cgo_set_stacklo (g=g@entry=0x533140 <runtime.g0>, pbounds=pbounds@entry=0xe1d6b0) at gcc_libinit.c:97
#4  0x0000000000482bac in x_cgo_init (g=0x533140 <runtime.g0>, setg=<optimized out>, tlsg=0x0, tlsbase=0x0) at gcc_linux_amd64.c:44
#5  0x0000000000461db1 in runtime.rt0_go () at /home/kir/git/go/src/runtime/asm_amd64.s:220
#6  0x0000000000000000 in ?? ()
(gdb) 

and the last few lines from strace -f:

brk(NULL)                               = 0xe1d000
brk(0xe3e000)                           = 0xe3e000
write(1, "STAGE_PARENT\n", 13STAGE_PARENT
)          = 13
clone(child_stack=0x7ffc5fec06d0, flags=CLONE_PARENT|SIGCHLD) = 404681
exit_group(0)                           = ?
strace: Process 404681 attached
[pid 404680] +++ exited with 0 +++
write(1, "STAGE_CHILD\n", 12STAGE_CHILD
)           = 12
clone(child_stack=0x7ffc5fec06d0, flags=CLONE_PARENT|SIGCHLD) = 404682
strace: Process 404682 attached
[pid 404681] exit_group(0)              = ?
[pid 404682] write(1, "STAGE_INIT\n", 11STAGE_INIT
) = 11
[pid 404682] write(1, "This from nsexec\n", 17 <unfinished ...>
[pid 404681] +++ exited with 0 +++
This from nsexec
<... write resumed>)                    = 17
openat(AT_FDCWD, "/proc/self/maps", O_RDONLY|O_CLOEXEC) = 3
prlimit64(0, RLIMIT_STACK, NULL, {rlim_cur=8192*1024, rlim_max=RLIM64_INFINITY}) = 0
fstat(3, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
read(3, "00400000-00402000 r--p 00000000 "..., 1024) = 1024
read(3, "460a1a000-7f2460a1e000 r--p 001e"..., 1024) = 1024
read(3, "0a59000-7f2460a7c000 r-xp 000010"..., 1024) = 809
close(3)                                = 0
sched_getaffinity(404680, 32, 0xe1dd40) = -1 ESRCH (No such process)
--- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0xf8} ---
+++ killed by SIGSEGV (core dumped) +++

This is glibc 2.31 (libc6:amd64 2.31-0ubuntu9.14)

I will run it with newer glibc later today.

[prattmic]

We initialize the attr passed to pthread_attr_destroy at https://cs.opensource.google/go/go/+/master:src/runtime/cgo/gcc_stack_unix.c;l=25, but don't check for errors.

I bet that call is failing when the sched_getaffinity fails. man 3 pthread_getattr_np says "In addition, if thread refers to the main thread, then pthread_getattr_np() can fail because of errors from various underlying calls: fopen(3), if /proc/self/maps can't be opened; and getrlimit(2), if the RLIMIT_STACK resource limit is not supported." sched_getaffinity isn't mentioned here, but we do see the /proc/self/maps and prlimit64, so presumably this could be within pthread_getattr_np.

sched_getaffinity is passed PID 404680, which is the PID of one of the exited children (exit message above). That seems wrong. Did pthreads get confused about what the current PID is?

[prattmic]

Did pthreads get confused about what the current PID is?

After reading the reproducer more closely, I can see how this could happen.

clone(CLONE_PARENT) is effectively fork; the only difference is that the parent of the new PID is the same as the parent of the current PID (rather than being the current PID).

This reproducer is doing clone(CLONE_PARENT), letting the original process exit, and letting the new process continue to Go and execute as normal. If some part of pthreads stores global state (such as the current PID) prior to the clone(), that state will propagate into the child, which would be problematic if it is no longer accurate.

I don't have a reproducer environment, but I suspect if you adjust the program like this:

void __attribute__((constructor)) init(void) {
	nsexec();

	pthread_attr_t attr;
	int ret = pthread_getattr_np(pthread_self(), &attr);
	if (ret < 0) {
		abort();
	}
}

That it will hit the abort. And that even extracting this program from the Go file and making it 100% C would have the same effect.

Why working on some versions of glibc and not on others? I suspect that the old version of glibc sets some global state prior to clone() (either in a glibc constructor, on inside clone() itself), while the new version of glibc has either removed this early initialization of global state or learned to clear it after a fork-ish clone.