Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: "fatal error: signal_recv: inconsistent state" crash on ios/arm64 #46466

Closed
stonelgh opened this issue May 31, 2021 · 15 comments
Closed

runtime: "fatal error: signal_recv: inconsistent state" crash on ios/arm64 #46466

stonelgh opened this issue May 31, 2021 · 15 comments

Comments

@stonelgh
Copy link

@stonelgh stonelgh commented May 31, 2021

What version of Go are you using (go version)?

$ go version
go version go1.16.4 darwin/amd64

Does this issue reproduce with the latest release?

yes.

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE="off"
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/stone/Library/Caches/go-build"
GOENV="/Users/stone/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/stone/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/stone/go"
GOPRIVATE=""
GOPROXY="https://goproxy.cn,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
GOVCS=""
GOVERSION="go1.16.4"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -arch x86_64 -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/kk/t_y76cl95cz5y1dtl9spqxt00000gn/T/go-build233946771=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

Make a library for a iOS App with the latest gomobile. The program crashes a lot of times with the following output:

fatal error: signal_recv: inconsistent state

goroutine 20 [running]:
runtime.throw(0x10686cce1, 0x1f)
        /usr/local/go/src/runtime/panic.go:1117 +0x54 fp=0x13004df90 sp=0x13004df60 pc=0x105c804b4
os/signal.signal_recv(0x0)
        /usr/local/go/src/runtime/sigqueue.go:161 +0x1d4 fp=0x13004dfb0 sp=0x13004df90 pc=0x105cb0684
os/signal.loop()
        /usr/local/go/src/os/signal/signal_unix.go:23 +0x20 fp=0x13004dfd0 sp=0x13004dfb0 pc=0x10603ab50
runtime.goexit()
        /usr/local/go/src/runtime/asm_arm64.s:1130 +0x4 fp=0x13004dfd0 sp=0x13004dfd0 pc=0x105cb3a94
created by os/signal.Notify.func1.1
        /usr/local/go/src/os/signal/signal.go:151 +0x44

...

We haven't found a way to consistently reproduce the failure. It usually crashes when users try to resume the app.

What did you expect to see?

The app is resumed and runs normally.

What did you see instead?

The app crashed.

@stonelgh
Copy link
Author

@stonelgh stonelgh commented May 31, 2021

I modified runtime/os_darwin.go to print the return value of read(sigNoteRead, unsafe.Pointer(&b), 1) called in function sigNoteSleep.
It outputs -4 when it crashes. According to runtime/stubs2.go, -4 is a negative errno value. Errno 4 means EINTR(interrupted system call).
That indicates the read system call is interrupted and sigNoteSleep returns prematurely. As a result, sig.state = sigReceiving is detected by runtime/sigqueue.go: func signal_recv() as an inconsistent state and throws.

Shall we check the return value of read and retry when it is interrupted?

Loading

@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented May 31, 2021

Yes.

Loading

@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented May 31, 2021

At least, yes, try that, and see if it fixes the problem. Thanks.

Loading

@stonelgh
Copy link
Author

@stonelgh stonelgh commented Jun 1, 2021

I've made a code change in sigNoteSleep. Will let you know the result.

func sigNoteSleep(*note) {
	for {
		entersyscallblock()
		var b byte
		n := read(sigNoteRead, unsafe.Pointer(&b), 1)
		exitsyscall()
		if n != -_EINTR {
			break
		}
	}
}

Updated at 2021-06-10 to remove the check to sig.state.

Loading

@jim3ma
Copy link

@jim3ma jim3ma commented Jun 1, 2021

Same issue in macos/amd64

Loading

@dr2chase
Copy link
Contributor

@dr2chase dr2chase commented Jun 1, 2021

You have a reproducer for macos that does not require gomobile?

Loading

@stonelgh
Copy link
Author

@stonelgh stonelgh commented Jun 2, 2021

Not yet. Maybe @jim3ma have one?

Loading

@dr2chase
Copy link
Contributor

@dr2chase dr2chase commented Jun 2, 2021

Any news on the goodness of the proposed fix?

Loading

@stonelgh
Copy link
Author

@stonelgh stonelgh commented Jun 3, 2021

Due to the fact that we haven't found a way to consistently reproduce the issue, we haven't observed that the new logic have been triggered by now. We'll continue testing until we get enough evidences. Will let you know ASAP.

Loading

@jim3ma
Copy link

@jim3ma jim3ma commented Jun 3, 2021

Emm, observed many times, but can not consistently reproduce the issue.

Loading

@cherrymui
Copy link
Contributor

@cherrymui cherrymui commented Jun 7, 2021

@jim3ma It doesn't have to consistently reproduce. As long as it fails somewhat frequently, that would be very helpful. Thanks!

Loading

@stonelgh
Copy link
Author

@stonelgh stonelgh commented Jun 10, 2021

@dr2chase @cherrymui @ianlancetaylor Confirmed: the proposed patch (with an update to drop the check to sig.state) can fix the mentioned crash.

Loading

@kavin808
Copy link

@kavin808 kavin808 commented Jun 10, 2021

Same issue on our iOS App which causing crashes with SIGPIPE exception.
We are now adding signal(SIGPIPE, SIG_IGN) to ignore this exception type as a temporary solution.Hope this issue can be fixed soon.

Loading

@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Jun 10, 2021

@stonelgh Thanks for testing and reporting back.

Loading

@gopherbot
Copy link

@gopherbot gopherbot commented Jun 10, 2021

Change https://golang.org/cl/326778 mentions this issue: runtime: loop on EINTR in macOS sigNoteSleep

Loading

@gopherbot gopherbot closed this in 77aa209 Jun 11, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
7 participants