Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: with GDB, callstack falls into infinite loop with CGO on GOOS=linux GOARCH=arm64 #37238

Closed
iswade opened this issue Feb 15, 2020 · 9 comments

Comments

@iswade
Copy link

@iswade iswade commented Feb 15, 2020

What version of Go are you using (go version)?

go version go1.13.8 linux/arm64

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

GOARCH="arm64"
GOHOSTARCH="arm64"
GOHOSTOS="linux"
GOOS="linux"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build664418125=/tmp/go-build -gno-record-gcc-switches"

What did you do?

run go developped program

What did you expect to see?

program run normally

What did you see instead?

program run normally but the callstask seems fall into infinite loop:

Thread 30 (Thread 0xfffe1dde3f60 (LWP 610668)):
#0  runtime.futex () at /usr1/paxos/go/src/runtime/sys_linux_arm64.s:461
#1  0x0000fffe22fd7ab0 in runtime.futexsleep (addr=0xfffe23d072f8 <runtime.newmHandoff+24>, val=0, ns=-1) at /usr1/paxos/go/src/runtime/os_linux.go:44
#2  0x0000fffe22fb6db0 in runtime.notesleep (n=0xfffe23d072f8 <runtime.newmHandoff+24>) at /usr1/paxos/go/src/runtime/lock_futex.go:151
#3  0x0000fffe22fe00c0 in runtime.templateThread () at /usr1/paxos/go/src/runtime/proc.go:1906
#4  0x0000fffe22fdeb98 in runtime.mstart1 () at /usr1/paxos/go/src/runtime/proc.go:1201
#5  0x0000fffe22fdeab8 in runtime.mstart () at /usr1/paxos/go/src/runtime/proc.go:1167
#6  0x0000fffe2350176c in crosscall1 () at gcc_arm64.S:40
#7  0x0000fffe2350176c in crosscall1 () at gcc_arm64.S:40
#8  0x0000fffe2350176c in crosscall1 () at gcc_arm64.S:40
#9  0x0000fffe2350176c in crosscall1 () at gcc_arm64.S:40
#10 0x0000fffe2350176c in crosscall1 () at gcc_arm64.S:40
#11 0x0000fffe2350176c in crosscall1 () at gcc_arm64.S:40
#12 0x0000fffe2350176c in crosscall1 () at gcc_arm64.S:40
#13 0x0000fffe2350176c in crosscall1 () at gcc_arm64.S:40
#14 0x0000fffe2350176c in crosscall1 () at gcc_arm64.S:40
#15 0x0000fffe2350176c in crosscall1 () at gcc_arm64.S:40
.....

Application(C language) calls library1.so(Golang), and library1.so(Golang) call another library2.so(C language) with cgo.

We build the library1.so with -buildmode=c-shared, and it is used as dynamic library.

The problem only exists at linux/arm64.
The problem does not exists at linux/amd64.

@iswade iswade changed the title runtime: go program run normally, the the callstack seems fall into infinite loop runtime: go program run normally, but the callstack seems fall into infinite loop Feb 15, 2020
@iswade

This comment has been minimized.

Copy link
Author

@iswade iswade commented Feb 15, 2020

at linux/arm64, with the following code, the problem also exists:

package main
  
import "C" 

import (
        "fmt"
        "time"
)

func loop() {
        for i := 0; i < 100000000; i++ {
                fmt.Println("hello")
                time.Sleep(time.Second)
        }
}

func main() {
        go loop()
        time.Sleep(time.Minute * 2)
}

go build main.go

(gdb) bt
#0  runtime.futex () at /usr1/paxos/go/src/runtime/sys_linux_arm64.s:461
#1  0x0000000000426f60 in runtime.futexsleep (addr=0x574bb0 <runtime.sched+272>, val=0, ns=60000000000) at /usr1/paxos/go/src/runtime/os_linux.go:50
#2  0x000000000040a19c in runtime.notetsleep_internal (n=0x574bb0 <runtime.sched+272>, ns=60000000000, ~r2=<optimized out>) at /usr1/paxos/go/src/runtime/lock_futex.go:193
#3  0x000000000040a270 in runtime.notetsleep (n=0x574bb0 <runtime.sched+272>, ns=60000000000, ~r2=<optimized out>) at /usr1/paxos/go/src/runtime/lock_futex.go:216
#4  0x0000000000435680 in runtime.sysmon () at /usr1/paxos/go/src/runtime/proc.go:4316
#5  0x000000000042dab8 in runtime.mstart1 () at /usr1/paxos/go/src/runtime/proc.go:1201
#6  0x000000000042d9d8 in runtime.mstart () at /usr1/paxos/go/src/runtime/proc.go:1167
#7  0x0000000000488a24 in crosscall1 () at gcc_arm64.S:40
#8  0x0000000000488a24 in crosscall1 () at gcc_arm64.S:40
#9  0x0000000000488a24 in crosscall1 () at gcc_arm64.S:40
#10 0x0000000000488a24 in crosscall1 () at gcc_arm64.S:40
#11 0x0000000000488a24 in crosscall1 () at gcc_arm64.S:40
#12 0x0000000000488a24 in crosscall1 () at gcc_arm64.S:40
@odeke-em odeke-em changed the title runtime: go program run normally, but the callstack seems fall into infinite loop runtime: go program run normally, but the callstack seems fall into infinite loop with CGO on GOOS=Linux Feb 16, 2020
@odeke-em odeke-em changed the title runtime: go program run normally, but the callstack seems fall into infinite loop with CGO on GOOS=Linux runtime: callstack seems fall into infinite loop with CGO on GOOS=linux GOARCH=arm64 Feb 16, 2020
@odeke-em

This comment has been minimized.

Copy link
Member

@odeke-em odeke-em commented Feb 16, 2020

Thank you @iswade for this report, the reproducing code and welcome to the Go project!

I’ll kindly loop in some CGO, runtime and arm64 experts: @randall77 @cherrymui @zhangfannie @aclements @ianlancetaylor

@cherrymui

This comment has been minimized.

Copy link
Contributor

@cherrymui cherrymui commented Feb 16, 2020

crosscall1 is basically the bottom of the stack. Maybe we could do something so gdb has the correct unwind info.

It does not affect program execution.

@cherrymui cherrymui added this to the Unplanned milestone Feb 16, 2020
@odeke-em odeke-em changed the title runtime: callstack seems fall into infinite loop with CGO on GOOS=linux GOARCH=arm64 runtime: with GDB, callstack seems fall into infinite loop with CGO on GOOS=linux GOARCH=arm64 Feb 16, 2020
@odeke-em odeke-em changed the title runtime: with GDB, callstack seems fall into infinite loop with CGO on GOOS=linux GOARCH=arm64 runtime: with GDB, callstack falls into infinite loop with CGO on GOOS=linux GOARCH=arm64 Feb 16, 2020
@shawn-xdji

This comment has been minimized.

Copy link
Contributor

@shawn-xdji shawn-xdji commented Feb 28, 2020

Hi, is anyone looking into the issue, I'd like to take a look if needed. thanks.

@ALTree

This comment has been minimized.

Copy link
Member

@ALTree ALTree commented Feb 28, 2020

@shawn-xdji I see no assignee and no gerrit change, so I assume nobody is working on this. Feel free to take it.

@shawn-xdji

This comment has been minimized.

Copy link
Contributor

@shawn-xdji shawn-xdji commented Mar 1, 2020

@shawn-xdji I see no assignee and no gerrit change, so I assume nobody is working on this. Feel free to take it.

Thanks for confirming.

@shawn-xdji

This comment has been minimized.

Copy link
Contributor

@shawn-xdji shawn-xdji commented Mar 27, 2020

Hi @iswade, I got a local fix, am trying to test more scenarios, is it feasible to create a sample case for the c-shared flow you mentioned? Thanks.

@iswade

This comment has been minimized.

Copy link
Author

@iswade iswade commented Mar 31, 2020

@shawn-xdji c-shared flow is a big project and it is not open source, sorry cannot do it. I think the sample code is enough.

at linux/arm64, with the following code, the problem also exists:

package main
  
import "C" 

import (
        "fmt"
        "time"
)

func loop() {
        for i := 0; i < 100000000; i++ {
                fmt.Println("hello")
                time.Sleep(time.Second)
        }
}

func main() {
        go loop()
        time.Sleep(time.Minute * 2)
}

go build main.go

(gdb) bt
#0  runtime.futex () at /usr1/paxos/go/src/runtime/sys_linux_arm64.s:461
#1  0x0000000000426f60 in runtime.futexsleep (addr=0x574bb0 <runtime.sched+272>, val=0, ns=60000000000) at /usr1/paxos/go/src/runtime/os_linux.go:50
#2  0x000000000040a19c in runtime.notetsleep_internal (n=0x574bb0 <runtime.sched+272>, ns=60000000000, ~r2=<optimized out>) at /usr1/paxos/go/src/runtime/lock_futex.go:193
#3  0x000000000040a270 in runtime.notetsleep (n=0x574bb0 <runtime.sched+272>, ns=60000000000, ~r2=<optimized out>) at /usr1/paxos/go/src/runtime/lock_futex.go:216
#4  0x0000000000435680 in runtime.sysmon () at /usr1/paxos/go/src/runtime/proc.go:4316
#5  0x000000000042dab8 in runtime.mstart1 () at /usr1/paxos/go/src/runtime/proc.go:1201
#6  0x000000000042d9d8 in runtime.mstart () at /usr1/paxos/go/src/runtime/proc.go:1167
#7  0x0000000000488a24 in crosscall1 () at gcc_arm64.S:40
#8  0x0000000000488a24 in crosscall1 () at gcc_arm64.S:40
#9  0x0000000000488a24 in crosscall1 () at gcc_arm64.S:40
#10 0x0000000000488a24 in crosscall1 () at gcc_arm64.S:40
#11 0x0000000000488a24 in crosscall1 () at gcc_arm64.S:40
#12 0x0000000000488a24 in crosscall1 () at gcc_arm64.S:40
@gopherbot

This comment has been minimized.

Copy link

@gopherbot gopherbot commented Apr 3, 2020

Change https://golang.org/cl/226999 mentions this issue: runtime: fix infinite callstack of cgo on arm64

@gopherbot gopherbot closed this in 8e121b1 Apr 8, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
6 participants
You can’t perform that action at this time.