Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: SIGSEGV in `runtime.shrinkstack` #35916

Open
junchih opened this issue Dec 1, 2019 · 4 comments

Comments

@junchih
Copy link

@junchih junchih commented Dec 1, 2019

$ go version
go version go1.12.5 linux/amd64

I'm not sure about what I've done. Just get below panic sometime. And I've noticed that the 3rd number of runtime.memmove is a little weird, each time I've got this panic, this number is same.

Any idea why or any way how could I reproduce this kind of panic? Or any method how to debug that, I'll be appreciate it.

BTW, no Cgo function calling, no unsafe.Pointer misusing. I've checked that. And we are frequently using reflect package, including the function reflect.Select.

fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0xbfffffffe0 pc=0x46cb04]
runtime stack:
runtime.throw(0x25c2e51, 0x2a)
    golang/go/src/runtime/panic.go:617 +0x72 fp=0x7f986c1ae880 sp=0x7f986c1ae850 pc=0x43ba92
runtime.sigpanic()
    golang/go/src/runtime/signal_unix.go:374 +0x4a9 fp=0x7f986c1ae8b0 sp=0x7f986c1ae880 pc=0x451669
runtime.memmove(0xc023ea9b08, 0xc00a66fb08, 0xffffffffffffc4f8)
    golang/go/src/runtime/memmove_amd64.s:500 +0x644 fp=0x7f986c1ae8b8 sp=0x7f986c1ae8b0 pc=0x46cb04
runtime.syncadjustsudogs(0xc00232de00, 0x4f8, 0x7f986c1ae980, 0xc00a66c000)
    golang/go/src/runtime/stack.go:797 +0x127 fp=0x7f986c1ae8f8 sp=0x7f986c1ae8b8 pc=0x454417
runtime.copystack(0xc00232de00, 0x2000, 0x412f800)
    golang/go/src/runtime/stack.go:858 +0x2d6 fp=0x7f986c1aeab0 sp=0x7f986c1ae8f8 pc=0x454716
runtime.shrinkstack(0xc00232de00)
    golang/go/src/runtime/stack.go:1137 +0x107 fp=0x7f986c1aead8 sp=0x7f986c1aeab0 pc=0x4555d7
runtime.scanstack(0xc00232de00, 0xc00005f770)
    golang/go/src/runtime/mgcmark.go:690 +0x90 fp=0x7f986c1aecd0 sp=0x7f986c1aead8 pc=0x42d180
runtime.scang(0xc00232de00, 0xc00005f770)
    golang/go/src/runtime/proc.go:888 +0x1e4 fp=0x7f986c1aed20 sp=0x7f986c1aecd0 pc=0x43f634
runtime.markroot.func1()
    golang/go/src/runtime/mgcmark.go:221 +0x6e fp=0x7f986c1aed60 sp=0x7f986c1aed20 pc=0x4669ae
runtime.markroot(0xc00005f770, 0x7f9800000191)
    golang/go/src/runtime/mgcmark.go:202 +0x2e9 fp=0x7f986c1aede0 sp=0x7f986c1aed60 pc=0x42c2c9
runtime.gcDrain(0xc00005f770, 0xb)
    golang/go/src/runtime/mgcmark.go:899 +0x112 fp=0x7f986c1aee38 sp=0x7f986c1aede0 pc=0x42daa2
runtime.gcBgMarkWorker.func2()
    golang/go/src/runtime/mgc.go:1924 +0x186 fp=0x7f986c1aee78 sp=0x7f986c1aee38 pc=0x466916
runtime.systemstack(0x0)
    golang/go/src/runtime/asm_amd64.s:351 +0x66 fp=0x7f986c1aee80 sp=0x7f986c1aee78 pc=0x469386
runtime.mstart()
    golang/go/src/runtime/proc.go:1153 fp=0x7f986c1aee88 sp=0x7f986c1aee80 pc=0x440040
@ALTree

This comment has been minimized.

Copy link
Member

@ALTree ALTree commented Dec 1, 2019

Thanks for reporting this.

go version go1.12.5 linux/amd64

There has been eight 1.12 point-releases after 1.12.5, several of which fixed issues in the runtime, the compiler, and the linker.

Is your issue reproducible on the latest (go1.12.13)?

BTW, no Cgo function calling, no unsafe.Pointer misusing. I've checked that.

Does the program has any data race? If you run it with go run -race ..., does the race detector print any warnings?

@junchih

This comment has been minimized.

Copy link
Author

@junchih junchih commented Dec 1, 2019

There has been eight 1.12 point-releases after 1.12.5, several of which fixed issues in the runtime, the compiler, and the linker.

Is your issue reproducible on the latest (go1.12.13)?

This issue can't be reproduced manually easily, this panic will occur about every 7 - 14 days uptime. I don't know how to reproduce it manually. So I'm not sure about if it could be reproduced at go 1.12.13. I'm still trying.

Does the program has any data race? If you run it with go run -race ..., does the race detector print any warnings?

I've detected the data racing, and still running the detecting. And for the reported data race issues, all I've fixed, and still panicing.

@junchih

This comment has been minimized.

Copy link
Author

@junchih junchih commented Dec 1, 2019

hi @ALTree , I even can not reproduce this issue with a simple main.go. So the code and the running environment should not be easily shared and modified for testing, sorry.

Any clue would be appreciated.

@randall77

This comment has been minimized.

Copy link
Contributor

@randall77 randall77 commented Dec 2, 2019

Thanks for the trace.
The immediate problem is the 3rd arg to memmove as you noticed - it is negative.
That looks like it might be caused by a sudog which has an elem pointer that points below the current bottom of the stack (but still within the stack bounds). I don't see how that would happen.

We could paper over the problem by changing runtime.findsghi to trim to gp.sched.sp instead of stk.lo. That I think would guarantee a non-negative memmove size.
But that probably just papers over the real problem - why is there a sudog with a below-bottom-of-stack pointer? Without a reproducer it's going to be hard to track down.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.