Skip to content

runtime: fatal error: acquireSudog: found s.elem != nil in cache #10541

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
dvyukov opened this issue Apr 22, 2015 · 4 comments
Closed

runtime: fatal error: acquireSudog: found s.elem != nil in cache #10541

dvyukov opened this issue Apr 22, 2015 · 4 comments
Milestone

Comments

@dvyukov
Copy link
Member

dvyukov commented Apr 22, 2015

Just observed the following crash:

fatal error: acquireSudog: found s.elem != nil in cache

goroutine 54 [running]:
runtime.throw(0x8a6bd0, 0x2a)
    src/runtime/panic.go:543 +0x96 fp=0xc209b939a8 sp=0xc209b93990
runtime.acquireSudog(0xc20801edc8)
    src/runtime/proc.go:232 +0x456 fp=0xc209b93a48 sp=0xc209b939a8
runtime.chanrecv(0x6ecee0, 0xc20801ed80, 0xc209b93b78, 0x1, 0x0)
    src/runtime/chan.go:422 +0x306 fp=0xc209b93b08 sp=0xc209b93a48
runtime.chanrecv1(0x6ecee0, 0xc20801ed80, 0xc209b93b78)
    src/runtime/chan.go:335 +0x2b fp=0xc209b93b38 sp=0xc209b93b08
main.(*Testee).shutdown(0xc2081c2380, 0x0, 0x0, 0x0)
    src/github.com/dvyukov/go-fuzz/go-fuzz/testee.go:200 +0xd4 fp=0xc209b93b98 sp=0xc209b93b38
main.(*Slave).exec(0xc208188210, 0xc208010380, 0x7, 0xb, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    src/github.com/dvyukov/go-fuzz/go-fuzz/slave.go:344 +0x20a fp=0xc209b93bf8 sp=0xc209b93b98
main.(*Slave).minimizeInput(0xc208188210, 0xc208011c20, 0xb, 0xb, 0x6dc301, 0xc209b93de8, 0x0, 0x0, 0x0)
    src/github.com/dvyukov/go-fuzz/go-fuzz/slave.go:179 +0x171 fp=0xc209b93da0 sp=0xc209b93bf8
main.(*Slave).processCrasher(0xc208188210, 0xc208011c20, 0xb, 0xb, 0xc20a505000, 0xbbb, 0xbbb, 0xc208110e40, 0xb4, 0xc0, ...)
    src/github.com/dvyukov/go-fuzz/go-fuzz/slave.go:166 +0x93 fp=0xc209b93e58 sp=0xc209b93da0
main.(*Slave).loop(0xc208188210)
    src/github.com/dvyukov/go-fuzz/go-fuzz/slave.go:67 +0x124 fp=0xc209b93fd8 sp=0xc209b93e58
runtime.goexit()
    src/runtime/asm_amd64.s:1671 +0x1 fp=0xc209b93fe0 sp=0xc209b93fd8
created by main.slaveMain
    src/github.com/dvyukov/go-fuzz/go-fuzz/slave.go:56 +0xf6

I see it for the first time.

go version devel +87054c4 Wed Apr 22 02:50:48 2015 +0000 linux/amd64

@dvyukov dvyukov added this to the Go1.5 milestone Apr 22, 2015
@dvyukov
Copy link
Member Author

dvyukov commented Apr 22, 2015

/cc @rsc @randall77 @RLH @aclements

@gopherbot
Copy link
Contributor

CL https://golang.org/cl/10713 mentions this issue.

@gopherbot
Copy link
Contributor

CL https://golang.org/cl/10791 mentions this issue.

methane pushed a commit to methane/go that referenced this issue Jun 6, 2015
Issues golang#10240, golang#10541, golang#10941, golang#11023, golang#11027 and possibly others are
indicating memory corruption in the runtime. One of the easiest places
to both get corruption and detect it is in the allocator's free lists
since they appear throughout memory and follow strict invariants. This
commit adds a check when sweeping a span that its free list is sane
and, if not, it prints the corrupted free list and panics. Hopefully
this will help us collect more information on these failures.

Change-Id: I6d417bcaeedf654943a5e068bd76b58bb02d4a64
aclements added a commit that referenced this issue Jun 7, 2015
Stack barriers assume that writes through pointers to frames above the
current frame will get write barriers, and hence these frames do not
need to be re-scanned to pick up these changes. For normal writes,
this is true. However, there are places in the runtime that use
typedmemmove to potentially write through pointers to higher frames
(such as mapassign1). Currently, typedmemmove does not execute write
barriers if the destination is on the stack. If there's a stack
barrier between the current frame and the frame being modified with
typedmemmove, and the stack barrier is not otherwise hit, it's
possible that the garbage collector will never see the updated pointer
and incorrectly reclaim the object.

Fix this by making heapBitsBulkBarrier (which lies behind typedmemmove
and its variants) detect when the destination is in the stack and
unwind stack barriers up to the point, forcing mark termination to
later rescan the effected frame and collect these pointers.

Fixes #11084. Might be related to #10240, #10541, #10941, #11023,
 #11027 and possibly others.

Change-Id: I323d6cd0f1d29fa01f8fc946f4b90e04ef210efd
Reviewed-on: https://go-review.googlesource.com/10791
Reviewed-by: Russ Cox <rsc@golang.org>
aclements added a commit that referenced this issue Jun 16, 2015
Issues #10240, #10541, #10941, #11023, #11027 and possibly others are
indicating memory corruption in the runtime. One of the easiest places
to both get corruption and detect it is in the allocator's free lists
since they appear throughout memory and follow strict invariants. This
commit adds a check when sweeping a span that its free list is sane
and, if not, it prints the corrupted free list and panics. Hopefully
this will help us collect more information on these failures.

Change-Id: I6d417bcaeedf654943a5e068bd76b58bb02d4a64
Reviewed-on: https://go-review.googlesource.com/10713
Reviewed-by: Keith Randall <khr@golang.org>
Reviewed-by: Russ Cox <rsc@golang.org>
Run-TryBot: Austin Clements <austin@google.com>
@aclements
Copy link
Member

Hi Dmitry. I think we've fixed this issue in the course of fixing various memory corruption and lost write barrier problems over the past few weeks. Please reopen if you're still seeing this or come across it again.

@golang golang locked and limited conversation to collaborators Jun 25, 2016
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

3 participants