Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: "fatal error: schedule: holding locks" during runtime.Stack() #9321

Closed
rina-sleeping opened this issue Dec 15, 2014 · 4 comments
Closed
Milestone

Comments

@rina-sleeping
Copy link

@rina-sleeping rina-sleeping commented Dec 15, 2014

info

  1. What version of Go are you using (go version)?
    • go version go1.4 linux/amd64.
  2. What operating system and processor architecture are you using?
    • Ubuntu 12.04LTS 64bit, Linux 3.8.0-41-generic, Core i7-4770.
  3. What did you do?
    • Run the following test with GOMAXPROCS=4.
  4. What did you expect to see?
    • pass the test.
  5. What did you see instead?
    • "fatal error: schedule: holding locks" occurred.

test code

func TestRepro(t *testing.T) {
    var wg sync.WaitGroup
    wg.Add(2)
    test := func() {
        for i := 0; i < 100; i++ {
            buf := &bytes.Buffer{}
            pprof.Lookup("goroutine").WriteTo(buf, 2)
        }
        wg.Done()
    }

    go test()
    go test()
    wg.Wait()
}

result

=== RUN TestRepro-4
fatal error: schedule: holding locks

runtime stack:
runtime.throw(0x5fce0a)
    /usr/local/go/src/runtime/panic.go:491 +0xad
schedule()
    /usr/local/go/src/runtime/proc.c:1542 +0x45
runtime.park_m(0xc2080007e0)
    /usr/local/go/src/runtime/proc.c:1654 +0x113
runtime.mcall(0x42da74)
    /usr/local/go/src/runtime/asm_amd64.s:186 +0x5a

goroutine 1 [chan receive]:
testing.RunTests(0x58eb80, 0x5ff370, 0x1, 0x1, 0xe6ed7f7c23f1d501)
    /usr/local/go/src/testing/testing.go:556 +0xad6
testing.(*M).Run(0xc20806a0a0, 0x60b9a0)
    /usr/local/go/src/testing/testing.go:485 +0x6c
main.main()
    _/home/rina/repro/_test/_testmain.go:52 +0x1d5

goroutine 5 [semacquire]:
sync.(*WaitGroup).Wait(0xc20808c000)
    /usr/local/go/src/sync/waitgroup.go:132 +0x169
_/home/rina/repro.TestRepro(0xc208088000)
    /home/rina/repro/sample_test.go:23 +0xba
testing.tRunner(0xc208088000, 0x5ff370)
    /usr/local/go/src/testing/testing.go:447 +0xbf
created by testing.RunTests
    /usr/local/go/src/testing/testing.go:555 +0xa8b

goroutine 17 [semacquire]:
runtime.Stack(0xc208098000, 0x100000, 0x100000, 0xc208098001, 0x100000)
    /usr/local/go/src/runtime/mprof.go:581 +0x55
runtime/pprof.writeGoroutineStacks(0x7f2b76ce2c20, 0xc2080100e0, 0x0, 0x0)
    /usr/local/go/src/runtime/pprof/pprof.go:511 +0x8d
runtime/pprof.writeGoroutine(0x7f2b76ce2c20, 0xc2080100e0, 0x2, 0x0, 0x0)
    /usr/local/go/src/runtime/pprof/pprof.go:500 +0x4f
runtime/pprof.(*Profile).WriteTo(0x5ff9a0, 0x7f2b76ce2c20, 0xc2080100e0, 0x2, 0x0, 0x0)
    /usr/local/go/src/runtime/pprof/pprof.go:229 +0xd5
_/home/rina/repro.func·001()
    /home/rina/repro/sample_test.go:16 +0xce
created by _/home/rina/repro.TestRepro
    /home/rina/repro/sample_test.go:21 +0x9b

goroutine 18 [running]:
    goroutine running on other thread; stack unavailable
created by _/home/rina/repro.TestRepro
    /home/rina/repro/sample_test.go:22 +0xaa
exit status 2
FAIL    _/home/rina/repro   0.030s
@minux
Copy link
Member

@minux minux commented Dec 15, 2014

It seems this is due to two goroutines running runtime.Stack() concurrently
and both are trying to stop the world.

@bradfitz
Copy link
Contributor

@bradfitz bradfitz commented Dec 15, 2014

@t-yuki
Copy link

@t-yuki t-yuki commented Dec 15, 2014

It occurs even if the other goroutine is not runtime.Stack(). Extremely, it crashes with runtime.GC(), wow!

In src/runtime/mprof.go#L581, goroutine X does acquirem then semacquire worldsema without releasem.
If another goroutine Y has worldsema, goroutine X get back to the queue at semacquire and wait for schedule.
But goroutine X still holds m->locks so it crashes.

I'm not sure for runtime, but it seems that it should releasem before calling semacquire and call acquirem again during runtime.Stack() func.

@randall77
Copy link
Contributor

@randall77 randall77 commented Dec 15, 2014

Fix at tip pending here: https://go-review.googlesource.com/1600

I'll work on a 1.4 patch as well.

@gopherbot gopherbot closed this in 50bc3d5 Dec 16, 2014
@rsc rsc added this to the Go1.4.1 milestone Dec 18, 2014
@rsc rsc removed the release-go1.4.1 label Dec 18, 2014
rsc added a commit that referenced this issue Jan 14, 2015
It shouldn't semacquire() inside an acquirem(), the runtime
thinks that means deadlock.  It actually isn't a deadlock, but it
looks like it because acquirem() does m.locks++.

Candidate for inclusion in 1.4.1.  runtime.Stack with all=true
is pretty unuseable in GOMAXPROCS>1 environment.

fixes #9321

Change-Id: Iac6b664217d24763b9878c20e49229a1ecffc805
Reviewed-on: https://go-review.googlesource.com/1600
Reviewed-by: Dmitry Vyukov <dvyukov@google.com>
(cherry picked from commit 50bc3d5)
Reviewed-on: https://go-review.googlesource.com/2807
Reviewed-by: Andrew Gerrand <adg@golang.org>
@golang golang locked and limited conversation to collaborators Jun 25, 2016
wheatman added a commit to wheatman/go-akaros that referenced this issue Jun 25, 2018
It shouldn't semacquire() inside an acquirem(), the runtime
thinks that means deadlock.  It actually isn't a deadlock, but it
looks like it because acquirem() does m.locks++.

Candidate for inclusion in 1.4.1.  runtime.Stack with all=true
is pretty unuseable in GOMAXPROCS>1 environment.

fixes golang#9321

Change-Id: Iac6b664217d24763b9878c20e49229a1ecffc805
Reviewed-on: https://go-review.googlesource.com/1600
Reviewed-by: Dmitry Vyukov <dvyukov@google.com>
(cherry picked from commit 50bc3d5)
Reviewed-on: https://go-review.googlesource.com/2807
Reviewed-by: Andrew Gerrand <adg@golang.org>
wheatman added a commit to wheatman/go-akaros that referenced this issue Jun 26, 2018
It shouldn't semacquire() inside an acquirem(), the runtime
thinks that means deadlock.  It actually isn't a deadlock, but it
looks like it because acquirem() does m.locks++.

Candidate for inclusion in 1.4.1.  runtime.Stack with all=true
is pretty unuseable in GOMAXPROCS>1 environment.

fixes golang#9321

Change-Id: Iac6b664217d24763b9878c20e49229a1ecffc805
Reviewed-on: https://go-review.googlesource.com/1600
Reviewed-by: Dmitry Vyukov <dvyukov@google.com>
(cherry picked from commit 50bc3d5)
Reviewed-on: https://go-review.googlesource.com/2807
Reviewed-by: Andrew Gerrand <adg@golang.org>
wheatman added a commit to wheatman/go-akaros that referenced this issue Jul 9, 2018
It shouldn't semacquire() inside an acquirem(), the runtime
thinks that means deadlock.  It actually isn't a deadlock, but it
looks like it because acquirem() does m.locks++.

Candidate for inclusion in 1.4.1.  runtime.Stack with all=true
is pretty unuseable in GOMAXPROCS>1 environment.

fixes golang#9321

Change-Id: Iac6b664217d24763b9878c20e49229a1ecffc805
Reviewed-on: https://go-review.googlesource.com/1600
Reviewed-by: Dmitry Vyukov <dvyukov@google.com>
(cherry picked from commit 50bc3d5)
Reviewed-on: https://go-review.googlesource.com/2807
Reviewed-by: Andrew Gerrand <adg@golang.org>
wheatman added a commit to wheatman/go-akaros that referenced this issue Jul 20, 2018
It shouldn't semacquire() inside an acquirem(), the runtime
thinks that means deadlock.  It actually isn't a deadlock, but it
looks like it because acquirem() does m.locks++.

Candidate for inclusion in 1.4.1.  runtime.Stack with all=true
is pretty unuseable in GOMAXPROCS>1 environment.

fixes golang#9321

Change-Id: Iac6b664217d24763b9878c20e49229a1ecffc805
Reviewed-on: https://go-review.googlesource.com/1600
Reviewed-by: Dmitry Vyukov <dvyukov@google.com>
(cherry picked from commit 50bc3d5)
Reviewed-on: https://go-review.googlesource.com/2807
Reviewed-by: Andrew Gerrand <adg@golang.org>
wheatman added a commit to wheatman/go-akaros that referenced this issue Jul 30, 2018
It shouldn't semacquire() inside an acquirem(), the runtime
thinks that means deadlock.  It actually isn't a deadlock, but it
looks like it because acquirem() does m.locks++.

Candidate for inclusion in 1.4.1.  runtime.Stack with all=true
is pretty unuseable in GOMAXPROCS>1 environment.

fixes golang#9321

Change-Id: Iac6b664217d24763b9878c20e49229a1ecffc805
Reviewed-on: https://go-review.googlesource.com/1600
Reviewed-by: Dmitry Vyukov <dvyukov@google.com>
(cherry picked from commit 50bc3d5)
Reviewed-on: https://go-review.googlesource.com/2807
Reviewed-by: Andrew Gerrand <adg@golang.org>
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
8 participants
You can’t perform that action at this time.