Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: "schedule: holding locks" #44544

bcmills opened this issue Feb 23, 2021 · 9 comments

runtime: "schedule: holding locks" #44544

bcmills opened this issue Feb 23, 2021 · 9 comments


Copy link

@bcmills bcmills commented Feb 23, 2021

##### GOMAXPROCS=2 runtime -cpu=1,2,4 -quick
fatal error: fatal error: schedule: holding locks
fatal error: unexpected signal during runtime execution
panic during panic
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x0]

runtime stack:
runtime.throw(0x6aede6, 0x2a)
	/workdir/go/src/runtime/panic.go:1126 +0x74 fp=0x7ff7dc87ebe0 sp=0x7ff7dc87ebb0 pc=0x43f2f4
runtime: unexpected return pc for runtime.sigpanic called from 0x0
stack: frame={sp:0x7ff7dc87ebe0, fp:0x7ff7dc87ec40} stack=[0x7ff7dc07f280,0x7ff7dc87ee80)
0x00007ff7dc87eae0:  0x000000c000440300  0x0100000000000013 
0x00007ff7dc87eaf0:  0x000000000000000e  0x000000000000001f 
0x00007ff7dc87eb00:  0x0000000000000000  0x0000000000000000 
0x00007ff7dc87eb10:  0x0000000000000001  0x00000000006aa7d3 
0x00007ff7dc87eb20:  0x00007ff7dc87eb68  0x000000000043f5ff <runtime.fatalthrow.func1+0x000000000000005f> 
0x00007ff7dc87eb30:  0x000000c000440300  0x000000000043f2f4 <runtime.throw+0x0000000000000074> 
0x00007ff7dc87eb40:  0x00007ff7dc87ebb0  0x0000000000000001 
0x00007ff7dc87eb50:  0x00007ff7dc87ebb0  0x000000000043f2f4 <runtime.throw+0x0000000000000074> 
0x00007ff7dc87eb60:  0x000000c000440300  0x00007ff7dc87eba0 
0x00007ff7dc87eb70:  0x000000000043f57e <runtime.fatalthrow+0x000000000000005e>  0x00007ff7dc87eb80 
0x00007ff7dc87eb80:  0x000000000043f5a0 <runtime.fatalthrow.func1+0x0000000000000000>  0x000000c000440300 
0x00007ff7dc87eb90:  0x000000000043f2f4 <runtime.throw+0x0000000000000074>  0x00007ff7dc87ebb0 
0x00007ff7dc87eba0:  0x00007ff7dc87ebd0  0x000000000043f2f4 <runtime.throw+0x0000000000000074> 
0x00007ff7dc87ebb0:  0x00007ff7dc87ebb8  0x000000000043f320 <runtime.throw.func1+0x0000000000000000> 
0x00007ff7dc87ebc0:  0x00000000006aede6  0x000000000000002a 
0x00007ff7dc87ebd0:  0x00007ff7dc87ec30  0x0000000000457bcc <runtime.sigpanic+0x000000000000050c> 
0x00007ff7dc87ebe0: <0x00000000006aede6  0x000000000000002a 
0x00007ff7dc87ebf0:  0x0000000000440ea5 <runtime.gwrite+0x00000000000000a5>  0x0000000000599d55 <runtime_test.TestDebugCallGC+0x0000000000000095> 
0x00007ff7dc87ec00:  0x00000080000601ef  0x0000000000000000 
0x00007ff7dc87ec10:  0x000000000000000d  0x0000000000000000 
0x00007ff7dc87ec20:  0x0000000000000000  0x0000000000000000 
0x00007ff7dc87ec30:  0x0000000000000000 !0x0000000000000000 
0x00007ff7dc87ec40: >0x0000000000000000  0x0000000000470105 <runtime.InjectDebugCall+0x0000000000000205> 
0x00007ff7dc87ec50:  0x000000000043b7c5 <runtime.signalM+0x0000000000000045>  0x0000000000014f29 
0x00007ff7dc87ec60:  0x0000000000014f43  0x0000000000000017 
0x00007ff7dc87ec70:  0x00007ff7dc87ec90  0x000000000044e965 <runtime.preemptone+0x00000000000000c5> 
0x00007ff7dc87ec80:  0x000000c000108400  0x0000000000000017 
0x00007ff7dc87ec90:  0x00007ff7dc87ecd0  0x000000000044e865 <runtime.preemptall+0x0000000000000065> 
0x00007ff7dc87eca0:  0x000000c00002b800  0x0000000000000001 
0x00007ff7dc87ecb0:  0x0100000000000000  0x0000000000000003 
0x00007ff7dc87ecc0:  0x0000000000000000  0x00000000000f4240 
0x00007ff7dc87ecd0:  0x00007ff7dc87ecf0  0x000000000044399f <runtime.freezetheworld+0x000000000000003f> 
0x00007ff7dc87ece0:  0x00000000000003e8  0x0000000000000000 
0x00007ff7dc87ecf0:  0x00007ff7dc87ed18  0x000000000043f935 <runtime.startpanic_m+0x0000000000000155> 
0x00007ff7dc87ed00:  0x00000000008654c0  0x00000000006b4ad8 
0x00007ff7dc87ed10:  0x0000000000000001  0x00007ff7dc87ed60 
0x00007ff7dc87ed20:  0x000000000043f5dd <runtime.fatalthrow.func1+0x000000000000003d>  0x0000000000000001 
0x00007ff7dc87ed30:  0x00000000006b4ad8  0x0000000000000001 
	/workdir/go/src/runtime/signal_unix.go:719 +0x50c fp=0x7ff7dc87ec40 sp=0x7ff7dc87ebe0 pc=0x457bcc


CC @danscales @cherrymui @mknyszek @jeremyfaller

@bcmills bcmills added the NeedsInvestigation label Feb 23, 2021
@bcmills bcmills added this to the Go1.17 milestone Feb 23, 2021
Copy link

@danscales danscales commented Feb 23, 2021

@prattmic in case you have any thoughts, since you have been looking at the scheduling code and at held locks recently

Copy link

@prattmic prattmic commented Feb 24, 2021

I still need to look more closely at this, but there is a lot of suspicious behavior going on:

  1. This is an InjectDebugCall test.
  2. The test seems to be panicking on a g.m.locks++ line in acquirem() (bad g.m pointer?).
  3. The stack dump indicates preemptone calls at some point (though probably after the first throw?).

Copy link

@mknyszek mknyszek commented Apr 15, 2021

I reworked some of the details re: how InjectDebugCall interacts with the scheduler, so if this isn't showing up anymore that might be one reason why.

Copy link

@dmitshur dmitshur commented May 21, 2021

It seems there haven't been more occurrences reported. @prattmic Do you expect to look more at this in time for Go 1.17, or should we move this to Backlog if this can happen later on?

@prattmic prattmic removed this from the Go1.17 milestone Jun 1, 2021
@prattmic prattmic added this to the Backlog milestone Jun 1, 2021
Copy link
Member Author

@bcmills bcmills commented Nov 17, 2021

(None of the above failures seem to have anything to do with InjectDebugCall or linux-amd64, though.)

@bcmills bcmills changed the title runtime: "schedule: holding locks" on linux-amd64 runtime: "schedule: holding locks" Jan 12, 2022
Copy link
Member Author

@bcmills bcmills commented Jan 12, 2022

(Hmm, probably work opening a separate issue for that, since it has a separate root cause.)

Closing as obsolete per #44544 (comment).

@bcmills bcmills closed this as completed Jan 12, 2022
Copy link
Member Author

@bcmills bcmills commented Jan 12, 2022

(Filed plan9-arm as #50560.)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
None yet

No branches or pull requests

5 participants