Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: periodic crash in runtime/asm_amd64.s:623 #27978

rcoreilly opened this issue Oct 3, 2018 · 7 comments

runtime: periodic crash in runtime/asm_amd64.s:623 #27978

rcoreilly opened this issue Oct 3, 2018 · 7 comments


Copy link

@rcoreilly rcoreilly commented Oct 3, 2018

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go version go1.11 darwin/amd64

Does this issue reproduce with the latest release?


What operating system and processor architecture are you using (go env)?

MacOS 10.13.6 Macbook Pro, amd64

What did you do?

calling cocoa api using cgo for gui app: -- I tried to make a small standalone app to reproduce it and was unable to reproduce:

If possible, provide a recipe for reproducing the error.
A complete runnable program is good.
A link on is best.

What did you expect to see?

no crashing :)

What did you see instead?

crashing :( This is very random -- mostly works fine but randomly crashes. I have attempted to protect everything with mutexes etc and nothing seems to make any difference. I have no idea what the proximal cause of the crash is -- can't seem to get good debugging in asm.s file or in the call above it. Couldn't find similar issues. Any ideas as to what might be going on here and how I could get more useful debugging info? Is it just some kind of random memory corruption? The C.doCloseWindow call is protected by a mutex that is used for all cgo calls. The "goroutines" list shows that a few other routines are at this exact same point in asm_amd64 -- both are in same thread..

func closeWindow(id uintptr) {
	w :=[id]
	C.doCloseWindow(C.uintptr_t(id))  <- cgo call causing crash; w, id both look good / normal

Command failed: bad access
> runtime.asmcgocall() /usr/local/Cellar/go/1.11/libexec/src/runtime/asm_amd64.s:623 (PC: 0x4061232)
Warning: debugging optimized function
   618:		JEQ	nosave
   620:		// Switch to system stack.
   621:		MOVQ	m_g0(R8), SI
   622:		CALL	gosave<>(SB)
=> 623:		MOVQ	SI, g(CX)
   624:		MOVQ	(g_sched+gobuf_sp)(SI), SP
   626:		// Now on a scheduling stack (a pthread-created stack).
   627:		// Make sure we have enough room for 4 stack-backed fast-call
   628:		// registers as per windows amd64 calling convention.

(dlv) bt
 0  0x0000000004061232 in runtime.asmcgocall
    at /usr/local/Cellar/go/1.11/libexec/src/runtime/asm_amd64.s:623
 1  0x0000000004005672 in runtime.cgocall
    at /usr/local/Cellar/go/1.11/libexec/src/runtime/cgocall.go:131
 2  0x0000000004884675 in
    at _cgo_gotypes.go:334
 3  0x0000000004887f7a in
    at /Users/oreilly/go/src/
 4  0x0000000004883db1 in*windowImpl).Close
    at /Users/oreilly/go/src/
 5  0x00000000043fc603 in*Dialog).Close
    at /Users/oreilly/go/src/
 6  0x00000000043fc711 in*Dialog).Accept
    at /Users/oreilly/go/src/
 7  0x000000000448cfc5 in*Dialog).Open.func2
    at /Users/oreilly/go/src/
 8  0x0000000004482006 in*WinEventRecv).Call
    at /Users/oreilly/go/src/
 9  0x0000000004482f17 in*Window).SendEventSignal
    at /Users/oreilly/go/src/
10  0x0000000004480549 in*Window).EventLoop
    at /Users/oreilly/go/src/

Frame 1: /usr/local/Cellar/go/1.11/libexec/src/runtime/cgocall.go:131 (PC: 4005672)
   126:		// and then re-enter the "system call" reusing the PC and SP
   127:		// saved by entersyscall here.
   128:		entersyscall()
   130:		mp.incgo = true
=> 131:		errno := asmcgocall(fn, arg)
   133:		// Call endcgo before exitsyscall because exitsyscall may
   134:		// reschedule us on to a different M.
   135:		endcgo(mp)
(dlv) p fn 
(unreadable could not find loclist entry at 0xc4e for address 0x4005671)
(dlv) p mp
(unreadable could not find loclist entry at 0xc71 for address 0x4005671)

I get the same "could not find loclist entry.." errors in the other routines..

Thanks for any insights!

@agnivade agnivade changed the title periodic crash in runtime/asm_amd64.s:623 runtime: periodic crash in runtime/asm_amd64.s:623 Oct 3, 2018
Copy link

@agnivade agnivade commented Oct 3, 2018

Can you try with 1.10 and see if it crashes there too ? It would be good to see if this is a new bug or was always there.

Copy link

@rcoreilly rcoreilly commented Oct 3, 2018

yes it does happen with 1.10 -- just got it. Line 674 in this version:

Frame 0: /usr/local/Cellar/go@1.10/1.10.4/libexec/src/runtime/asm_amd64.s:674 (PC: 405e912)
   669:		JEQ	nosave
   671:		// Switch to system stack.
   672:		MOVQ	m_g0(R8), SI
   673:		CALL	gosave<>(SB)
=> 674:		MOVQ	SI, g(CX)
   675:		MOVQ	(g_sched+gobuf_sp)(SI), SP
Copy link

@rcoreilly rcoreilly commented Oct 3, 2018

I turned off the cursor blinking which uses a time.NewTicker -- that is what was those other routines sitting at asm_amd64.s:623 were. But just got the crash again. So that doesn't seem relevant, which is consistent with the "bug" version that also fails to replicate the crash, despite starting a lot of those tickers..

Anyway, would definitely appreciate if there is anything you could tell me about WHY it might be randomly crashing so reliably at this one place (i.e., what is that line of code doing, and how can I figure out what bit of memory is now corrupted / wrong etc to cause the crash)? The random nature of the crash combined with the fact that it ALWAYS happens at this one particular line of code seems like it can't be just pure memory corruption...

  Goroutine 1 - User: _cgo_gotypes.go:688 (0x4885af1)
  Goroutine 2 - User: /usr/local/Cellar/go/1.11/libexec/src/runtime/proc.go:303 runtime.gopark (0x4033ed4)
  Goroutine 3 - User: /Users/oreilly/go/src/ (0x48877e4)
  Goroutine 4 - User: /usr/local/Cellar/go/1.11/libexec/src/runtime/sema.go:510 sync.runtime_notifyListWait (0x404633b)
  Goroutine 5 - User: /usr/local/Cellar/go/1.11/libexec/src/runtime/proc.go:303 runtime.gopark (0x4033ed4)
  Goroutine 18 - User: /usr/local/Cellar/go/1.11/libexec/src/runtime/proc.go:303 runtime.gopark (0x4033ed4)
  Goroutine 19 - User: /usr/local/Cellar/go/1.11/libexec/src/runtime/proc.go:303 runtime.gopark (0x4033ed4)
  Goroutine 20 - User: /usr/local/Cellar/go/1.11/libexec/src/runtime/proc.go:303 runtime.gopark (0x4033ed4)
  Goroutine 21 - User: /usr/local/Cellar/go/1.11/libexec/src/runtime/proc.go:303 runtime.gopark (0x4033ed4)
  Goroutine 22 - User: /usr/local/Cellar/go/1.11/libexec/src/runtime/proc.go:303 runtime.gopark (0x4033ed4)
  Goroutine 34 - User: /usr/local/Cellar/go/1.11/libexec/src/runtime/proc.go:303 runtime.gopark (0x4033ed4)
  Goroutine 35 - User: /usr/local/Cellar/go/1.11/libexec/src/runtime/proc.go:303 runtime.gopark (0x4033ed4)
  Goroutine 36 - User: /usr/local/Cellar/go/1.11/libexec/src/runtime/proc.go:303 runtime.gopark (0x4033ed4)
  Goroutine 37 - User: /usr/local/Cellar/go/1.11/libexec/src/runtime/proc.go:303 runtime.gopark (0x4033ed4)
  Goroutine 38 - User: /usr/local/Cellar/go/1.11/libexec/src/runtime/proc.go:303 runtime.gopark (0x4033ed4)
  Goroutine 50 - User: /usr/local/Cellar/go/1.11/libexec/src/runtime/proc.go:303 runtime.gopark (0x4033ed4)
  Goroutine 51 - User: /usr/local/Cellar/go/1.11/libexec/src/runtime/proc.go:303 runtime.gopark (0x4033ed4)
  Goroutine 52 - User: /usr/local/Cellar/go/1.11/libexec/src/runtime/proc.go:303 runtime.gopark (0x4033ed4)
  Goroutine 66 - User: /usr/local/Cellar/go/1.11/libexec/src/runtime/sema.go:56 sync.runtime_Semacquire (0x4044f89)
  Goroutine 68 - User: /usr/local/Cellar/go/1.11/libexec/src/runtime/proc.go:303 runtime.gopark (0x4033ed4)
  Goroutine 82 - User: /usr/local/Cellar/go/1.11/libexec/src/runtime/proc.go:303 runtime.gopark (0x4033ed4)
  Goroutine 88 - User: /Users/oreilly/go/src/ (0x48877e4)
  Goroutine 160 - User: /usr/local/Cellar/go/1.11/libexec/src/runtime/sema.go:510 sync.runtime_notifyListWait (0x404633b)
  Goroutine 162 - User: /usr/local/Cellar/go/1.11/libexec/src/runtime/proc.go:303 runtime.gopark (0x4033ed4)
  Goroutine 658 - User: /usr/local/Cellar/go/1.11/libexec/src/runtime/proc.go:303 runtime.gopark (0x4033ed4)
  Goroutine 1050 - User: /usr/local/Cellar/go/1.11/libexec/src/runtime/netpoll.go:173 internal/poll.runtime_pollWait (0x402e8ae)
  Goroutine 2818 - User: _cgo_gotypes.go:334 (0x4884675)
  Goroutine 2819 - User: /Users/oreilly/go/src/*Window).SendEventSignal (0x4482c1c) (thread 1651889)
Copy link

@rcoreilly rcoreilly commented Oct 3, 2018

Also, it isn't just the close window call that causes the crash: this clipClear is the ONLY other one that also causes the crash. There is NOTHING special I can see about this clipClear call, so it is very puzzling.

(dlv) bt
 0  0x0000000004060ba2 in runtime.asmcgocall
    at /usr/local/Cellar/go/1.11/libexec/src/runtime/asm_amd64.s:623
 1  0x0000000004004fe2 in runtime.cgocall
    at /usr/local/Cellar/go/1.11/libexec/src/runtime/cgocall.go:131
 2  0x0000000004884291 in
    at _cgo_gotypes.go:237
 3  0x000000000488aaec in*clipImpl).Clear
    at /Users/oreilly/go/src/
 4  0x000000000488a89a in*clipImpl).Write
    at /Users/oreilly/go/src/
 5  0x00000000047e42be in*TextView).Copy
    at /Users/oreilly/go/src/
 6  0x00000000047ed946 in*TextView).KeyInput
    at /Users/oreilly/go/src/
 7  0x00000000048122ff in*TextView).TextViewEvents.func4
    at /Users/oreilly/go/src/
 8  0x0000000004481b16 in*WinEventRecv).Call
    at /Users/oreilly/go/src/
 9  0x0000000004482a27 in*Window).SendEventSignal
    at /Users/oreilly/go/src/
10  0x0000000004480059 in*Window).EventLoop
    at /Users/oreilly/go/src/
Copy link

@rcoreilly rcoreilly commented Oct 3, 2018

Actually, I think I might have found an issue with clipClear -- will test and report back later. If so, it suggests that this is a failure mode for multiple free in the objective C code -- clipClear was potentially freeing something that it didn't own anymore..

Copy link

@rcoreilly rcoreilly commented Oct 3, 2018

no crashing so far in clipClear -- it seems highly likely that this is just a double-free corruption kind of issue.

The one remaining question is: shouldn't this be a relatively common type of crash when using cgo, and would it be possible to add just a tiny bit of checking at the point of the crash, along with some more useful diagnostic message that this is likely some kind of c-side memory corruption issue?

Copy link

@rcoreilly rcoreilly commented Oct 6, 2018

in case anyone finds this later: the runtime/asm_amd64.s:623 appears to be some kind of wait state around the cgo call, and is NOT where the crash is happening -- it is just the delve doesn't give you accurate debugging info for the cgo part of the code. Now that I'm using lldb (couldn't get gdb working on mac despite all attempts), I can see where the crash is actually happening deep inside of AppKit.. Still no idea about what is actually causing the crash, but anyway it is not a go issue.

@rcoreilly rcoreilly closed this Oct 6, 2018
@golang golang locked and limited conversation to collaborators Oct 6, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
5 participants