Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: panic fatal error: malloc deadlock #24059

Closed
dgryski opened this issue Feb 23, 2018 · 11 comments

Comments

Projects
None yet
8 participants
@dgryski
Copy link
Contributor

commented Feb 23, 2018

What version of Go are you using (go version)?

I've bisected the issue to 2edc4d4

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

darwin/amd64

What did you do?

During development of a large internal service, we were running our test suite. The tests triggered a nil pointer dereference. On 1.9.4 the output was the usual

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1300854]

But on 1.10 we instead got

panic: fatal error: malloc deadlock
panic during panic

runtime stack:
runtime.startpanic_m()
	/Users/dgryski/work/src/go/src/runtime/panic.go:693 +0x179
runtime.systemstack(0x0)
	/Users/dgryski/work/src/go/src/runtime/asm_amd64.s:409 +0x79
runtime.mstart()
	/Users/dgryski/work/src/go/src/runtime/proc.go:1170

goroutine 207 [running]:
runtime.systemstack_switch()
	/Users/dgryski/work/src/go/src/runtime/asm_amd64.s:363 fp=0xc420373a88 sp=0xc420373a80 pc=0x1057fe0
runtime.startpanic()
	/Users/dgryski/work/src/go/src/runtime/panic.go:592 +0x1e fp=0xc420373aa0 sp=0xc420373a88 pc=0x102c8be
runtime.throw(0x1367aa4, 0xf)
	/Users/dgryski/work/src/go/src/runtime/panic.go:618 +0x74 fp=0xc420373ac0 sp=0xc420373aa0 pc=0x102c9e4
runtime.mallocgc(0x2010, 0x0, 0x1303401, 0x12ae3a8)
	/Users/dgryski/work/src/go/src/runtime/malloc.go:621 +0x99d fp=0xc420373b60 sp=0xc420373ac0 pc=0x10126cd
runtime.itabAdd(0x2541840)
	/Users/dgryski/work/src/go/src/runtime/iface.go:122 +0x83 fp=0xc420373bb0 sp=0xc420373b60 pc=0x100f403
runtime.getitab(0x1303480, 0x12e2680, 0xc4202e4301, 0xc4202e43b0)
	/Users/dgryski/work/src/go/src/runtime/iface.go:70 +0x404 fp=0xc420373c28 sp=0xc420373bb0 pc=0x100f284
runtime.assertE2I2(0x1303480, 0x12e2680, 0xc42022ad30, 0x7, 0xc400000007, 0xc420373cb8)
	/Users/dgryski/work/src/go/src/runtime/iface.go:592 +0x43 fp=0xc420373c58 sp=0xc420373c28 pc=0x10107e3
runtime.printany(0x12e2680, 0xc42022ad30)
	/Users/dgryski/work/src/go/src/runtime/error.go:77 +0x69 fp=0xc420373d18 sp=0xc420373c58 pc=0x1007709
runtime.printpanics(0xc420373da0)
	/Users/dgryski/work/src/go/src/runtime/panic.go:420 +0x71 fp=0xc420373d38 sp=0xc420373d18 pc=0x102c101
panic(0x12fd360, 0x1528330)
	/Users/dgryski/work/src/go/src/runtime/panic.go:553 +0x3b4 fp=0xc420373dd8 sp=0xc420373d38 pc=0x102c544
runtime.panicmem()
	/Users/dgryski/work/src/go/src/runtime/panic.go:63 +0x5e fp=0xc420373df8 sp=0xc420373dd8 pc=0x102b24e
runtime.sigpanic()
	/Users/dgryski/work/src/go/src/runtime/signal_unix.go:388 +0x17a fp=0xc420373e48 sp=0xc420373df8 pc=0x1041fca
github.com/fastly/service1/service2.(*GetStats).Add(...)
	/Users/dgryski/go/src/github.com/fastly/service1/service2/client.go:222
github.com/fastly/service1/service2.(*Client).GetAll.func2()
	/Users/dgryski/go/src/github.com/fastly/service1/service2/client.go:276 +0x27d fp=0xc420373fe0 sp=0xc420373e48 pc=0x1292c8d
runtime.goexit()
	/Users/dgryski/work/src/go/src/runtime/asm_amd64.s:2361 +0x1 fp=0xc420373fe8 sp=0xc420373fe0 pc=0x105ab31
created by github.com/fastly/service1/service2.(*Client).GetAll
	/Users/dgryski/go/src/github.com/fastly/service1/service2/client.go:285 +0x232

[ output trimmed of all the other goroutine stack traces]

Some notes from trying to minimize this into a useful test case:

  • There are lots of goroutines running, and the nil dereference happens in one
  • When running with -test.v, the normal panic output is produced
  • I need to run multiple tests from this package in addition to the failing test. Running just the one test that fails produced the normal panic output.
  • Adding -race doesn't make a difference

@mvdan mvdan added this to the Go1.10.1 milestone Feb 23, 2018

@mvdan mvdan added the NeedsFix label Feb 23, 2018

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

commented Feb 23, 2018

We crash in the unlikely case that we have to grow the itab table while adding an itab entry for converting string to stringer. I will send a CL.

@gopherbot

This comment has been minimized.

Copy link

commented Feb 23, 2018

Change https://golang.org/cl/96755 mentions this issue: runtime: in prepanic, ensure printany doesn't allocate printing a string

@dgryski

This comment has been minimized.

Copy link
Contributor Author

commented Feb 23, 2018

I can confirm this CL fixes my issue.

@andybons

This comment has been minimized.

Copy link
Member

commented Mar 26, 2018

Reopening for cherry-pick

@andybons

This comment has been minimized.

Copy link
Member

commented Mar 27, 2018

CL 96755 OK for Go 1.10.1

@gopherbot

This comment has been minimized.

Copy link

commented Mar 27, 2018

Change https://golang.org/cl/102781 mentions this issue: [release-branch.go1.10] runtime: don't check for String/Error methods in printany

gopherbot pushed a commit that referenced this issue Mar 29, 2018

[release-branch.go1.10] runtime: don't check for String/Error methods…
… in printany

They have either already been called by preprintpanics, or they can
not be called safely because of the various conditions checked at the
start of gopanic.

Fixes #24059

Change-Id: I4a6233d12c9f7aaaee72f343257ea108bae79241
Reviewed-on: https://go-review.googlesource.com/96755
Reviewed-by: Austin Clements <austin@google.com>
Reviewed-on: https://go-review.googlesource.com/102781
Run-TryBot: Andrew Bonventre <andybons@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>

@andybons andybons closed this Mar 29, 2018

grobie added a commit to prometheus/golang-builder that referenced this issue Mar 31, 2018

Use go1.10.1 and go1.9.5
The latest patch versions include two important fixes for the Prometheus
project:

* golang/go#23995
* golang/go#24059

grobie added a commit to prometheus/golang-builder that referenced this issue Mar 31, 2018

Use go1.10.1 and go1.9.5
The latest patch versions include two important fixes for the Prometheus
project:

* golang/go#23995
* golang/go#24059

grobie added a commit to prometheus/golang-builder that referenced this issue Mar 31, 2018

Use go1.10.1 and go1.9.5 (#39)
The latest patch versions include two important fixes for the Prometheus
project:

* golang/go#23995
* golang/go#24059
@SuperQ

This comment has been minimized.

Copy link

commented Apr 27, 2018

We're still seeing issues that appear to be related on go1.10.1, should we file a separate issue or reopen?

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

commented Apr 27, 2018

@SuperQ Please file a new issue. We're pretty sure this problem is fixed. I would guess that you are seeing a different problem with similar symptoms.

@denismakogon

This comment has been minimized.

Copy link

commented May 1, 2018

faced with this type of error while was doing something like this:

a function that generates a slice of int64 items:

func makeInt64Range(min, max int64) []int64 {
	a := make([]int64, max-min+1)
	for i := range a {
		a[i] = min + int64(i)
	}
	return a
}

accidentally (was distracted by random convo) it was called with mismatched boundaries:

	int64Range := makeInt64Range(1025, 1000)

and got this:

panic: fatal error: malloc deadlock
panic during panic

runtime stack:
runtime.startpanic_m()
	/usr/local/go/src/runtime/panic.go:693 +0x179
runtime.systemstack(0x0)
	/usr/local/go/src/runtime/asm_amd64.s:409 +0x79
runtime.mstart()
	/usr/local/go/src/runtime/proc.go:1170

Golang comes from: golang:1.10-alpine3.7 docker image.

Tried that on playground: https://play.golang.org/p/D5j5EzDgNiW
Error is: panic: runtime error: makeslice: len out of range which appear to be different from what i have.

@aclements

This comment has been minimized.

Copy link
Member

commented May 1, 2018

@denismakogon, please open a new issue, thanks!

@denismakogon

This comment has been minimized.

Copy link

commented May 1, 2018

@aclements thanks, it appears that fixed was merged somewhere on the road to 1.10.2 since i don't get malloc error on the playground (which is 1.10.2). Will test on golang:1.10.2-alpine3.7 (when that will be available) and if it's still there I will definitely open an issue.

@golang golang locked and limited conversation to collaborators May 1, 2019

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.