2.6 deadlocked loading targets page

[tomwilkie]

See https://gist.github.com/tomwilkie/43b99a28cebe39d22e7c3b6e12c545bd for stack

[tomwilkie]

goroutine 5873214 [semacquire, 2 minutes]:
sync.runtime_SemacquireMutex(0xc0004e434c, 0xc077f91300)
	/usr/local/Cellar/go/1.11.2/libexec/src/runtime/sema.go:71 +0x3d
sync.(*Mutex).Lock(0xc0004e4348)
	/usr/local/Cellar/go/1.11.2/libexec/src/sync/mutex.go:134 +0xff
github.com/prometheus/prometheus/scrape.(*Manager).TargetsActive(0xc0004e4320, 0x0)
	/Users/twilkie/Documents/src/github.com/prometheus/prometheus/scrape/manager.go:189 +0x57
github.com/prometheus/prometheus/web.(*Handler).targets(0xc000468d00, 0x7fafc97e78a8, 0xc0d7861d60, 0xc0017bc300)
	/Users/twilkie/Documents/src/github.com/prometheus/prometheus/web/web.go:708 +0x51
github.com/prometheus/prometheus/web.(*Handler).targets-fm(0x7fafc97e78a8, 0xc0d7861d60, 0xc0017bc300)
	/Users/twilkie/Documents/src/github.com/prometheus/prometheus/web/web.go:286 +0x48
github.com/prometheus/prometheus/web.(*Handler).testReady.func1(0x7fafc97e78a8, 0xc0d7861d60, 0xc0017bc300)
	/Users/twilkie/Documents/src/github.com/prometheus/prometheus/web/web.go:404 +0x55

[tomwilkie]

Lock: https://github.com/prometheus/prometheus/blob/v2.6.0/scrape/manager.go#L189

Looks like its ApplyConfig thats holding the lock, and it itself is blocked in scrapePool.reload on a wg.Wait, which I suspect is blocked on a old scrape look stopping.

goroutine 119 [semacquire, 3 minutes]:
sync.runtime_Semacquire(0xc0e389e638)
	/usr/local/Cellar/go/1.11.2/libexec/src/runtime/sema.go:56 +0x39
sync.(*WaitGroup).Wait(0xc0e389e630)
	/usr/local/Cellar/go/1.11.2/libexec/src/sync/waitgroup.go:130 +0x64
github.com/prometheus/prometheus/scrape.(*scrapePool).reload(0xc0038a4c80, 0xc0fb300240)
	/Users/twilkie/Documents/src/github.com/prometheus/prometheus/scrape/scrape.go:271 +0x4c8
github.com/prometheus/prometheus/scrape.(*Manager).ApplyConfig(0xc0004e4320, 0xc0909c2d80, 0x0, 0x0)
	/Users/twilkie/Documents/src/github.com/prometheus/prometheus/scrape/manager.go:167 +0x259
github.com/prometheus/prometheus/scrape.(*Manager).ApplyConfig-fm(0xc0909c2d80, 0x0, 0x0)
	/Users/twilkie/Documents/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:336 +0x34
main.reloadConfig(0x7ffe28fa96e3, 0x1e, 0x1e99b60, 0xc000690a50, 0xc0006949c0, 0x7, 0x7, 0x0, 0x0)
	/Users/twilkie/Documents/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:649 +0x228
main.main.func13(0x0, 0x0)
	/Users/twilkie/Documents/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:490 +0x1e6
github.com/oklog/oklog/pkg/group.(*Group).Run.func1(0xc0002b6540, 0xc0002b6480, 0xc0001d4140)
	/Users/twilkie/Documents/src/github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group/group.go:38 +0x27
created by github.com/oklog/oklog/pkg/group.(*Group).Run
	/Users/twilkie/Documents/src/github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group/group.go:37 +0xbe

[tomwilkie]

There are 162 scrapeloops, anecdotally a bunch are blocks trying to append to the remote write queues.

I don't see whos holding that lock yet.

[simonpasquier]

Have you been able to troubleshoot the problem?

[mmerrill3]

I'm being hit by this issue as well. +1

[tomwilkie]

Yeah I’ve had reports from others too. We’re investigating.

…

On Fri, 18 Jan 2019 at 19:48, Michael Merrill ***@***.***> wrote: I'm being hit by this issue as well. +1 — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#5082 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAbGhUcuQPKv-j9sMYE0aDPz5aU10q3nks5vEiUUgaJpZM4Z1jrC> .

[mmerrill3]

I've also seen it when the remote write queues are full. When I turn of remote writes, its all good, no issues.

This bit looks suspicious to me, especially if enqueue fails.

prometheus/storage/remote/queue_manager.go

Lines 222 to 224 in 24f19f0

	t.shardsMtx.RLock()
	enqueued := t.shards.enqueue(&snew)
	t.shardsMtx.RUnlock()

[michael-doubez]

I have also had this issue; in particular when changing a target from one job to another.

[simonpasquier]

@tomwilkie is this issue still relevant considering that the remote write code has changed a lot in 2.8?

[michael-doubez]

Since I switched to 2.8.1, I now longer have slowness and freeze when reloading.

[simonpasquier]

Closing, feel free to re-open if it still occurs with 2.9.