Prometheus v2.2.0 deadlocked loading targets page

[tomwilkie]

Heres the stack trace: https://gist.github.com/tomwilkie/af4ffa4c8f50f723b55f8d4fd82f7c9c

I'm diagnosing...

[tomwilkie]

Looks like its blocked here: https://github.com/prometheus/prometheus/blob/master/scrape/manager.go#L120

[fabxc]

@krasi-georgiev that has good odds of being a regression from the scrape refactoring. Over the years we had more deadlocks caused by loading target page than I can count – and they all came somewhere from there :)

[tomwilkie]

Sorry I didn't add more background:

• I think I hit the targets page very quickly after the instance came up.
• I was debugging something else so I got the stacks and then threw the instance away, and it didn't reoccur.
• This was with the v2.2.0 docker release image.
• The instance was around for about 5mins and other endpoints worked, but couldn't load the targets page for the entire time.
• It didn't look like it had managed to find any targets though, as there was no data in it.

[krasi-georgiev]

I will have a look when I get bored poking with tsdb 👍

[tomwilkie]

There are a bunch of goroutines all blocked at roughly the same point - goroutine 1114, 1293, 1118. Goroutine 1118 is holding m.mtx write lock, and blocked on the getting the scrape pool read lock. The scrape pool write lock is being held by scrapePool.Sync (goroutine 183) which is waiting for scrapers to stop. The scrapers in turn are blocked on remote write, which is blocked on resharding, which is blocked on the remote write bug I was trying to fix (#3809).

[tomwilkie]

As 3809 is fixed, looks like nothing to fix here.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.