dev-2.0 stuck trying to exit

[gouthamve]

Prometheus dev-2.0 branch did not exit after SIGTERM.

Relevant logs: https://gist.github.com/Gouthamve/ba1c52b09034a96d3b73fc85c8939011

I see fdatasync(17) in the middle which makes me think that Prometheus is stuck trying to flush.

cc @fabxc

[gouthamve]

Could grab the output of: http://localhost:9090/debug/pprof/goroutine?debug=2

Gisted here: https://gist.github.com/Gouthamve/ba1c52b09034a96d3b73fc85c8939011#file-goroutine-debug-2

[gouthamve]

Looking at the goroutine dump, I see that this could have been caused by #2629

This is because there are 2 goroutines stuck trying to acquire db.mtx and the exiting is blocked on the db.Close() call which is trying to acquire the lock.

The panic caused by the parser might not have let Querier gracefully RUnlock causing the deadlock.

[gouthamve]

This is due to a panic.

What potentially happened was we panicked due to a bug and that go-routine had acquired a lock. And we cannot gracefully exit until that lock is released. But as this is caused due to a panic, something which should never happen, can we close this?

[fabxc]

I think so yes. We definitely fixed a bunch of cases where we put lock releases into defer statements to prevent this. In general though, if it's a panic, these things might just happen.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.