it takes too much time to stop prometheus using SIGTERM

[guanglinlv]

hi,

it will takes about one hour to stop prometheus using kill -SIGTERM <PID>. most of the time is stopping remote storage.

• startup falgs

• prometheus log

• /debug/pprof/goroutine?debug=1

https://gist.github.com/guanglinlv/ebdc1e237c261b9fda5d70fb6ada3d52

• /debug/pprof/goroutine?debug=2 partial

https://gist.github.com/guanglinlv/543c57646b4c30c57d129e619afd18d0

the dump is very large, ~ 25M, so i can not put it here. but i can give some summary

there are 54690 goroutines, sendSamples take 54628 goroutines.

any help? thanks.

[brian-brazil]

It sounds like you've found the cause of #1549. Do you also have unbounded memory growth?

I'd consider 5 minutes to be the upper limits on stopping local storage though, so a 5m shutdown isn't necessarily a problem.

[guanglinlv]

@brian-brazil , the memory is normal, ~3.8G. it takes one hour not 5 minutes to stop prometheus.

i think too many sendSamples goroutines is there, so the stopping(remote storage) will be blocked.

https://github.com/prometheus/prometheus/blob/master/storage/remote/queue_manager.go#L149-L156

[brian-brazil]

Yeah, that code needs to be rewritten to keep the number of goroutines bounded to maxConcurrentSends.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.