High goroutine usage in serverset.go

[tommyulfsparre]

One of our prometheus instances panic:ed with runtime/cgo: out of memory in thread_start(possible due to #1223 ) and noticed that > 3k goroutines where in used by:

goroutine 101621016 [select, 191 minutes]:
github.com/prometheus/prometheus/retrieval/discovery.(*zookeeperTreeCache).recursiveNodeUpdate.func1(0xcd28eedf20, 0xc8203a9d40, 0xcbee1e2300)
        /tmp/gowork/src/github.com/prometheus/prometheus/retrieval/discovery/serverset.go:346 +0x35f
created by github.com/prometheus/prometheus/retrieval/discovery.(*zookeeperTreeCache).recursiveNodeUpdate
        /tmp/gowork/src/github.com/prometheus/prometheus/retrieval/discovery/serverset.go:353 +0x9f4

As the serveset impl. is applying watches for the entire tree any changes to the members will spawn additional goroutines for already watched members. I added some debugging to prometheus to illustrate this:

DEBU[50188] /aurora/jobs:1                                source=serverset.go:192
DEBU[50188] /aurora/jobs/www-data/devel/hello_world/member_0000000149:2  source=serverset.go:192
DEBU[50188] /aurora/jobs/www-data:1                       source=serverset.go:192
DEBU[50188] /aurora/jobs/www-data/devel/hello_world:1     source=serverset.go:192
DEBU[50188] /aurora/jobs/www-data/devel/hello_world/member_0000000140:13  source=serverset.go:192
DEBU[50188] /aurora/jobs/www-data/devel/hello_world/member_0000000150:1  source=serverset.go:192
DEBU[50188] /aurora/jobs/www-data/devel/hello_world/member_0000000148:3  source=serverset.go:192
DEBU[50188] /aurora/jobs/www-data/devel:1                 source=serverset.go:192
DEBU[50188] /aurora/jobs/www-data/devel/hello_world/member_0000000144:8  source=serverset.go:192
DEBU[50188] /aurora/jobs/www-data/devel/hello_world/member_0000000145:8  source=serverset.go:192

The above is an aurora job which has been scaled up and down a couple of times member_0000000140 has 13 watches and therefore uses 13 goroutines.

Wanted to file this to see if other has run into this and if my observations holds true.
If nobody bets me to it I can take a stab at it when time permits and file a PR.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.