Frequent config reloading cause multi connection of same target

[ZeeShen]

I set a cron job that check prometheus config repo and SIGHUP prometheus when config changes.

A bug in the job cause prometheus reloading config every minute. Then prometheus kept logging
error like too many open files.

↑ Open fd and goroutine count increasing after setting ulimit to 4096, target count 700+, avg scrape duration 60ms.

Then I found most of these opening fds were socket files. And with netstat, I found prometheus kept several(like 5 or 6) ESTABLISHED connections with same targets.

Not familiar with the code base but it might be helpful to check here that scrape pool will start new scrape loops when config changes. Maybe something wrong with scrape loop's run or stop?

I fixed the cron job and prometheus is running correctly now(open fd 600+, socket 500+). But I think maybe the prometheus server should deal with this frequent reloading situation. Or maybe something wrong with my configuration?

Environment

• System information:

Linux 4.4.0-62-generic x86_64

• Prometheus version:

  build user:       root@615b82cb36b6
  build date:       20171108-07:11:59
  go version:       go1.9.2

running with
--storage.tsdb.path=/mnt/prometheus --config.file=/xxx/prometheus.yml --storage.tsdb.retention=360h

and the prometheus.yaml is kinda big, main part is an ec2-sd and a file-sd of static targets.

[gouthamve]

#2830 Should fix this. Will get around to it soon.

[ZeeShen]

Reloading only when config changes is a good solution for my case.

But what if the config is actually changes and should be reloaded frequently? Like add or remove some targets every minute, most targets remains unchanged and this multi-connection may still happen?

[gouthamve]

Hi, you should use file SD for that.

…

On Tue 12 Dec, 2017, 9:25 PM ZeeShen, ***@***.***> wrote: Reloading only when config changes is a good solution for my case. But what if the config is actually changes and should be reloaded frequently? Like add or remove some targets every minute, most targets remains unchanged and this multi-connection may still happen? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#3577 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AHA3H7A1aj-dpIZZK9iZGCjPuqGBTP85ks5s_0OvgaJpZM4Q-s2B> .

[brian-brazil]

Is this still happening?

[ZeeShen]

We gave prometheus a BIG fd limit to avoid such too many open files error. Will open new issue if we found this happen again.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.