Too Many Open Files every once a day

[alileza]

Every once a day my prometheus server crash because too many open files

• System information:

Linux 4.4.0-83-generic x86_64

• Prometheus version:

prometheus, version 1.5.0 (branch: master, revision: d840f2c)
build user: root@a04ed5b536e3
build date: 20170123-13:56:24
go version: go1.7.4

• Alertmanager version:

  alertmanager, version 0.5.1 (branch: master, revision: 0ea1cac51e6a620ec09d053f0484b97932b5c902)
  build user: root@fb407787b8bf
  build date: 20161125-08:14:40
  go version: go1.7.3

• Prometheus configuration file:

insert configuration here

• Alertmanager configuration file:

insert configuration here (if relevant to the issue)

• Logs:

[fabxc]

Practically, Prometheus just needs a good amount of file descriptors to do its work and simply raising your file descriptor limit for the process should solve the issue. There's no significant cost to raising the limit.

[alileza]

can we calculate the proper amount of file descriptor by metrics count? or just do a trial?

[gouthamve]

@alileza It largely depends on the number of targets you have and the scrape-interval. It requires the fds for the HTTP requests and for the database. If you have too many open-requests due to small scrape-interval or large number of targets, you can easily exhaust the default 1024 limit.

You could start with 4096 and increase it if you still see issues.

Closing this as it is not a Prometheus bug, please feel free to reopen if you think otherwise.

[alileza]

@gouthamve perfect, thank you !

[beorn7]

Just for the record: In all but the most extreme use cases, the vast majority of open fd's are created by leveldb. I guess those go up under heavier indexing load.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.