Prometheus does not scale well with a large number of entries in discover

[bahamat]

What did you do?

Configure prometheus with 9 discover endpoints. Each endpoint includes 1200-1800 containers, totaling 14389 metrics endpoints to be scraped.

What did you expect to see?

Prometheus works

What did you see instead? Under which circumstances?

Thousands of connections open for DNS resolution, prometheus gathers little to no data (because DNS requests ultimately time out), the rest of the system has a hard tome functioning due to constant DNS failure (because all sockets are used up).

Environment

• System information:

  $ uname -srm
  Linux 3.10.0 x86_64

• Prometheus version:

prometheus, version 1.5.2 (branch: master, revision: bd1182d29f462c39544f94cc822830e1c64cf55b)
  build user:       root@a8af9200f95d
  build date:       20170210-14:41:22
  go version:       go1.7.5

[bahamat]

For what it's worth, I tried installing a local DNS resolver for caching and used 127.0.0.1 for DNS resolution, which itself melted down under the load of DNS requests.

[grobie]

This seems to be rather a usage/configuration question and a problem of your DNS server than Prometheus bug report.

• What is the scrape interval?
• What are the values of process_open_fds and process_max_fds of the Prometheus server (usually {job="prometheus"}?

[grobie]

Closing due to lack of response.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.