too many connect from grafana-server process

[heijiu]

• grafana-4.0.0-1480439068
• ES1.7
• RH6
• refresh every 5s
• normal
• too many connect from 127.0.0.1 to 127.0.0.1:9300 from grafana-server process

until grafana didn't work.

t=2016-12-01T17:01:49+0800 lvl=eror msg="Failed to start session" logger=context error="open /app/grafana-4.0.0-1480439068/data/sessions/4/f/4f603663296b4553: too many open files"
t=2016-12-01T17:01:49+0800 lvl=info msg="Request Completed" logger=context userId=0 orgId=0 uname= method=GET path=/api/login/ping status=401 remote_addr=10.xx.xx.xx time_ms=0s size=26
t=2016-12-01T17:01:54+0800 lvl=eror msg="Failed to start session" logger=context error="open /app/grafana-4.0.0-1480439068/data/sessions/4/f/4f603663296b4553: too many open files"
t=2016-12-01T17:01:54+0800 lvl=info msg="Request Completed" logger=context userId=0 orgId=0 uname= method=POST path=/api/datasources/proxy/1/_msearch status=401 remote_addr=10.xx.xx.xx time_ms=0s size=26
t=2016-12-01T17:01:55+0800 lvl=eror msg="Failed to start session" logger=context error="open /app/grafana-4.0.0-1480439068/data/sessions/4/f/4f603663296b4553: too many open files"
t=2016-12-01T17:01:55+0800 lvl=info msg="Request Completed" logger=context userId=0 orgId=0 uname= method=GET path=/api/login/ping status=401 remote_addr=10.xx.xx.xx time_ms=0s size=26

[heijiu]

after start grafana 30 min

#ss -s
Total: 1975 (kernel 2015)
TCP: 1591 (estab 1571, closed 6, orphaned 0, synrecv 0, timewait 0/0), ports 815

Transport Total IP IPv6

• 2015      -         -        
  

RAW 0 0 0
UDP 11 11 0
TCP 1585 1585 0
INET 1596 1596 0
FRAG 0 0 0

[warmfusion]

We've seen a similar issue; the connections are all pointing to our ElasticSearch datasource (Port 9200).

As shown below, there are nearly 10k connections to elasticsearch that appear to be sitting around.

lsof -p <pidofgrafana> | awk '{print $5}' | sort | uniq -c
      1 0000
      4 CHR
      2 DIR
      1 IPv4
   9992 IPv6
      7 REG
      1 TYPE

All connections are marked as 'established'

grafana-4.0.0-1480439068
ES1.6
Debian Wheezy

A number of dashboards open on TV's with refreshes at different rates.

[klausman]

Same here, with Prometheus as a DS. As far as I can tell, as long as nobody is looking at the dashboards, everything is fine, but as soon as you do, several tens of FDs are opened. I've started monitoring and graphing the FDs used, and it looks like they are actually leaked, since the numbers never drop.

As a side effect, this eventually also starves Prometheus of FDs.

grafana-4.0.0 (release). Prometheus 1.3 and 1.4 both are affected. OS is Gentoo Linux.

Graph:
https://snapshot.raintank.io/dashboard/snapshot/mxMIx1UhjnGYkok68egGVMB2FI4hhgGg

The steep increases are me having the dashboard open with a 5m autorefresh

[tanner-bruce]

I am experiencing this as well with Prometheus as a data source, this is a major issue affecting Prometheus, as like klausman said it starves Prometheus of file descriptors. The current awful solution to this is to restart grafana-server every TEN minutes

[bergquist]

This is a duplicate of #6774

We suggest that you rollback to beta2 if you encounter this problem. We will release 4.0.1 with a fix for this shortly.

[warmfusion]

I think @bergquist meant this is a duplicate of #6759