Bug: tsdbrelay socket: too many open files

[langerma]

Expected behaviour

tsdbrelay should forward all connections to opentsdb instantly

Current behaviour

we get a huge bunge of messages like this from tsdbrelay:
2020/03/04 11:28:51 http: Accept error: accept tcp [::]:14242: accept4: too many open files; retrying in 5ms 2020/03/04 11:28:51 http: proxy error: dial tcp 127.0.0.1:4242: socket: too many open files 2020/03/04 11:28:51 http: proxy error: dial tcp 127.0.0.1:4242: socket: too many open files 2020/03/04 11:28:51 http: Accept error: accept tcp [::]:14242: accept4: too many open files; retrying in 5ms 2020/03/04 11:28:51 http: Accept error: accept tcp [::]:14242: accept4: too many open files; retrying in 10ms 2020/03/04 11:28:51 http: Accept error: accept tcp [::]:14242: accept4: too many open files; retrying in 20ms 2020/03/04 11:28:51 http: Accept error: accept tcp [::]:14242: accept4: too many open files; retrying in 5ms 2020/03/04 11:28:51 http: proxy error: dial tcp 127.0.0.1:4242: socket: too many open files 2020/03/04 11:28:51 http: proxy error: dial tcp 127.0.0.1:4242: socket: too many open files 2020/03/04 11:28:51 http: Accept error: accept tcp [::]:14242: accept4: too many open files; retrying in 5ms 2020/03/04 11:28:51 http: proxy error: dial tcp 127.0.0.1:4242: socket: too many open files 2020/03/04 11:28:51 http: Accept error: accept tcp [::]:14242: accept4: too many open files; retrying in 5ms 2020/03/04 11:28:51 http: proxy error: dial tcp 127.0.0.1:4242: socket: too many open files

Steps to reproduce

we are forwarding like: 80k datapoints per sec / which was very irregular
and since we took out tsdbrelay the ingestion rate stabilized.

Context

we have 6 tsdbs on our hadoop/hbase data nodes
each of 'em has 96 gb of ram from which tsdbs get 8 hadoop(hdfs) gets 4 and hbase gets 32
all of the nodes are equipped with ssds
on each tsdb async network io is turned to false.

tsdbrelay is configured like this:
/app/tsdbrelay/tsdbrelay -b bosun:8070 -t 127.0.0.1:4242 -l :14242 -redis redis:6379 -db 0

Logs

see above

[muffix]

Thanks for reporting. Can you please add a bit more detail about your setup? What system are you running your Bosun on and if it's a Linux, please can you post the contents of /proc/sys/fs/file-max and the output of ulimit -n?

[langerma]

good morning,

ulimit -n:
65535
cat /proc/sys/fs/file-max:
8143410

we are running on rhel 7. and opentsdb 2.4

kind regards

[langerma]

any news on that?

[muffix]

I'm suspecting it might have to do with the HTTP client and the maximum nymber of idle connections that it allows. 🤔 The default seems to be 2:

const DefaultMaxIdleConnsPerHost = 2
https://golang.org/pkg/net/http/#pkg-constants

We could try increasing MaxIdleConnsPerHost, i.e. something like this:
http.DefaultTransport.(*http.Transport).MaxIdleConnsPerHost = higher_value, but I'll need to look at the implications and whether that would actually fix your problem.

[langerma]

okay....so if you have anything to test i would push it to my infra :-)

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.