-
Notifications
You must be signed in to change notification settings - Fork 5.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
File descriptor / HTTP client connection leak on telegraf reload #15177
Comments
@nick-kentik please submit your change as a PR! |
@srebhan as I said, I don't think it's a complete fix |
Yeah, just poking one at random, it looks like the kibana input plugin has the same issue, for instance. A complete fix would need to audit each of these files and make sure
The config workaround doesn't work for me, since our config files need to work across both 1.21.4 and 1.27.4 at the moment, so I'm unlikely to have time to dedicate to upstream for a bit. |
PR: #15213 |
Relevant telegraf.conf
Logs from Telegraf
System info
Telegraf 1.27.4
Docker
No response
Steps to reproduce
lsof -np <telegraf-pid>
)Expected behavior
Config reloads should not cause file descriptors to accumulate
Actual behavior
EMFILE
Additional info
We're pushing telegraf 1.27.4 everywhere, with a nofile=16384 limit. Just after a restart, we hover around 200-300 file descriptors open, but within 24 hours we're seeing the 16384 limit exceeded on busier hosts.
Our infrastructure automatically issues
systemctl reload telegraf
commands quite a lot.Digging into the code a bit, I see new
http.Transport
instances (wrapped inhttp.Client
) created by the prometheus input plugin each time it's instantiated, butCloseIdleConnections()
is never called on them - so when theproemtheus.Prometheus
struct is discarded at reload, the idle connections will hang around forever.There is a configuration workaround - set the
idle_conn_timeout
config value to something non-zero - but at minimum I think telegraf needs:This is an issue rather than a PR because the http client is part of
inputs/common
, so presumably used elsewhere - all sites should be audited.There's also a
uClient
with a custom transport that is created per-gather(!) in the prometheus input when hitting a UNIX socket URL. I've not dug into that at all, but would suggest some eyes on it to make sure it's OK.The text was updated successfully, but these errors were encountered: