influxdb output leaks connections

[page31]

I putted the wrong retention policy in the config file yesterday and every write failed since that. I found there were about 5k tcp connections between telegraf and influxdb(port 8086) this morning.
The issue may be caused by Query function in influxdb/client/v2/client.go which uses JSON decoder to read the response body. The decoder just reads the first value and if there is still some data in the response stream the connection won't be closed or reused.

[sparrc]

thanks for the report, I don't think there's anything wrong with the client, it's this that's causing the bug: https://github.com/influxdata/telegraf/blob/master/plugins/outputs/influxdb/influxdb.go#L198

[sparrc]

Although it's not 100% clear to me why your OS is keeping alive so many dead TCP connections, can you give me the output of these commands?

cat /proc/sys/net/ipv4/tcp_keepalive_time
cat /proc/sys/net/ipv4/tcp_keepalive_intvl
cat /proc/sys/net/ipv4/tcp_keepalive_probes

[page31]

Of course, you should close the client before clear the array. But Close method of http client is just empty.

server running influxdb

$ cat /proc/sys/net/ipv4/tcp_keepalive_time
600
$ cat /proc/sys/net/ipv4/tcp_keepalive_intvl
15
$ cat /proc/sys/net/ipv4/tcp_keepalive_probes
5

server running telegraf:

$cat /proc/sys/net/ipv4/tcp_keepalive_time
7200
$ cat /proc/sys/net/ipv4/tcp_keepalive_intvl
75
$ cat /proc/sys/net/ipv4/tcp_keepalive_probes
9

[sparrc]

you're right, I don't think changing the close method will actually do anything to help this issue.

But I'm still not quite sure I follow the logic on reading the response object. When querying influx there is only a single json object that gets returned (it begins with {), and then there is an array internal to that, which is part of the object that it gets decoded to. I don't really see how there would still be data left in the stream (we also close the response body before exiting the function).

[sparrc]

From what I can tell, the problem is that Telegraf creates a new http client by calling Connect() every time there is a write failure. The idea here was that if the database goes away, Telegraf could recreate on restart. (see #836)

The problem in this case is that influxdb never goes away, and thus all tcp connections are still valid, and thus creating new http clients just results in more and more tcp connections.

So by "fixing" #836 I actually unintentionally created a situation where this could happen.I don't think there is a way that I can support #836 without the possibility of leaking tcp connections, so I'm going to revert that change.

EDIT: I am going to change the implementation so that it checks the error message for "database not found", and if that's there then it tries to recreate. It will no longer create new http clients.

cc @jchauncey

[page31]

@sparrc After digging a lot I just realized that too. Implementing HttpClient's Close method and call CloseIdleConnections of Transport should solve this problem. I'm sorry that I just guessed the reason this morning and that may be misleading.

[sparrc]

no problem, thanks for the report!, I've submitted a PR to influxdb to fix that, and a separate PR to telegraf. I think it's probably best to leave the http clients open and only recreate the db if that is the specific error encountered.