high load average every 1h 45m

[akaDJon]

every 1h 45m load average my server up to 3-4 (normal 0.2-0.5)
if stop telegraf service load average dont up every 1h 45m.

Why is this happening?
Is it possible to adjust the time or period?

telegraf v1.4.3
influxdb v1.3.7

[danielnelson]

Unless you have an input configured to run every 1h45m, there isn't any periodic tasks that would get ran by Telegraf. Is InfluxDB on the same system?

[akaDJon]

I have standart config for telegraf and dont have special inputs. InfluxDB on the same server. Maybe Telegraf send data to InfluxDB and InfluxDB up load average?

[SM616]

Also having this on my Telegraf servers (version 1.4.3, InfluxDB version 1.3.6). I have no specific periodic tasks configured to run every 1.5 hours, and I'm using almost default configuration with one HTTP listener input and two InfluxDB outputs.

Is it something like garbage collection?

[danielnelson]

Could you try using the procstat input to watch the Telegraf and the InfluxDB processes, this should allow us to see which process is causing the extra load. I think it could also be useful to enable the internal plugin, which will report the memory usage by Telegraf, and you could use the influxdb input to monitor InfluxDB.

[akaDJon]

Error in plugin [inputs.influxdb]: [url=http://127.0.0.1:8086/debug/vars]: Get http://127.0.0.1:8086/debug/vars: dial tcp 127.0.0.1:8086: getsockopt: connection refused

But wget "http://127.0.0.1:8086/debug/vars" works normal and get json

[akaDJon]

procstat info

[danielnelson]

Hmm, nothing appears to line up. Can you try to discover which process is causing the high load?

[akaDJon]

I tried stop service telegraf and get data load average in script

while true; do uptime >> uptime.log; sleep 10; done

and load average dont up. now try again

[akaDJon]

I do "systemctl stop telegraf" and for 4 hours load average dont up even upper 0.9

uptime.zip

as you can see clearly that the problem is in the telegraf or in using it together with influxdb

[danielnelson]

I suspect it may be caused by a process that Telegraf is starting, or possibly a subprocess of InfluxDB, because when we monitored the Telegraf process there was no corresponding spike in cpu usage. Can you try to find the name of the process that is producing the cpu usage during one of the spikes? It might be easiest to just run top during the time period with the increase.

[akaDJon]

visually on htop program i see 3-4 influxdb and telegraf. I try do print screen htop in spike period

[akaDJon]

on 2 minutes load average from 1 to 5

[danielnelson]

It looks to me that the load is not caused by cpu, telegraf is still only taking 7% as a peek reading, but I do notice that the influxdb process is often in state D. It might be interesting to take a look at the data collected by the processes input, which is enabled by default in telegraf.

[akaDJon]

What I need do?