Telegraf Configuration - Recommended approach for multiple .conf files?

[n1nj4888]

Hi There,

I run Telegraf docker (v1.11.5- Awesome utility by the way!) and originally had one big /etc/telegraf/telegraf.conf file with a single (InfluxDB) output and a few different input plugins (each sometimes with multiple agents) - Something like this structure of /etc/telegraf/telegraf.conf:

/etc/telegraf/telegraf.conf:

[Output Plugin - InfluxDB]
[Input Plugin - local metrics]
[Input Plugin - SNMP] - Agent IP1, Agent IP2, ..., Agent IPx
[Input Plugin - Docker] - Agent IP1
[Input Plugin - Docker] - Agent IP2
...

Now, since I understand that the above single config file causes Telegraf to run the file in series, I was seeing an issue where if Agent IPx was down this could cause the SNMP Input Plugin to timeout before it was able to contact/return results from Agent IPy. So I decided to break the config up into multiple .conf files as follows, all under /etc/telegraf/telegraf.d/ and this time with no main /etc/telegraf/telegraf.conf file:

/etc/telegraf/telegraf.d/telegraf-input-metrics.conf:

[Output Plugin - InfluxDB]
[Input Plugin - local metrics]

/etc/telegraf/telegraf.d/telegraf-snmp-AgentIP1.conf:

[Output Plugin - InfluxDB]
[Input Plugin - SNMP] - Agent IP1

...

/etc/telegraf/telegraf.d/telegraf-snmp-AgentIPX.conf:

[Output Plugin - InfluxDB]
[Input Plugin - SNMP] - Agent IPx

/etc/telegraf/telegraf.d/telegraf-docker-AgentIP1.conf:

[Output Plugin - InfluxDB]
[Input Plugin - Docker] - Agent IP1

/etc/telegraf/telegraf.d/telegraf-docker-AgentIP2.conf:

[Output Plugin - InfluxDB]
[Input Plugin - Docker] - Agent IP2

In doing this, I have a few questions:

(1) With Telegraf command configured to run as telegraf --config-directory /etc/telegraf/telegraf.d in the container, and with main /etc/telegraf/telegraf.conf file (all split conf files under /etc/telegraf/telegraf.d/): I got the following error on starting the container - I assume this means that there must be a /etc/telegraf/telegraf.conf file, even if the --config-directory /etc/telegraf/telegraf.d switch is passed to the telegraf command in the container?

2019-08-30T01:01:52Z I! Starting Telegraf 1.11.5
2019-08-30T01:01:52Z E! [telegraf] Error running agent: No config file specified, and could not find one in $TELEGRAF_CONFIG_PATH, /root/.telegraf/telegraf.conf, or /etc/telegraf/telegraf.conf

(2) To work around the above, I moved one of the config files (/etc/telegraf/telegraf.d/telegraf-input-metrics.conf) to /etc/telegraf/telegraf.conf and was then able to start the application with the following output to the log:

2019-08-30T01:04:48Z I! Starting Telegraf 1.11.5
2019-08-30T01:04:48Z I! Using config file: /etc/telegraf/telegraf.conf
2019-08-30T01:04:49Z I! Loaded inputs: disk diskio kernel mem processes swap system cpu docker docker file file snmp snmp snmp snmp snmp snmp snmp snmp snmp snmp
2019-08-30T01:04:49Z I! Loaded aggregators: 
2019-08-30T01:04:49Z I! Loaded processors: 
2019-08-30T01:04:49Z I! Loaded outputs: influxdb influxdb influxdb influxdb influxdb influxdb influxdb influxdb influxdb influxdb influxdb influxdb influxdb influxdb

I assume that the many "influxdb" outputs that are shown (last line above) are because each of my config files has the [Output Plugin - Influx DB] header therefore, is the recommended approach to:

Configure /etc/telegraf/telegraf.conf as just the main [Output Plugin - InfluxDB] and configure all the .conf files under /etc/telegraf/telegraf.d/ to just be [Input Plugin - X]?

If so, I assume this means that all the .conf files under /etc/telegraf/telegraf.d inherit the [Output Plugin - InfluxDB] from the main /etc/telegraf/telegraf.conf file so they know where to write the output to?

OR do I really still need to declare the [Output Plugin - InfluxDB] header at the top of each config file under /etc/telegraf/telegraf.d/?

[danielnelson]

The way to think about it is that all of the files: /etc/telegraf/telegraf.conf + everything in /etc/telegraf/telegraf.d are logically combined into one giant config file.

So there are a variety of way to do this, but here is what I recommend as a starting point:

• Delete or comment out all plugins in /etc/telegraf/telegraf.conf, setting only the [agent] table and [global_tags].
• Create one file for each input plugin and one file for each output plugin in /etc/telegraf/telegraf.d:
  • /etc/telegraf/telegraf.d/inputs.disk.conf
  • /etc/telegraf/telegraf.d/inputs.diskio.conf
  • /etc/telegraf/telegraf.d/inputs.kernel.conf
  • /etc/telegraf/telegraf.d/outputs.influxdb.conf

The /etc/telegraf/telegraf.conf file is required, though it could even be an empty file. This is something we want to change but it turns out to need more refactoring than you might imagine.

You will just have a single [[output.influxdb]] definition.

Feel free to group or split plugins in the telegraf.d directory as desired, for example maybe it's easier for you to keep all the default plugins in one inputs.default.conf file. Or perhaps you will want to split the snmp plugins by the type of device.

Let me know if you have more questions.

[n1nj4888]

Thanks for the info, I've now aligned the config accordingly...

[n1nj4888]

hi @danielnelson,

I've got the above working on an Ubuntu host but when using the official Telegraf Docker image on a Synology NAS, there doesn't seem to be a way through the Synology Docker GUI to change the telegraf command to include the --config-directory switch (i.e. to be "telegraf --config-directory /etc/telegraf/telegraf.d)... As such the Telegraf container only seems to monitor the /etc/telegraf/telegraf.conf file and no others (no extra conf files are picked up under /etc/telegraf or /etc/telegraf/telegraf.d)....

Is there a way to specify the --config-directory switch through either the container environment variables (which can be amended via the Synology Docker GUI) OR in the initial /etc/telegraf/telegraf.conf file itself to get Telegraf to also look in /etc/telegraf/telegraf.d/ ?

Thanks!

[n1nj4888]

I figured this out - Although there is no option to edit the telegraf container command on an existing Synology Telegraf container, the option is there to change the execution command when creating a new container.

Thanks!

[danielnelson]

Glad you have it going, #5656 is still on the plan to simply this a bit and perhaps we can improve the documentation on docker hub.

[a13519]

The way to think about it is that all of the files: /etc/telegraf/telegraf.conf + everything in /etc/telegraf/telegraf.d are logically combined into one giant config file.

So there are a variety of way to do this, but here is what I recommend as a starting point:

• Delete or comment out all plugins in /etc/telegraf/telegraf.conf, setting only the [agent] table and [global_tags].

• Create one file for each input plugin and one file for each output plugin in /etc/telegraf/telegraf.d:

  • /etc/telegraf/telegraf.d/inputs.disk.conf
  • /etc/telegraf/telegraf.d/inputs.diskio.conf
  • /etc/telegraf/telegraf.d/inputs.kernel.conf
  • /etc/telegraf/telegraf.d/outputs.influxdb.conf

The /etc/telegraf/telegraf.conf file is required, though it could even be an empty file. This is something we want to change but it turns out to need more refactoring than you might imagine.

You will just have a single [[output.influxdb]] definition.

Feel free to group or split plugins in the telegraf.d directory as desired, for example maybe it's easier for you to keep all the default plugins in one inputs.default.conf file. Or perhaps you will want to split the snmp plugins by the type of device.

Let me know if you have more questions.

Thanks!

May I do this to put a pair of input and output into one config under

/etc/telegraf/telegraf.d/abc.conf (inputs.abc.conf and outputs.influxdb.abc)
/etc/telegraf/telegraf.d/XYZ.conf (inputs.XYZ.conf and outputs.influxdb.XYZ)

can telegraf to send inputs.abc data to influxdb.abc and inputs.XYZ to influxdb.XYZ respectively?

Thanks!

[SokoFromNZ]

@a13519 If I understand @danielnelson correctly this is not possible.
He said all files "...are logically combined into one giant config file." So keeping separate files in the telegraf.d directory is just to easily organize them, not to run completely separate telegraf jobs.

I am looking for the same thing as you:

/etc/telegraf/telegraf.conf (inputs.abc.conf and outputs.influxdb.abc)
/etc/telegraf/telegraf2.conf (inputs.XYZ.conf and outputs.influxdb.XYZ)

If you put them into the telegraf.d folder and start telegraf accordingly to read the folder you will end up with all inputs (abc and XYZ) written to all outputs (abc and XYZ).

So what worked for me was to copy /lib/systemd/system/telegraf.service to /etc/systemd/system/telegraf2.service which I changed to the following content:

[Unit]
Description=Telegraf2
Documentation=https://github.com/influxdata/telegraf
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
EnvironmentFile=-/etc/default/telegraf
User=telegraf
ExecStart=/usr/bin/telegraf -config /etc/telegraf/telegraf2.conf
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
RestartForceExitStatus=SIGPIPE
KillMode=control-group

[Install]
WantedBy=multi-user.target

Then:
Reload the service files to include the new service: sudo systemctl daemon-reload
Start your service: sudo systemctl start telegraf2.service
To check the status of your service: sudo systemctl status telegraf2.service
To enable your service on every reboot: sudo systemctl enable telegraf2.service

[bzhou-sw]

The way to think about it is that all of the files: /etc/telegraf/telegraf.conf + everything in /etc/telegraf/telegraf.d are logically combined into one giant config file.

@danielnelson
This statement seems not true.
If I have some plugins defined in telegraf.conf and some plugins defined in file(s) under telegraf.d, I see plugins defined in telegraf.conf are NOT loaded.
If I move plugins from telegraf.conf to a new file under telegraf.d, then they are loaded.

Just want to double check. Thanks.

[powersj]

Hi,

This statement seems not true.

This is not expected and if you find otherwise, please feel free to file a new issue with your configs and how you run telegraf. With recent versions of Telegraf it will show what files are loaded, including the counts of plugins.