[WIP] Etcd integration for configuration

[titilambert]

Rebased PR #465

Hello !
I just started an example of what could be the etcd integration with telegraf

Here an example:
1 . Make a myconf.conf file, which will be stored in etcd, with the following content :

[tags]
  dc = "us-east-1"

[agent]
  interval = "10s"
  round_interval = true
  flush_interval = "10s"
  flush_jitter = "0s"
  debug = false
  hostname = ""


[[outputs.influxdb]]
  urls = ["http://localhost:8086"]
  database = "telegraf"
  precision = "s"


[[inputs.cpu]]
  percpu = true
  totalcpu = true
  drop = ["cpu_time*"]

2 . Send this file to etcd using the label mylabel

./telegraf  -etcd http://127.0.0.1:2379 -etcdwritelabel mylabel -etcdwriteconfig myconf.conf

3 . You can check if data is really written in etcd with

./etcdctl get /telegraf/labels/mylabel

3 . Now any telegraf agent can load this config use the label mylabel

./telegraf  -etcd http://127.0.0.1:2379 -etcdreadlabels mylabel
Config read with label mylabel
2015/12/27 20:58:34 Database creation failed: Get http://localhost:8086/query?db=&q=CREATE+DATABASE+IF+NOT+EXISTS+telegraf: dial tcp 127.0.0.1:8086: getsockopt: connection refused
2015/12/27 20:58:34 Starting Telegraf (version v0.2.4-16-ga0bb7db)
2015/12/27 20:58:34 Loaded outputs: influxdb
2015/12/27 20:58:34 Loaded plugins: cpu
2015/12/27 20:58:34 Tags enabled: dc=us-east-1 host=osselait
2015/12/27 20:58:34 Agent Config: Interval:{10s}, Debug:false, Hostname:"osselait", Flush Interval:{10s}

Notes:

• DO NOT forget to change your etcd server URL
• Tested with etcd 2.2.2

Agent config reading order:

1. /telegraf/main key
2. /telegraf/hosts/HOSTNAME key
3. /telegraf/labels/LABEL1 key
   4./telegraf/labels/LABEL2 key

Features:

• Main config file can be loaded in etcd
• Each agent try automatically to find its own key in etcd (/telegraf/hosts/HOSTNAME)
• Labels can be configured in config files, so labels could be from etcd
• Etcd config watcher, that reload telegraf when a change is detected in etcd.
• You can write all configuration of ALL your telegraf agent in a folder then send it to etcd.

Missing:

• Add an option to select the root folder name in etcd (default "/telegraf") DONE
• Handle multiple etcd servers DONE
• Handle update/set/delete in etcd DONE
• Documentation DONE

[titilambert]

New features:

Agent specific config

Each agent start reading the value of key /telegraf/hosts/HOSTNAME.conf in etcd to get its default config. So you can now set your labels in this file and just start your agent like this:

./telegraf -etcd http://127.0.0.1:2379 

Configuration folder

You can now set your configuration in a folder like this:

testdata/
├── hosts
│   └── localhost.conf
├── labels
│   ├── influx.conf
│   ├── network2.conf
│   └── network.conf
└── main.conf

(An example is available here: https://github.com/titilambert/telegraf/tree/etcd/internal/etcd/testdata/test1)

Then you can send your configuration folder to etcd:

./telegraf -etcd http://127.0.0.1:2379  -etcdwriteconfigdir testdata/

Then you can start all your telegraf agents like this:

./telegraf  -etcd http://127.0.0.1:2379 -etcdreadlabels=influx,network

[titilambert]

@sparrc rebased ! (I don't give up ;) )

[sparrc]

great! Sorry I haven't had time to get a full review on this one, I've been slammed by some other features

[sparrc]

Can you remove the naoina/toml dependency here? I believe there is a function in influxdata/config that you can use to load and parse the toml file

[titilambert]

@sparrc changes done ! I add 2 tests to get more coverage.
And rebased :)

[titilambert]

@sparrc What you think about those features ?

BTW, it's still missing:

• Add an option to select the root folder name in etcd (default "/telegraf") DONE
• Handle multiple etcd servers DONE
• Handle update/set/delete in etcd
• Documentation DONE

[titilambert]

Added option to select the root folder name in etcd (default "/telegraf")

[titilambert]

Rebased with the new toml lib

[titilambert]

Rebased !
@sparrc What about create a new command telegrafctl ?
Instead of:

./telegraf -etcd http://127.0.0.1:2379  -etcdwriteconfigdir testdata/

We will use

./telegrafctl -etcd http://127.0.0.1:2379  -etcdwriteconfigdir testdata/

[sparrc]

why do you want to do that? is there a precedent?

[titilambert]

It's just to separate daemon binary from utils binary. I don't know if it's a good idea, it's just to copy etcd/kubernetes/...

[titilambert]

@sparrc could you just confirm that this PR is in scope of Telegraf ? :)

[sparrc]

yes, it is :)

[titilambert]

@sparrc cool :)

[titilambert]

@sparrc Rebased !
I also add a parameter to be able to erase config in etcd
I think all it's here. You can begin to review it, I'm waiting for your feedbacks :)
Thanks !

[balboah]

Why not just use https://github.com/kelseyhightower/confd together with a kill HUP signal?

[titilambert]

@balboah This does add a single point of failure, doesn't it ?
And this is not really cool when you re using telegraf inside a container...

[sparrc]

@titilambert As I've looked through this PR more, and looked into etcd and configuration management options, I feel like this is going to get messy.

Yes, it's nice that we could have etcd directly baked into telegraf in some ways. But it's also complicated and it ignores all the other options that there are out there for achieving this (consul, redis, vault, etc)

As @balboah suggests, this seems more like a configuration management issue, so why not use configuration management tools to solve it? Why should telegraf become a configuration management tool on top of all the other things it does?

[sparrc]

ps: do you have any examples of a project similar to telegraf that integrates directly with etcd? Or are there any libraries that we could use to generically get conf files? (like a library version of confd?)

[balboah]

@titilambert not sure what you mean with single point of failure. In my case confd runs inside the container, monitors the etcd cluster for changes and updates the config file + sends the kill signal when there is some change.
As long as telegraf is good at handling that config file reloading (like re-connecting if influxdb hosts changes) all is good imo

[titilambert]

@balboah Single point of failure: What's happen if confd crashed ? We don't have any fallback for confd. I can not see how running confd inside a container can solve this issue. Docker can still restart condf but the single point of failure is now on Docker. The only single point of failure should be Telegraf.

@sparrc I'm agree with you, this PR limits Telegraf to Etcd. But, imo, confd seems a single point of failure.

do you have any examples of a project similar to telegraf that integrates directly with etcd?
Yes, Kubernetes, fleet, locksmith, vulcand, calico, flannel, ...

Or are there any libraries that we could use to generically get conf files? (like a library version of confd?)
Good question !!!

[sparrc]

@titilambert but what if your etcd server goes down? This could be a problem if etcd is integrated directly into telegraf. If you decouple the two services (telegraf and config management) then Telegraf is completely unaffected by any status or change in etcd.

[titilambert]

@sparrc etcd can run as a cluster (with at least 3 nodes) which means etcd isn't a single point of failure.
With confd I just can't see how we can eliminate this issue (because you need to run several confd daemons on the same machine)

[balboah]

sorry maybe I still don't understand, but I fail to see how integrating basically the same use case as confd into telegraf solves the availability issue differently? The code would be pretty much the same, the number of processes would be the same, how is the "single point of failure" different?
To clarify: confd talks with etcd nodes and writes a config file. Telegraf would do the same, or update configuration in its memory. Sure confd process could die because of bugs, but so could telegraf?

Also confd already supports toml templating that you're introducing, has plugins for different statements and supports more sources than etcd.

[titilambert]

@balboah With this PR Telegraf will never get config files. It loads conf directly from etcd in memory. This eliminate the write config step.
Of course, Telegraf can crash in both cases but you have one step less without confd.

@sparrc I think you're right ! Maybe we need to use https://github.com/spf13/viper ? We can use it to read config only from remote sources or for both remote and local files. What should be the best choice ?

[myontop]

when you plan to release a version with etcd integration?

[johnrengelman]

I'll chime in my 2 cents - I don't think telegraf should go down the road of supporting config backends directly. Once etcd is added, then there will be requests for consul and zookeeper, etc.
It becomes a maintenance nightmare.
A better approach is to have best practices for how to integrate with these services externally.
As for the SPOF argument, by using something like confd to write out a config file and have telegraf simply load that file, you are actually reducing the failure modes. In this scenario, if etcd or confd fails, then there is still a configuration file for telegraf to load which will allow it to startup up.

If integrated directly and etcd is down, then telegraf can't run because the configuration is coming from there.

It also protects telegraf from any changes in the APIs of these tools. You don't want to to have to release a new version of telegraf due to a compatibility issue with etcd.

[titilambert]

@johnrengelman what do you think about https://github.com/spf13/viper ?
I understand the use case:

In this scenario, if etcd or confd fails, then there is still a configuration file for telegraf to load which will allow it to startup up.

but I'm sure about using confd+telegraf in docker envs...
For example in Kubernetes environnement, this means adding a new container inside telegraf pod... this will multiplicate by 2 your number of containers, just for the configuration. And you prefer using ressources/containers for you own applications.

[gunnaraasen]

Not sure if this has been mentioned before. Docker's libkv is another potential option for supporting multiple distributed config stores.

[sparrc]

that library looks fantastic, thanks @gunnaraasen

[titilambert]

@sparrc what do you think ? I rewrite the PR with https://github.com/docker/libkv or https://github.com/spf13/viper ?

[sparrc]

closing this because I prefer to have this conversation in #272