New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Routing metrics by plugin #1778

Open
joakimberglund opened this Issue Sep 16, 2016 · 6 comments

Comments

Projects
None yet
6 participants
@joakimberglund
Copy link

joakimberglund commented Sep 16, 2016

Feature Request

Add the possibility to select which inputs a certain output should use.

Proposal:

[[outputs.influxdb]]
  urls = ["http://localhost:8086"]
  database = "DB1"
  inputs = [ "exec" ]

[[outputs.influxdb]]
  urls = ["http://localhost:8086"]
  database = "DB2"
  inputs = [ "mem", "cpu" ]

And/or.....

[[outputs.influxdb]]
  name ="output1"
  urls = ["http://localhost:8086"]
  database = "linux"

[[inputs.cpu]]
  output = ["output1", "output2"]
  percpu = false
  totalcpu = true

Default should be that it behaves as it does today.

Current behavior:

Not possible

Desired behavior:

Use case: [Why is this important (helps with prioritizing requests)]

Using the namepass/fieldpass/tagpass is not flexible enough especially when using the exec och httpjson inputs where the name can change or contain multiple names which might collide namewise with other inputs.

Also supporting dividing up the measurements in different databases so that 1) it's easier to scale influxdb instanses 2) get better performance and 3) easier for users to design dashboards

@sparrc

This comment has been minimized.

Copy link
Contributor

sparrc commented Sep 16, 2016

this would be a nice feature to have, but it's actually quite a tricky problem. Internally all plugins are placed in lists and there is no way to differentiate between two different [[outputs.influxdb]] output instances.

FWIW, the 3 current best workarounds are:

  • use name_override, name_suffix, or name_prefix config options to prepend names to your measurements on the input-side, then use namepass to route to outputs, ie:
[[inputs.exec]]
  # ...
  name_prefix = "exec1_"

[[inputs.exec]]
  # ...
  name_prefix = "exec2_"

# this influxdb will only get exec1
[[outputs.influxdb]]
  # ...
  namepass = "exec1_*"

# this will only get exec2
[[outputs.influxdb]]
  # ...
  namepass = "exec2_*"
  • Use tags, tagdrop, tagpass routing, see @phemmer's comment below
  • run multiple telegraf processes.
@phemmer

This comment has been minimized.

Copy link
Contributor

phemmer commented Sep 16, 2016

I currently do this in my environment. My solution looks like:

[[outputs.influxdb]]
  urls = ["http://influxdb.example.com"]
  database = "db_default"
  [outputs.influxdb.tagdrop]
    influxdb_database = ["*"]

[[outputs.influxdb]]
  urls = ["http://influxdb.example.com"]
  database = "db_other"
  tagexclude = ["influxdb_database"]
  [ouputs.influxdb.tagpass]
    influxdb_database = ["other"]

[[inputs.foo]]
  [inputs.foo.tags]
    influxdb_database = "other"

This will send inputs.foo to the "db_other" influxdb output. All other inputs (which do not have a influxdb_database tag) will go to the "db_default" influxdb output.

@joakimberglund

This comment has been minimized.

Copy link
Author

joakimberglund commented Sep 19, 2016

@phemmer Now thats clever, I can use that, thanks

Another way that this might be doable to create almost the same effect is that the configs in telegraf.d is their own instanses.
Aka with "private" input/outputs lists per file.

@daraeburn

This comment has been minimized.

Copy link

daraeburn commented Sep 21, 2016

I have tried that method using config files @joakimberglund but can't seem to get it to work. When I run a config test on it once I move my input and output files to /etc/telegraf/telegraf.d/ it fails with the following error

2016/09/21 09:37:25 Error: no outputs found, did you provide a valid config file?

It looks like it isn't reading the files in the telegraf.d folder even after trying to add include "/etc/telegraf/telegraf.d/*.conf" to the end of the file. If I add that it fails with this error :

Error parsing telegraf.conf, toml: line 147: parse error

I'm wondering if there is some kind of test in telegraf that checks for an output section before allowing it to run. I'm going to try adding just one input and output plugin then add the others to the telegraf.d folder to see if that works.

@daraeburn

This comment has been minimized.

Copy link

daraeburn commented Sep 21, 2016

That doesnt work either. I added just the default input and output plugins to the config and added in the other input and output plugins in to telegraf.d folder but it only loads the default one and ignores the others.

@sparrc

This comment has been minimized.

Copy link
Contributor

sparrc commented Sep 21, 2016

@daraeburn: please open a separate issue with steps to reproduce if you are having a configuration problem, I don't think that's related to this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment