Support repeated input types

[cfbarbero]

I'd like to be able to use the same input type multiple times with different configurations. For example, I have 2 urls that I'd like to monitor with http_response inputs. I'd like to use hiera to configure the inputs. Right now given the way the hiera hash is used it will only keep 1 of the repeated inputs.

[yankcrime]

Is this the feature you were alluding to as well @millingworth ?

[doomnuggets]

Would this slight modification work?

define telegraf::input (
  $plugin_type = $name,
  $options     = undef,
  $sections    = undef,
) {
  include telegraf

  if $options {
    validate_hash($options)
  }

  if $sections {
    validate_hash($sections)
  }

  Class['::telegraf::config']
  ->
  file {"${telegraf::config_folder}/${name}.conf $title":
    path    => "${telegraf::config_folder}/${name}.conf"
    content => template('telegraf/input.conf.erb')
  }
  ~>
  Class['::telegraf::service']
}

Note the file resource name change.

[cfbarbero]

I'm not a puppet/hiera expert, but I'm not sure that will solve my issue.

This is an example of the telegraf.conf that I'm hoping to generate:

[[inputs.httpjson]]
   name = "endpoint1"
   servers = [
      "http://localhost/foo"
   ]
   method = "GET"

[[inputs.httpjson]]
   name = "endpoint2"
   servers = [
      "http://localhost/bar"
   ]
   method = "GET"

Here's the hiera that I would think might work:

telegraf::inputs:
  httpjson:
    name: "endpoint1"
    servers:
      - "http://localhost/foo"
    method: "GET"
  httpjson:
    name: "endpoint2"
    servers:
      - "http://localhost/bar"
    method: "GET"

Would this scenario be supported?

[doomnuggets]

I doubt it. Because a hash is not able to store the same key more than once.
I tried your example code and ended up with only the last definition of httpjson (endpoint2).

I'm working on a new commit which would solve this problem by introducing a separate variable for repeated inputs.

[7yl4r]

This applies to outputs as well. For instance, consider:

            'file' => {
                'files' => [
                    "/tmp/telegraf_output_influx.out",
                ],
                'data_format' => "influx",
            },
            'file' => {
                'files' => [
                    "/tmp/telegraf_output_graphite.out",
                ],
                'data_format' => "graphite",
            },

[gevial]

Same issue for me. Is it possible to configure multiple outputs of the same type using telegraf::output defined type?

[7yl4r]

I think this is going to require a re-structuring of the parameters into something like:

telegraf::inputs:
  - endpoint1:
        input_type: "httpjson"
        servers:
          - "http://localhost/foo"
        method: "GET"
  - endpoint2:
        input_type: "httpjson"
        servers:
          - "http://localhost/bar"
        method: "GET"

The limitation of one key having one value is part of hiera, so using input/output types (which can be repeated in the telegraf.conf) as hiera keys was a mistake. This would be a compatibility breaking feature unless the code can be modified to check telegraf::inputs for sub-keys vs an array.

[yankcrime]

@7yl4r You're right, it was shortsighted and we're jumping through hoops in the meantime as a result.

My plan is to draw a line under this module for 1.x, and to introduce 2.0 which will depend on Puppet 4 and address problems such as this.

[druchoo]

FWIW, this is my current solution which I've picked up from other modules:

Hiera:

profiles::telegraf::inputs:
  cloudwatch_qa_alb:
    plugin_type: 'cloudwatch'
    options:
      profile: 'qa'
      region: 'us-east-1'
      period: '1m'
      delay: '1m'
      interval: '1m'
      namespace: 'AWS/ApplicationELB'
      ratelimit: 20
  cloudwatch_qa_elb:
    plugin_type: 'cloudwatch'
    options:
      profile: 'qa'
      region: 'us-east-1'
      period: '1m'
      delay: '1m'
      interval: '1m'
      namespace: 'AWS/ELB'
      ratelimit: 20
  cloudwatch_dev_alb:
    plugin_type: 'cloudwatch'
    options:
      profile: 'dev'
      region: 'us-east-1'
      period: '1m'
      delay: '1m'
      interval: '1m'
      namespace: 'AWS/ApplicationELB'
      ratelimit: 20
  cloudwatch_dev_elb:
    plugin_type: 'cloudwatch'
    options:
      profile: 'dev'
      region: 'us-east-1'
      period: '1m'
      delay: '1m'
      interval: '1m'
      namespace: 'AWS/ELB'
      ratelimit: 20

Manifest:

class profiles::telegraf (
  Hash $inputs = {},
) {
  include ::telegraf

  create_resources(telegraf::input, $inputs)
}

Which generates these configs:

$ ls -1 /etc/telegraf/telegraf.d/
cloudwatch_dev_alb.conf
cloudwatch_dev_elb.conf
cloudwatch_qa_alb.conf
cloudwatch_qa_elb.conf

==> /etc/telegraf/telegraf.d/cloudwatch_dev_alb.conf <==

[[inputs.cloudwatch]]
  delay = "1m"
  interval = "1m"
  namespace = "AWS/ApplicationELB"
  period = "1m"
  profile = "dev"
  ratelimit = 20
  region = "us-east-1"

==> /etc/telegraf/telegraf.d/cloudwatch_dev_elb.conf <==

[[inputs.cloudwatch]]
  delay = "1m"
  interval = "1m"
  namespace = "AWS/ELB"
  period = "1m"
  profile = "dev"
  ratelimit = 20
  region = "us-east-1"

==> /etc/telegraf/telegraf.d/cloudwatch_qa_alb.conf <==

[[inputs.cloudwatch]]
  delay = "1m"
  interval = "1m"
  namespace = "AWS/ApplicationELB"
  period = "1m"
  profile = "qa"
  ratelimit = 20
  region = "us-east-1"

==> /etc/telegraf/telegraf.d/cloudwatch_qa_elb.conf <==

[[inputs.cloudwatch]]
  delay = "1m"
  interval = "1m"
  namespace = "AWS/ELB"
  period = "1m"
  profile = "qa"
  ratelimit = 20
  region = "us-east-1"

[mxjessie]

@yankcrime Any plans to (maybe) cut a branch for 2.0? I'm currently running into this issue with input and output plugins and I would love to try and help with this.

[nrdmn]

I've modified this module to use a real toml generator rather than templates to fix this issue. Tests are still failing because I don't know any ruby, but everything else works fine.
See https://github.com/nrdmn/puppet-telegraf/tree/toml

This requires the toml-rb gem and slightly changes the configuration structure to match the generated toml output.

[AndreGirbal]

We use the plugin snmp and need many "snmp.field" entries. It's impossible to do this with a hash.
It's why I change code on the template to use "sections" as an array and I could do what I want. But with this I lost ability to merge multiples files.

I use this code for "sections" in input.conf.erb:

<% @sections.each do |item| -%>
<% item.sort.each do |section, option| -%>
[[inputs.<%= section %>]]
<%      unless option == nil -%>
<%          option.sort.each do | suboption, value | -%>
  <%= suboption -%> = <% if value.is_a?(String) %>"<%= value %>"<% elsif value.is_a?(Array) %><%= value.inspect %><% else %><%= value %><% end %>
<%          end -%>
<%      end -%>
<%   end -%>
<% end -%>

Don't forget to comment "validate_hash($sections)" in input.pp

My hiera file:

telegraf::input:
  vm-labo-puppet.domain:
    plugin_type: snmp
    options:
      agents: ["vm-labo-puppet.integ.gnc:161"]
      version: 2
      name : 'vm-labo-puppet.domain*SNMP*Field'
      community: public
    sections:
    - snmp.field:
        name: 'LoadAverage_1m'
        oid: '.1.3.6.1.4.1.2021.10.1.5.1'
    - snmp.field:
        name: 'LoadAverage_5m'
        oid: '.1.3.6.1.4.1.2021.10.1.5.2'
    - snmp.field:
        name: 'LoadAverage_15m'
        oid: '.1.3.6.1.4.1.2021.10.1.5.3'
    - snmp.field:
        name: 'Memory_total'
        oid: '.1.3.6.1.4.1.2021.4.5.0'
    - snmp.field:
        name: 'Memory_Usage_availReal'
        oid: '.1.3.6.1.4.1.2021.4.6.0'
    - snmp.field:
        name: 'Memory_Usage_buffered'
        oid: '.1.3.6.1.4.1.2021.4.14.0

The conf file:

[[inputs.snmp]]
  agents = ["vm-labo-puppet.domain:161"]
  community = "public"
  name = "vm-labo-puppet.domain*SNMP*Field"
  version = 2
[[inputs.snmp.field]]
  name = "LoadAverage_1m"
  oid = ".1.3.6.1.4.1.2021.10.1.5.1"
[[inputs.snmp.field]]
  name = "LoadAverage_5m"
  oid = ".1.3.6.1.4.1.2021.10.1.5.2"
[[inputs.snmp.field]]
  name = "LoadAverage_15m"
  oid = ".1.3.6.1.4.1.2021.10.1.5.3"
[[inputs.snmp.field]]
  name = "Memory_total"
  oid = ".1.3.6.1.4.1.2021.4.5.0"
[[inputs.snmp.field]]
  name = "Memory_Usage_availReal"
  oid = ".1.3.6.1.4.1.2021.4.6.0"
[[inputs.snmp.field]]
  name = "Memory_Usage_buffered"
  oid = ".1.3.6.1.4.1.2021.4.14.0"

[yankcrime]

Closing this now that #80 has been merged which should address these use cases / issues.

[sparr]

@yankcrime it is not obvious to me how to utilize the new functionality in #80 to accomplish the goal here. Can you give an example?

[sparr]

Answering my own question:

telegraf::inputs:
  cpu:
    - percpu: true
      totalcpu: true
  exec:
    - commands:
        - who | wc -l
  exec-uptime:
    - commands:
        - cat /proc/uptime | awk '{print $1}'
      plugin_type: exec

Two exec inputs. The first one doesn't need a plugin_type because it has the "right" name. The second one the plugin_type overrides the name

[nrdmn]

This should work too (I didn't test this, but that's how #80 was supposed to be used)

telegraf::inputs:
  cpu:
    - percpu: true
      totalcpu: true
  exec:
    - commands:
        - who | wc -l
    - commands:
        - cat /proc/uptime | awk '{print $1}'

[sparr]

Does this change mean that every hiera config needs to be changed, or do old style just-one-hash-no-array configs still work?

[nrdmn]

Every hiera config needs to be changed.