Graphite Input Parsing Proposal

[jwilder]

Introduction

This issue provides details on the proposed changes to the graphite plugin. The current graphite input assumes a format for incoming metrics in order to be able to pull out relevant pieces to use as tags.

Unfortunately, many systems do not use this format and are unable to change how metrics are sent which makes it difficult to switch to this plugin. The graphite plugin needs more flexibility in how it receives and parses graphite metrics.

There has been some discussion and proposed changes to the graphite plugin. See these issues for more discussion and background:

• Re-work Graphite input and metric processing #2102 @otoolep
• Graphite name parser #2572 @cannium

This proposals builds on the ideas found in those two issues and aims to make the plugin usable for all graphite inputs but also allow for the ability to extract tags without requiring a custom format to be used.

Design

By default, enabling the graphite plugin will allow you to collect metrics and store them as they are written. If you send a metric named servers.localhost.cpu.loadavg.10, it will store the the full metric name as the measurement with no extracted tags.

While this default setup works, it is not the ideal way to store measurements in InfluxDB since it does not take advantage of tags. It also will not perform optimally with a large dataset sizes since queries will be forced to use regexes which is known to not scale well.

To extract tags from metrics, one or more templates must be configured to parse metrics into tags and measurements.

Templates

Templates allow matching parts of a metric name to use as tag names in the stored metric. They have a similar format to graphite metric names. The values in between the separators are used as the tag name. The location of the tag name that matches the same position as the graphite metric section is used as the value. If there is no value, the graphite portion is skipped.

The special value measurement is used to define the measurement name. It can have a trailing * to indicate that the remainder of the metric should be used. If a measurement is not specified, the full metic name is used.

Basic Matching

servers.localhost.cpu.loadavg.10

• Template: .host.resource.measurement*
• Output: measurement =loading.10 tags =host=localhost resource=cpu

Adding Tags

Additional tags can be added to a metric that don't exist on the received metric. You can add additional tags by specifying them after the pattern. Tags have the same format as the line protocol. Multiple tags are separate by commas.

servers.localhost.cpu.loadavg.10

• Template: .host.resource.measurement* region=us-west,zone=1a
• Output: measurement = loading.10 tags = host=localhost resource=cpu region=us-west zone=1a

Multiple Templates

One template may not match all metrics. For example, using multiple plugins with diamond will produce metrics is different formats. If you need to use multiple template, you'll need to define a prefix filter that must match before the template can be applied.

Filters

Filters have a similar format to templates but work more like wildcard expressions. When multiple filters would match a metric, the more specific one is chosen. Filters are configured by adding them before the template.

For example,

servers.localhost.cpu.loadavg.10
servers.host123.elasticsearch.cache_hits 100
servers.host456.mysql.tx_count 10

• servers.* would match all values
• servers.*.mysql would match servers.host456.mysql.tx_count 10
• servers.localhost.* would match servers.localhost.cpu.loadavg

Default Templates

If no template filters are defined or you want to just have one basic template, you can define a default template. This template will apply to any metric that has not already matched a filter.

dev.http.requests.200
prod.myapp.errors.count
dev.db.queries.count

• env.app.measurement* would create
  • measurement=requests.200 tags=env=dev,app=http
  • measurement= errors.count tags=env=prod,app=myapp
  • measurement=queries.count tags=env=dev,app=db

Global Tags

If you need to add the same set of tags to all metrics, you can define them globally at the plugin level and not within each template description.

Minimal Config

[[graphite]]
  enabled = true
  # bind-address = ":2003"
  # protocol = "tcp"
  # consistency-level = "one"
  # tags = "key=value key= value"
  ### Each template line requires a template pattern.  It can have an optional
  ### filter before the template and separated by spaces.  It can also have optional extra
  ### tags following the template.  Multiple tags should be separated by commas and no spaces
  ### similar to the line protocol format.  The can be only one default template.
  # templates = [
  #   "*.app env.service.resource.measurement",
  #   # Default template
  #   "server.*",
 #]

Customized Config

[[graphite]]
  enabled = true
  tags = "key=value key= value"
  templates = [
     # filter + template
     "*.app env.service.resource.measurement", 

     # filter + template + extra tag
     "stats.* .host.measurement* region=us-west,agent=sensu",

      # default template. Ignore the first graphite component "servers"
     ".measurement*", 
 ]

[otoolep]

Quick suggestion -- how about making this detailed description a README.md in services/graphite? That way it acts as design documentation whenever the repo is checked out.

[otoolep]

And Support can also lift the markdown into online docs.

[jwilder]

Sounds good. I'll move this there once there is a PR ready.

[otoolep]

Great, I did that for precreator, and it is pretty handy.

https://github.com/influxdb/influxdb/blob/master/services/precreator/README.md

[otoolep]

Generally makes sense to me.

• Interesting idea about being able to add extra tags (at a global level, and per template). It'll be interesting to see if people use it.
• Can you show an example of a filter followed by a template? Combined with multiple templates? I guess I'd like to see a more complex config example, to be sure I understand what you are proposing.

[otoolep]

Oh -- I see the "customized config" -- studying that now.

[otoolep]

OK, cool, I think this should work.

[cannium]

Generally looks brilliant to me, and I have a few questions.

1. I'm not quite get the config tags = "key=value key= value", can you explain it more?
2. I somehow concern about the overhead it introduced. The filter design is nearly the complexity of regular expression. Would that be a problem?

[kyhavlov]

Looks great; simple but flexible to use.
@cannium caching filter results should avoid that problem, right? that way you avoid parsing series names too frequently

[torkelo]

I like the proposal, was thinking along similar lines for how to translate graphite metric paths to measurement and key/value pairs

[jwilder]

Fixed by #3125