Add fluentd input plugin #2661

dsalbert · 2017-04-12T15:02:50Z

Plugin is scraping metrics from JSON endpoint exposed by in_monitor plugin.
I've used custom function to parse JSON output due to output incoherency.

Please review and share your thoughts.

Thanks!

Required for all PRs:

CHANGELOG.md updated (we recommend not updating this until the PR has been approved by a maintainer)
Sign CLA (if not already signed)
README.md updated (if adding a new plugin)

dsalbert · 2017-04-12T15:15:16Z

I did not catch this:

plugins/inputs/fluentd/fluentd_test.go:126: arg pluginOutput for printf verb %s of wrong type: []fluentd.pluginData

sorry

danielnelson

Looks good, I have just a few questions and suggestions:

danielnelson · 2017-07-11T21:19:50Z

plugins/inputs/fluentd/fluentd.go

+	description  = "Read metrics exposed by fluentd in_monitor plugin"
+	sampleConfig = `
+	## This plugin only reads information exposed by fluentd using /api/plugins.json.
+	## Tested using 'fluentd' version '0.14.9'


Move this comment about version in with the tests, so we can see the version you generated the sample data from. If you know the plugin will not work with any specific versions of fluentd, then we can put that in the README.

Nothing that I've found during tests.

danielnelson · 2017-07-11T21:46:06Z

plugins/inputs/fluentd/README.md

+- All measurements have the following tags:
+	- PluginID        (unique plugin id)
+	- PluginType      (type of the plugin e.g. s3)
+    - PluginCategory  (plugin category e.g. output)


Telegraf normally uses snake_case for tag and field names, can you update these to use it?

Will do.
Thanks!

danielnelson · 2017-07-11T21:55:52Z

plugins/inputs/fluentd/fluentd.go

+	var (
+		pdPoint      pluginData
+		pdPointArray []pluginData
+		parsed       map[string]interface{}


Can you briefly explain why you decided to use an interface{} and reflection? It seems like since you have the struct defined it would be easy to Unmarshal directly into it.

As I recall it properly, I've done it like this, because for some plugins we do not have fields e.g. buffer_queue_length or buffer_total_queued_size.

In this case it may be better to use pointers in the struct and Unmarshal directly into it. If the field is not in the JSON it will be nil in the struct.

Thanks for hint. I'm going to work on this tomorrow and ask you for review.

dsalbert · 2017-07-12T14:23:34Z

CI failed because of unrecognized zookeeper service:

sudo service zookeeper stop
zookeeper: unrecognized service

sudo service zookeeper stop returned exit code 1

Action failed: sudo service zookeeper stop

danielnelson · 2017-07-12T18:50:05Z

plugins/inputs/fluentd/fluentd.go

+	exclude = [
+		"monitor_agent",
+		"dummy",
+	]


Make sure the sample config has 2 space indention here and in the README, so that it will have the same indention as with the other plugins when you run telegraf config.

danielnelson · 2017-07-13T00:33:14Z

I think if you rebase it will fix the circleci issue.

Plugin is scraping metrics from JSON endpoint exposed by in_monitor plugin. I've used custom function to parse JSON output due to output incoherency. Please review and share your thoughts. Thanks!

- Change CamelCase tags, measurements and values names to sneak_case - Move information about used version during tests to test file

As danielnelson noticed, there is no need to use relfect and we can simplify whole parser using just Unmarshal method from json library, what I have done.

danielnelson

The test issue cleared up when I ran the test again, unfortunately we have several intermittent test issues that can cause this from time to time.

One potential issue I ran into when testing was that every time you restart the fluentd daemon the plugins are given a new random plugin_id. If you restart the server frequently this could result in high series cardinality, though I suspect normally you would restart infrequently. I also might make queries a little harder write if you want to group on plugin_id.

I found that you can set the plugin_id in the configuration using @id, which would preserve the id across restarts. If I am understanding this correctly, we should add a paragraph to the configuration suggesting its use. Here is an example from my config which seems to work:

<source>
  @type http
  @id http
  port 8888
</source>

danielnelson · 2017-07-13T20:22:08Z

plugins/inputs/fluentd/fluentd.go

+	PluginCategory        string  `json:"plugin_category"`
+	RetryCount            float64 `json:"retry_count"`
+	BufferQueueLength     float64 `json:"buffer_queue_length"`
+	BufferTotalQueuedSize float64 `json:"buffer_total_queued_size"`


One difference in this revision of your plugin is that it will report 0 for these fields if they do not exist. In order to handle this you could have these be *float64 and test for nil before assigning the field.

Good point. According to documentation we can setup this field for all plugins:
@id: Specify plugin id. in_monitor_agent uses this value for plugin_id field. I'm going to add this to README.md

For second thing I will change this to pointer and add some checks around:

149 150 tmpFields["buffer_queue_length"] = p.BufferQueueLength 151 tmpFields["retry_count"] = p.RetryCount 152 tmpFields["buffer_total_queued_size"] = p.BufferTotalQueuedSize 153

Thanks!

New logic is checking if one of 3 fields (retry_count, buffer_queue_length and buffer_total_queued_size) is present. If there is at least one field, metric will be emitted and missing fields won't be added as values. In case where plugin doesn't have any of 3 fields, metric is skipped (not added to accumulator). Also unit test has been simplify.

danielnelson · 2017-07-13T23:00:46Z

Thank you!

dsalbert · 2017-07-14T07:23:02Z

@danielnelson Thanks for help!

danielnelson added the triage label Apr 12, 2017

danielnelson added this to the 1.4.0 milestone Apr 13, 2017

timhallinflux added the pm/extensibility label Apr 13, 2017

danielnelson removed the triage label Apr 13, 2017

danielnelson suggested changes Jul 11, 2017

View reviewed changes

danielnelson reviewed Jul 12, 2017

View reviewed changes

dsalbert and others added 5 commits July 13, 2017 11:08

Add fluentd input plugin

01fbca0

Plugin is scraping metrics from JSON endpoint exposed by in_monitor plugin. I've used custom function to parse JSON output due to output incoherency. Please review and share your thoughts. Thanks!

Simplify error for slice comparision

6c71de8

Fix formatting

115d268

apply required changes

2ecbdc1

- Change CamelCase tags, measurements and values names to sneak_case - Move information about used version during tests to test file

simplify parse method and update README.md

24cf872

As danielnelson noticed, there is no need to use relfect and we can simplify whole parser using just Unmarshal method from json library, what I have done.

dsalbert force-pushed the input-fluentd-plugin branch from b71af40 to 24cf872 Compare July 13, 2017 10:10

danielnelson reviewed Jul 13, 2017

View reviewed changes

Daniel Salbert added 2 commits July 13, 2017 22:40

add information about @id parameter for fluentd

ce9d8df

danielnelson approved these changes Jul 13, 2017

View reviewed changes

danielnelson merged commit f4d67d8 into influxdata:master Jul 13, 2017

jeichorn pushed a commit to jeichorn/telegraf that referenced this pull request Jul 24, 2017

Add fluentd input plugin (influxdata#2661)

df3f1b7

maxunt pushed a commit that referenced this pull request Jun 26, 2018

Add fluentd input plugin (#2661)

0ca0e21

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add fluentd input plugin #2661

Add fluentd input plugin #2661

dsalbert commented Apr 12, 2017 •

edited

Loading

dsalbert commented Apr 12, 2017 •

edited

Loading

danielnelson left a comment

danielnelson Jul 11, 2017

dsalbert Jul 12, 2017

danielnelson Jul 11, 2017

dsalbert Jul 12, 2017 •

edited

Loading

danielnelson Jul 11, 2017

dsalbert Jul 12, 2017

danielnelson Jul 12, 2017

dsalbert Jul 12, 2017

dsalbert commented Jul 12, 2017

danielnelson Jul 12, 2017

danielnelson commented Jul 13, 2017

danielnelson left a comment

danielnelson Jul 13, 2017

dsalbert Jul 13, 2017 •

edited

Loading

danielnelson commented Jul 13, 2017

dsalbert commented Jul 14, 2017

Add fluentd input plugin #2661

Add fluentd input plugin #2661

Conversation

dsalbert commented Apr 12, 2017 • edited Loading

Required for all PRs:

dsalbert commented Apr 12, 2017 • edited Loading

danielnelson left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dsalbert Jul 12, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dsalbert commented Jul 12, 2017

Choose a reason for hiding this comment

danielnelson commented Jul 13, 2017

danielnelson left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dsalbert Jul 13, 2017 • edited Loading

Choose a reason for hiding this comment

danielnelson commented Jul 13, 2017

dsalbert commented Jul 14, 2017

dsalbert commented Apr 12, 2017 •

edited

Loading

dsalbert commented Apr 12, 2017 •

edited

Loading

dsalbert Jul 12, 2017 •

edited

Loading

dsalbert Jul 13, 2017 •

edited

Loading