elasticsearch input: ability to get individual node stats #3304
Conversation
| @@ -259,13 +290,17 @@ func (e *Elasticsearch) gatherNodeStats(url string, acc telegraf.Accumulator) er | |||
|
|
|||
| now := time.Now() | |||
| for p, s := range stats { | |||
| if !e.individualStatIncluded(p) { | |||
There was a problem hiding this comment.
I still don't understand why this is done. It seems to me that the stats not included won't be in the slice we are ranging over and so this will never be hit.
There was a problem hiding this comment.
We are ranging over the property-to-string-mapping stats. I cannot think of a better way of dynamically mapping the names. We go from snake case to pascal case back to snake case. Is there a way to re-use the original JSON mappings (JSON to struct) to get the snake case names back. We are defining these mappings twice right now...
I added a simple nil check for now.
| f := jsonparser.JSONFlattener{} | ||
| // parse Json, ignoring strings and bools | ||
| err := f.FlattenJSON("", s) | ||
| if err != nil { | ||
| return err | ||
| } | ||
| acc.AddFields("elasticsearch_"+p, f.Fields, tags, now) | ||
| measurement := "elasticsearch_" + p | ||
| acc.AddFields(measurement, f.Fields, tags, now) |
There was a problem hiding this comment.
Revert this hunk, it's just a style change and no reason to make history more complex.
There was a problem hiding this comment.
Agreed, it is a remainder of something I tried earlier.
| @@ -340,7 +375,8 @@ func (e *Elasticsearch) gatherClusterStats(url string, acc telegraf.Accumulator) | |||
| if err != nil { | |||
| return err | |||
| } | |||
| acc.AddFields("elasticsearch_clusterstats_"+p, f.Fields, tags, now) | |||
| measurement := "elasticsearch_clusterstats_" + p | |||
| acc.AddFields(measurement, f.Fields, tags, now) | |||
There was a problem hiding this comment.
Revert this hunk for the same reasons
| } | ||
|
|
||
| return url + individualStatsUrlPart | ||
| } |
There was a problem hiding this comment.
I think this can be simplified to:
+func (e *Elasticsearch) generateStatsUrl(baseUrl string) string {
var url string
if e.Local {
url = baseUrl + statsPathLocal
} else {
url = baseUrl + statsPath
}
return url + e.generateIndividualStatsUrlPart()
}
meilke
force-pushed
the
elasticsearch-input-individual-stats
branch
from
bc8697f
to
2cf7c6b
Compare
| @@ -33,6 +33,10 @@ or [cluster-stats](https://www.elastic.co/guide/en/elasticsearch/reference/curre | |||
| ## Master node. | |||
| cluster_stats = false | |||
|
|
|||
| ## Set individual_stats to a list of sub-stats you are interested in | |||
| ## default is all stats | |||
| # individual_stats = ["jvm"] | |||
There was a problem hiding this comment.
I think it will be difficult to know what these values apply to as is. Would it help if we call this like node_stats = ["jvm"]? We should show the default in the commented out line, and it will also help if we list the current valid options:
+ ## Node stats are the statistic categories to gather for each node. Valid
+ ## options are `indices`, `os`, `process`, `jvm`, `transport`, `http`, `fs`,
+ ## `breaker` and `thread_pool`. If empty, all categories are gathered.
+ # nodes_stats = []
| return "/" + strings.Join(e.IndividualStats, ",") | ||
| } | ||
|
|
||
| func (e *Elasticsearch) generateStatsUrl(baseUrl string) string { |
| @@ -229,6 +229,26 @@ func (e *Elasticsearch) createHttpClient() (*http.Client, error) { | |||
| return client, nil | |||
| } | |||
|
|
|||
| func (e *Elasticsearch) generateIndividualStatsUrlPart() string { | |||
There was a problem hiding this comment.
Rename nodeStatsUrlPart(, or maybe just integrate into the parent function since it is not called from anywhere else.
meilke
force-pushed
the
elasticsearch-input-individual-stats
branch
from
2cf7c6b
to
fe6d678
Compare
|
Done! 😉 |
We found that the plugin generates a lot of data and also makes Elasticsearch work quite hard to generate the data (in some cases not necessary, if you are not interested in all the data).
New configurability:
Example configuration to just get only JVM information:
Required for all PRs: