Adding cluster_stats metricset for elasticsearch module #7638

ycombinator · 2018-07-18T23:06:46Z

This PR adds a cluster_stats metricset for the elasticsearch Metricbeat module. Only an initial set of fields are added by this PR.

For follow up PRs

Adding event generation for X-Pack Monitoring indices.
Adding more fields, as necessary.

houndci-bot · 2018-07-18T23:06:56Z

metricbeat/module/elasticsearch/cluster_stats/data.go

+// specific language governing permissions and limitations
+// under the License.
+
+package cluster_stats


don't use an underscore in package name

houndci-bot · 2018-07-18T23:06:56Z

metricbeat/module/elasticsearch/cluster_stats/cluster_stats.go

+// specific language governing permissions and limitations
+// under the License.
+
+package cluster_stats


don't use an underscore in package name

houndci-bot · 2018-07-18T23:06:56Z

metricbeat/module/elasticsearch/cluster_stats/data.go

+// specific language governing permissions and limitations
+// under the License.
+
+package cluster_stats


don't use an underscore in package name

houndci-bot · 2018-07-18T23:06:57Z

metricbeat/module/elasticsearch/cluster_stats/cluster_stats.go

+// specific language governing permissions and limitations
+// under the License.
+
+package cluster_stats


don't use an underscore in package name

houndci-bot · 2018-07-18T23:17:11Z

metricbeat/module/elasticsearch/cluster_stats/data_test.go

+
+// +build !integration
+
+package cluster_stats


don't use an underscore in package name

ruflin · 2018-07-19T07:12:26Z

metricbeat/module/elasticsearch/cluster_stats/_meta/data.json

+            "id": "njqU4EQaTROIDlWPeUMQyw",
+            "name": "elasticsearch",
+            "stats": {
+                "indices": {


How does the cluster index data overlap with the index and node_stats metricset data?

index metricset has docs, segments, and store which are also present in this metricset here.

node_stats metricset has docs, segments, and store, which are also present in this metricset here.

node_stats metricset also has fs and jvm stats, which are currently not present in this metricset here but are available from the Elasticsearch GET _cluster/stats API.

This metricset here has shards which is absent in the index metricset and also not available from the Elasticsearch GET _stats API.

This metricset here has shards which is absent in the node_stats metricset and also not available from the Elasticsearch GET _nodes/_local API or the GET _nodex/_local/stats API.

This metricset here has fielddata which is absent in the index metricset but is available from the Elasticsearch GET _stats API.

This metricset here has fielddata which is absent in the node_stats metricset and also not available from the Elasticsearch GET _nodes/_local API. However it is available from the Elasticsearch GET _nodes/_local/stats API. Of course, the latter API is missing a bunch of other fields that are present in the former API so either we call both or we ask the ES team to enhance the latter API's response to include all the fields we use from the former API's response as well.

I do need to fix the field names in this metricset here so they match up with the ones in other elasticsearch metricsets (e.g. I need to use count instead of total, etc.)

Field names fixed in 6662787.

Should we report all the data we have in the index and node_stats metricset here too? Or should we focus on the data points that other metricsets don't have?

I think to start off we can only report the stats that are not already covered by other metricsets. Concretely, those would be fielddata and shards fields.

Fixed in 27ef3fb.

ruflin · 2018-07-19T07:13:42Z

metricbeat/module/elasticsearch/cluster_stats/data.go

+
+	event.ModuleFields = common.MapStr{}
+	event.ModuleFields.Put("cluster.name", info.ClusterName) // TODO cluster_name from cluster stats response could be used here?
+	event.ModuleFields.Put("cluster.id", info.ClusterID)     // TODO: Can we get this returned in cluster stats response itself?


++

I think both values are a must to be returned y cluster stats API

Created elastic/elasticsearch#32205 to request cluster UUID from cluster stats API itself.

... and put up PR as well: elastic/elasticsearch#32206

Keep in mind that older versions of ES will not have it in, so in these cases we need to skip it.

Good point! Should we skip it or make the extra API call in that case (I realize that complicates the logic/organization of the code a fair bit)?

Lets skip it to not add more complexity. New versions, new features :-)

Fixed in ed74319b4.

ruflin

Code LGTM. Could you add a CHANGELOG entry?

ruflin · 2018-07-23T21:06:11Z

metricbeat/module/elasticsearch/cluster_stats/_meta/fields.yml

+        - name: count
+          type: long
+          description: >
+            Total number of shards in cluster


Apply to all lines: I think normally we put a dot at the end for the docs.

ruflin · 2018-07-23T21:06:24Z

metricbeat/module/elasticsearch/cluster_stats/_meta/test/cluster_stats.700.json

@@ -0,0 +1,147 @@
+{


Can we also have one for 630?

Added in a5953bd.

ruflin · 2018-07-24T06:19:25Z

@ycombinator Looks like this could be merged before the ES PR is merged as we do check if the field exists?

Could you rebase and directly squash the commits?

ycombinator · 2018-07-24T11:39:39Z

Yes, good point @ruflin. I will fix up this PR now so it can be reviewed/merged soon.

ycombinator · 2018-07-24T12:02:02Z

Alright, all squashed up into 766b7f3 and ready for review/merge. Thanks @ruflin!

ycombinator added in progress Pull request is currently in progress. Metricbeat Metricbeat v7.0.0-alpha1 labels Jul 18, 2018

houndci-bot reviewed Jul 18, 2018

View reviewed changes

ycombinator force-pushed the metricbeat/elasticsearch/cluster-stats branch from d579910 to bebda92 Compare July 18, 2018 23:27

ycombinator requested review from jsoriano and ruflin July 18, 2018 23:34

ycombinator added review and removed in progress Pull request is currently in progress. labels Jul 18, 2018

ruflin reviewed Jul 19, 2018

View reviewed changes

ycombinator force-pushed the metricbeat/elasticsearch/cluster-stats branch from ed74319 to 290e7a8 Compare July 23, 2018 18:07

ruflin mentioned this pull request Jul 23, 2018

Monitor the Elastic Stack with Metricbeat #7035

Closed

39 tasks

ruflin reviewed Jul 23, 2018

View reviewed changes

ycombinator force-pushed the metricbeat/elasticsearch/cluster-stats branch 2 times, most recently from 346ba91 to 69ee642 Compare July 24, 2018 12:01

ycombinator added the v6.4.0 label Jul 24, 2018

Adding cluster_stats metricset for elasticsearch module

766b7f3

ycombinator force-pushed the metricbeat/elasticsearch/cluster-stats branch from 69ee642 to 766b7f3 Compare July 24, 2018 15:23

ruflin merged commit beada5f into elastic:master Jul 24, 2018

ycombinator deleted the metricbeat/elasticsearch/cluster-stats branch July 30, 2018 22:24

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding cluster_stats metricset for elasticsearch module #7638

Adding cluster_stats metricset for elasticsearch module #7638

ycombinator commented Jul 18, 2018 •

edited

Loading

houndci-bot Jul 18, 2018

houndci-bot Jul 18, 2018

houndci-bot Jul 18, 2018

houndci-bot Jul 18, 2018

houndci-bot Jul 18, 2018

ruflin Jul 19, 2018

ycombinator Jul 19, 2018 •

edited

Loading

ycombinator Jul 19, 2018

ycombinator Jul 19, 2018

ruflin Jul 20, 2018

ycombinator Jul 20, 2018

ruflin Jul 23, 2018

ycombinator Jul 23, 2018

ruflin Jul 19, 2018

ycombinator Jul 19, 2018

ycombinator Jul 19, 2018

ruflin Jul 19, 2018

ycombinator Jul 19, 2018

ruflin Jul 19, 2018

ycombinator Jul 19, 2018

ruflin left a comment

ruflin Jul 23, 2018

ruflin Jul 23, 2018

ycombinator Jul 23, 2018

ruflin commented Jul 24, 2018

ycombinator commented Jul 24, 2018

ycombinator commented Jul 24, 2018 •

edited

Loading

Adding cluster_stats metricset for elasticsearch module #7638

Adding cluster_stats metricset for elasticsearch module #7638

Conversation

ycombinator commented Jul 18, 2018 • edited Loading

For follow up PRs

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ycombinator Jul 19, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ruflin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ruflin commented Jul 24, 2018

ycombinator commented Jul 24, 2018

ycombinator commented Jul 24, 2018 • edited Loading

ycombinator commented Jul 18, 2018 •

edited

Loading

ycombinator Jul 19, 2018 •

edited

Loading

ycombinator commented Jul 24, 2018 •

edited

Loading