Adding x-pack monitoring code for elasticsearch/ccr metricset #8336

ycombinator · 2018-09-17T22:57:55Z

This PR teaches the elasticsearch/ccr metricset to index documents into .monitoring-es-6-* indices when xpack.enabled: true is set in modules.d/elasticsearch.yml.

To test this PR

The idea is that metricbeat (specifically the elasticsearch/ccr metricset with xpack.enabled: true) will create exactly the same documents in .monitoring-es-* indices as Elasticsearch's internal collection and reporting does today.

Start up Elasticsearch (using the latest build from master).
Start up Kibana.
Start a trial license from the Kibana Management UI.
Enable Monitoring in Elasticsearch (via the cluster setting xpack.monitoring.collection.enabled: true). You can do this via Elasticsearch's Cluster Update Settings API or by clicking the "Turn on Monitoring" button in the Monitoring UI in Kibana.
Set up CCR with a leader and follower index.
Let Elasticsearch run for ~20 seconds so a few documents are indexed into .monitoring-es-6-*.
From .monitoring-es-6-*, retrieve a document for type = ccr_stats
Turn off Elasticsearch's internal collection via the cluster setting xpack.monitoring.elasticsearch.collection.enabled: false.
Delete .monitoring-es-6-* indices.
Enable the elasticsearch module in metricbeat: ./metricbeat modules enable elasticsearch.

In modules.d/elasticsearch.yml, add the ccr metricset and set xpack.enabled: true. Concretely, your modules.d/elasticsearch.yml should look something like this:

- module: elasticsearch
   metricsets:
   - ccr
   period: 10s
   hosts: ["http://localhost:9200"]
   #username: "user"
   #password: "secret"
   xpack.enabled: true

Start metricbeat.
Let metricbeat run for ~20 seconds so a few documents are indexed into .monitoring-es-*.
Stop metricbeat
From .monitoring-es-6-*, retrieve a document for type = ccr_stats
Using a tool such as http://www.jsondiff.com, compare the documents indexed by Elasticsearch with those indexed by metricbeat. Verify that their structures are identical (same fields, not necessarily same values), except for these known and expected differences:
1. Only Metricbeat-indexed documents are expected to contain the fields @timestamp, beat, host, and metricset. These are "standard" fields added by beats and metricbeat and don't have an adverse impact since they are additive.
2. Only Elasticsearch-indexed documents are expected to contain the field source_node. This field is used for debugging purposes only and not actually consumed by either the Monitoring UI or Telemetry feature in Kibana.

ycombinator · 2018-09-18T22:34:55Z

Only CI failure is this unrelated flaky test: #8208.

jsoriano · 2018-09-19T09:34:48Z

metricbeat/module/elasticsearch/ccr/data_xpack.go

+				"timestamp":    common.Time(time.Now()),
+				"interval_ms":  m.Module().Config().Period / time.Millisecond,
+				"type":         "ccr_stats",
+				"ccr_stats":    shard,


Shouldn't schema be applied to this shard? or metrics in x-pack are expected to be collected as they are?

Metrics in x-pack are passed through to mirror what internal Elasticsearch collectors are doing today. Eventually (probably in 8.0.0), once we have completely migrated over to Metricbeat for monitoring the Elastic stack and the internal ES collector code has been removed, we can add schemas to these metricset to be stricter in what we collect and ship to Elasticsearch.

Ok, I was just wondering now what would be the experience of a user monitoring an OSS cluster with metricbeat when they start using x-pack 🤔

@jsoriano Just to make sure I'm understanding correctly: You mean if they try to monitor an OSS ES cluster with this metricset turned on and set xpack.enabled: true in their modules/elasticsearch.yml?

I mean if they are using OSS, and then at some moment they upgrade to x-pack.

chrisronline · 2018-09-19T19:37:03Z

@ycombinator

I followed the instructions but I'm not seeing any .monitoring-es-* indices. We zoomed about it but this is just a reminder ❤️

ycombinator · 2018-09-19T22:15:22Z

@chrisronline I'm able to reproduce what you're seeing locally as well. Will investigate and fix.

ycombinator · 2018-09-20T15:09:26Z

@chrisronline The issue you found should be fixed now. Ready for review again. Thanks!

chrisronline

LGTM!

Documents are as identical as expected!

…c#8336) This PR teaches the `elasticsearch/ccr` metricset to index documents into `.monitoring-es-6-*` indices when `xpack.enabled: true` is set in `modules.d/elasticsearch.yml`. ### To test this PR The idea is that metricbeat (specifically the elasticsearch/ccr metricset with `xpack.enabled: true`) will create exactly the same documents in `.monitoring-es-*` indices as Elasticsearch's internal collection and reporting does today. 1. Start up Elasticsearch (using the latest build from `master`). 2. Start up Kibana. 3. Start a trial license from the Kibana Management UI. 3. Enable Monitoring in Elasticsearch (via the cluster setting `xpack.monitoring.collection.enabled: true`). You can do this via [Elasticsearch's Cluster Update Settings API](https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-update-settings.html) or by clicking the "Turn on Monitoring" button in the Monitoring UI in Kibana. 4. Set up CCR with a leader and follower index. 4. Let Elasticsearch run for ~20 seconds so a few documents are indexed into `.monitoring-es-6-*`. 6. From `.monitoring-es-6-*`, retrieve a document for `type = ccr_stats` 8. Turn off Elasticsearch's internal collection via the cluster setting `xpack.monitoring.elasticsearch.collection.enabled: false`. 7. Delete `.monitoring-es-6-*` indices. 10. Enable the `elasticsearch` module in metricbeat: `./metricbeat modules enable elasticsearch`. 11. In `modules.d/elasticsearch.yml`, add the `ccr` metricset and set `xpack.enabled: true`. Concretely, your `modules.d/elasticsearch.yml` should look something like this: ```yaml - module: elasticsearch metricsets: - ccr period: 10s hosts: ["http://localhost:9200"] #username: "user" #password: "secret" xpack.enabled: true ``` 12. Start metricbeat. 13. Let metricbeat run for ~20 seconds so a few documents are indexed into `.monitoring-es-*`. 14. Stop metricbeat 6. From `.monitoring-es-6-*`, retrieve a document for `type = ccr_stats` 16. Using a tool such as http://www.jsondiff.com, compare the documents indexed by Elasticsearch with those indexed by metricbeat. Verify that their structures are identical (same fields, not necessarily same values), except for these known and expected differences: 1. Only Metricbeat-indexed documents are expected to contain the fields `@timestamp`, `beat`, `host`, and `metricset`. These are "standard" fields added by beats and metricbeat and don't have an adverse impact since they are additive. 3. Only Elasticsearch-indexed documents are expected to contain the field `source_node`. This field is used for debugging purposes only and not actually consumed by either the Monitoring UI or Telemetry feature in Kibana. (cherry picked from commit 35068ee)

…#8422) This PR teaches the `elasticsearch/ccr` metricset to index documents into `.monitoring-es-6-*` indices when `xpack.enabled: true` is set in `modules.d/elasticsearch.yml`. ### To test this PR The idea is that metricbeat (specifically the elasticsearch/ccr metricset with `xpack.enabled: true`) will create exactly the same documents in `.monitoring-es-*` indices as Elasticsearch's internal collection and reporting does today. 1. Start up Elasticsearch (using the latest build from `master`). 2. Start up Kibana. 3. Start a trial license from the Kibana Management UI. 3. Enable Monitoring in Elasticsearch (via the cluster setting `xpack.monitoring.collection.enabled: true`). You can do this via [Elasticsearch's Cluster Update Settings API](https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-update-settings.html) or by clicking the "Turn on Monitoring" button in the Monitoring UI in Kibana. 4. Set up CCR with a leader and follower index. 4. Let Elasticsearch run for ~20 seconds so a few documents are indexed into `.monitoring-es-6-*`. 6. From `.monitoring-es-6-*`, retrieve a document for `type = ccr_stats` 8. Turn off Elasticsearch's internal collection via the cluster setting `xpack.monitoring.elasticsearch.collection.enabled: false`. 7. Delete `.monitoring-es-6-*` indices. 10. Enable the `elasticsearch` module in metricbeat: `./metricbeat modules enable elasticsearch`. 11. In `modules.d/elasticsearch.yml`, add the `ccr` metricset and set `xpack.enabled: true`. Concretely, your `modules.d/elasticsearch.yml` should look something like this: ```yaml - module: elasticsearch metricsets: - ccr period: 10s hosts: ["http://localhost:9200"] #username: "user" #password: "secret" xpack.enabled: true ``` 12. Start metricbeat. 13. Let metricbeat run for ~20 seconds so a few documents are indexed into `.monitoring-es-*`. 14. Stop metricbeat 6. From `.monitoring-es-6-*`, retrieve a document for `type = ccr_stats` 16. Using a tool such as http://www.jsondiff.com, compare the documents indexed by Elasticsearch with those indexed by metricbeat. Verify that their structures are identical (same fields, not necessarily same values), except for these known and expected differences: 1. Only Metricbeat-indexed documents are expected to contain the fields `@timestamp`, `beat`, `host`, and `metricset`. These are "standard" fields added by beats and metricbeat and don't have an adverse impact since they are additive. 3. Only Elasticsearch-indexed documents are expected to contain the field `source_node`. This field is used for debugging purposes only and not actually consumed by either the Monitoring UI or Telemetry feature in Kibana. (cherry picked from commit 35068ee)

ycombinator added review Metricbeat Metricbeat needs_backport PR is waiting to be backported to other branches. v7.0.0-alpha1 monitoring v6.5.0 enhancement labels Sep 17, 2018

ruflin mentioned this pull request Sep 17, 2018

Monitor the Elastic Stack with Metricbeat #7035

Closed

39 tasks

ycombinator force-pushed the metricbeat-elasticsearch-ccr-xpack branch 6 times, most recently from 2eab02a to b75c7d2 Compare September 18, 2018 14:59

ycombinator added in progress Pull request is currently in progress. and removed review labels Sep 18, 2018

ycombinator added review and removed in progress Pull request is currently in progress. labels Sep 18, 2018

ycombinator requested a review from ruflin September 18, 2018 22:35

jsoriano reviewed Sep 19, 2018

View reviewed changes

ycombinator force-pushed the metricbeat-elasticsearch-ccr-xpack branch from b75c7d2 to d40611a Compare September 19, 2018 11:59

ycombinator requested a review from chrisronline September 19, 2018 11:59

jsoriano approved these changes Sep 19, 2018

View reviewed changes

chrisronline approved these changes Sep 24, 2018

View reviewed changes

ycombinator added 2 commits September 24, 2018 09:55

Add x-pack monitoring code for elasticsearch/ccr metricset

78ffe53

Fix reset path

667ff7b

ycombinator force-pushed the metricbeat-elasticsearch-ccr-xpack branch from c2f3a70 to 667ff7b Compare September 24, 2018 16:55

ycombinator merged commit 35068ee into elastic:master Sep 24, 2018

ycombinator mentioned this pull request Sep 24, 2018

Cherry-pick #8336 to 6.x: Adding x-pack monitoring code for elasticsearch/ccr metricset #8422

Merged

ycombinator added v6.5.0 and removed needs_backport PR is waiting to be backported to other branches. labels Sep 24, 2018

ycombinator deleted the metricbeat-elasticsearch-ccr-xpack branch September 26, 2018 14:12

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding x-pack monitoring code for elasticsearch/ccr metricset #8336

Adding x-pack monitoring code for elasticsearch/ccr metricset #8336

ycombinator commented Sep 17, 2018 •

edited

Loading

ycombinator commented Sep 18, 2018

jsoriano Sep 19, 2018 •

edited

Loading

ycombinator Sep 19, 2018

jsoriano Sep 19, 2018

ycombinator Sep 19, 2018

jsoriano Sep 19, 2018

chrisronline commented Sep 19, 2018

ycombinator commented Sep 19, 2018

ycombinator commented Sep 20, 2018

chrisronline left a comment

Adding x-pack monitoring code for elasticsearch/ccr metricset #8336

Adding x-pack monitoring code for elasticsearch/ccr metricset #8336

Conversation

ycombinator commented Sep 17, 2018 • edited Loading

To test this PR

ycombinator commented Sep 18, 2018

jsoriano Sep 19, 2018 • edited Loading

Choose a reason for hiding this comment

ycombinator Sep 19, 2018

Choose a reason for hiding this comment

jsoriano Sep 19, 2018

Choose a reason for hiding this comment

ycombinator Sep 19, 2018

Choose a reason for hiding this comment

jsoriano Sep 19, 2018

Choose a reason for hiding this comment

chrisronline commented Sep 19, 2018

ycombinator commented Sep 19, 2018

ycombinator commented Sep 20, 2018

chrisronline left a comment

Choose a reason for hiding this comment

ycombinator commented Sep 17, 2018 •

edited

Loading

jsoriano Sep 19, 2018 •

edited

Loading