Make elasticsearch/shard metricset work for Stack Monitoring without xpack.enabled flag #21389

sayden · 2020-09-29T15:25:04Z

Shard metricset maintains most of the Xpack flag functionality (indexing by the hash id created in the event) but includes now non-xpack things too.

elasticmachine · 2020-09-29T15:25:06Z

Pinging @elastic/stack-monitoring (Stack monitoring)

elasticmachine · 2020-09-29T15:25:06Z

Pinging @elastic/integrations-services (Team:Services)

elasticmachine · 2020-09-29T15:42:25Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: Pull request #21389 updated
Start Time: 2020-12-15T11:52:31.526+0000
Duration: 66 min 17 sec

Test stats 🧪

Test	Results
Failed	0
Passed	2312
Skipped	532
Total	2844

Steps errors

Expand to view the steps failures

`Terraform Apply on x-pack/metricbeat/module/aws`

Took 0 min 15 sec . View more details on here

`Terraform Apply on x-pack/metricbeat/module/aws`

Took 0 min 15 sec . View more details on here

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	2312
Skipped	532
Total	2844

chrisronline · 2020-10-19T16:42:31Z

@sayden The type needs to be shards (not shard). See https://github.com/elastic/elasticsearch/blob/master/x-pack/plugin/monitoring/src/main/java/org/elasticsearch/xpack/monitoring/collector/shards/ShardMonitoringDoc.java#L24

sayden · 2020-10-28T18:06:33Z

Not sure if I understand. In the mapping it is called shard https://gist.github.com/chrisronline/3b123c85d03b97ec3cbf63d9d72b3e2c#file-mb_mappings-json-L1246 and in the past it was shard too https://github.com/elastic/beats/blob/master/metricbeat/module/elasticsearch/shard/_meta/fields.yml#L1

I don't mind to change it, I just don't know where to change it 😄

chrisronline · 2020-10-29T17:16:23Z

I don't really know how the code works, but when I run the elasticsearch-xpack metricbeat module, and run this query:

POST .monitoring-es-*/_search?filter_path=aggregations.types.buckets
{
  "size": 0,
  "aggs": {
    "types": {
      "terms": {
        "field": "type",
        "size": 10
      }
    }
  }
}

I see:

{
  "aggregations" : {
    "types" : {
      "buckets" : [
        {
          "key" : "index_stats",
          "doc_count" : 4902
        },
        {
          "key" : "ccr_auto_follow_stats",
          "doc_count" : 961
        },
        {
          "key" : "cluster_stats",
          "doc_count" : 961
        },
        {
          "key" : "indices_stats",
          "doc_count" : 961
        },
        {
          "key" : "enrich_coordinator_stats",
          "doc_count" : 941
        },
        {
          "key" : "index_recovery",
          "doc_count" : 620
        },
        {
          "key" : "shards",
          "doc_count" : 294
        }
      ]
    }
  }
}

sayden · 2020-11-03T17:32:59Z

Ok, thanks for explaining. I think I understand what the problem is. If I understood it correctly, I see 2 options:

use event.dataset == "elasticsearch.shard" which will return all shard documents. Similar to the current "type": "shards".
I create an key in elasticsearch.type: "shards" field in the module to achieve the same thing.

Maybe @ycombinator has something to add too.

In general, no root "type": "shards" key is allowed in Metricbeat. I don't know if any of those ideas are an option in the UI

elasticmachine · 2020-11-06T17:54:25Z

🐛 Flaky test report

❕ There are test failures but not known flaky tests.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	3
Passed	2241
Skipped	478
Total	2722

Genuine test errors

💔 There are test failures but not known flaky tests, most likely a genuine test failure.

    * **Name**: `Build&Test / metricbeat-goIntegTest / TestXPackEnabled – elasticsearch`
    * **Name**: `Build&Test / metricbeat-goIntegTest / TestXPackEnabled/shard – elasticsearch`
    * **Name**: `Build&Test / metricbeat-pythonIntegTest / test_metricsets_8_shard – metricbeat.module.elasticsearch.test_elasticsearch.Test`

ycombinator · 2020-11-09T02:47:20Z

Re: the type: shards vs. type: shard discussion, I was looking at a discussion we had in the past about aliasing the root-level type field: #19747 (comment). In that discussion we concluded that the UI would not be relying on the type field but instead look at metricset.name.

The problem, of course, is that in the case of the elasticsearch/shard metricset, the value of metricset.name is set to shard when the UI really wants it to be shards. I guess up until this PR we just hadn't encountered such a discrepancy so looking at metricset.name in the UI code was okay.

To solve this issue in this PR but also in general, I like this idea:

I create an key in elasticsearch.type: "shards" field in the module to achieve the same thing.

This would mean:

In the elasticsearch module-level fields.yml (somewhere under here), we would define a type field.
Every elasticsearch module's metricset would need to set a value for this new elasticsearch.type field. In many/most cases this would end up being the same value as metricset.name but in some cases (like this PR), it would differ. The main thing is that the value of this field should be what the Stack Monitoring UI expects in the type field today (with xpack.enabled: true set).
The UI code should stop looking at metricset.name and instead start looking at elasticsearch.type.

@sayden @chrisronline WDYT?

chrisronline · 2020-11-09T17:48:04Z

Would it be easier if the UI code could just adapt to the documents being either shard or shards?

ycombinator · 2020-11-09T22:19:44Z

Would it be easier if the UI code could just adapt to the documents being either shard or shards?

If that's not difficult to do in the UI then sure! Also, before you do that, we should probably check all Stack Monitoring metricsets to see where else you might need to play this game. And then check how easy those would be to adapt in the UI as well. @sayden can you come up this information — basically the value of type vs. metricset.name for each Stack Monitoring metricset, highlighting the ones that are different.

chrisronline · 2020-11-10T16:01:33Z

And then check how easy those would be to adapt in the UI as well.

I can already tell you this will be easy. We can easily adapt https://github.com/elastic/kibana/blob/master/x-pack/plugins/monitoring/server/lib/create_query.js#L59 to support an array, just like we do now with type and metricbeat.name. We'll have to handle structural changes individually, but the shards structure change is do-able on the UI side.

ycombinator · 2020-11-11T00:05:44Z

@chrisronline thanks for the link. I understand now — you'll just add more clauses to the should as needed for special cases like type: shards. I know this isn't strictly your domain so feel free to pull someone else in if needed, but do you think there might be any issues with Telemetry code? Also, what about Cluster Alert watches, or are those gone now with the switch to Kibana Alerting?

Assuming all of the above checks out, I think the impact on this PR is a noop. That is, Metricbeat can keep sending metricset.name: shard.

chrisronline · 2020-11-11T15:00:55Z

but do you think there might be any issues with Telemetry code?

AFAIK, the telemetry code only concerns itself with cluster_stats documents (see here)

Also, what about Cluster Alert watches, or are those gone now with the switch to Kibana Alerting?

These should be going away in 7.11, with the ES team providing an API to permanently remove and disable the creation of new ones. However, we currently do not have a cluster alert that looks at this document type.

I'll work on the Kibana PR to handle both shard and shards.

chrisronline · 2020-11-11T16:39:55Z

One question - should I be seeing alias fields for this? I don't see anything setup for shard or shards yet.

sayden · 2020-11-11T21:23:12Z

Yes, I will also add an alias for this one, sorry for the delay!

…csearch/shard_xpack_flag

ycombinator · 2020-12-10T01:10:52Z

metricbeat/module/elasticsearch/shard/_meta/data.json

+            "source_node": {
+                "name": "1fb2aa83efac",
+                "uuid": "hx-oJ1-aT_-5pRG22JMI1Q"
+            },


I did not see the source_node.* fields mentioned in the metricset's fields.yml. Should they be?

Good catch! I completely forgot that I have to deactivate the test to check if all fields were documented.

ycombinator

Left one comment/question about fields potentially needing to be defined in the metricset's fields.yml.

ycombinator

LGTM.

…csearch/shard_xpack_flag # Conflicts: # metricbeat/module/elasticsearch/fields.go # metricbeat/module/elasticsearch/ml_job/data.go

…xpack.enabled flag (elastic#21389)

Atomic commit

aeca1c3

sayden added Metricbeat Metricbeat Feature:Stack Monitoring Team:Services (Deprecated) Label for the former Integrations-Services team labels Sep 29, 2020

sayden self-assigned this Sep 29, 2020

botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Sep 29, 2020

Fix linting

a215dcd

sayden added 2 commits December 3, 2020 09:27

Run make check and make update

d262c50

Fix broken test

380ed29

sayden requested a review from ycombinator December 9, 2020 15:17

sayden added 3 commits December 9, 2020 19:59

Merge branch 'feature-stack-monitoring-mb-ecs' into feature/mb/elasti…

0e20651

…csearch/shard_xpack_flag

Add a data.json generation test

1497a79

Update fields

3772ed2

ycombinator reviewed Dec 10, 2020

View reviewed changes

Add missing fields

f5f5a64

ycombinator approved these changes Dec 14, 2020

View reviewed changes

sayden added 2 commits December 15, 2020 12:50

Merge branch 'feature-stack-monitoring-mb-ecs' into feature/mb/elasti…

c90a391

…csearch/shard_xpack_flag # Conflicts: # metricbeat/module/elasticsearch/fields.go # metricbeat/module/elasticsearch/ml_job/data.go

Fix conflict

da85d1d

sayden merged commit 2c5742f into elastic:feature-stack-monitoring-mb-ecs Dec 15, 2020

sayden mentioned this pull request Apr 27, 2021

[Metricbeat] Remove xpack enabled flag on ES, Logstash, Beats and Kibana #24427

Merged

6 tasks

leweafan pushed a commit to leweafan/beats that referenced this pull request Apr 28, 2023

Make elasticsearch/shard metricset work for Stack Monitoring without …

306b025

…xpack.enabled flag (elastic#21389)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make elasticsearch/shard metricset work for Stack Monitoring without xpack.enabled flag #21389

Make elasticsearch/shard metricset work for Stack Monitoring without xpack.enabled flag #21389

sayden commented Sep 29, 2020

elasticmachine commented Sep 29, 2020

elasticmachine commented Sep 29, 2020

elasticmachine commented Sep 29, 2020 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

`Terraform Apply on x-pack/metricbeat/module/aws`

`Terraform Apply on x-pack/metricbeat/module/aws`

Test stats 🧪

chrisronline commented Oct 19, 2020

sayden commented Oct 28, 2020

chrisronline commented Oct 29, 2020

sayden commented Nov 3, 2020

elasticmachine commented Nov 6, 2020

Test stats 🧪

Genuine test errors

ycombinator commented Nov 9, 2020

chrisronline commented Nov 9, 2020

ycombinator commented Nov 9, 2020

chrisronline commented Nov 10, 2020

ycombinator commented Nov 11, 2020

chrisronline commented Nov 11, 2020

chrisronline commented Nov 11, 2020

sayden commented Nov 11, 2020

ycombinator Dec 10, 2020

sayden Dec 10, 2020

ycombinator left a comment

ycombinator left a comment

Make elasticsearch/shard metricset work for Stack Monitoring without xpack.enabled flag #21389

Make elasticsearch/shard metricset work for Stack Monitoring without xpack.enabled flag #21389

Conversation

sayden commented Sep 29, 2020

elasticmachine commented Sep 29, 2020

elasticmachine commented Sep 29, 2020

elasticmachine commented Sep 29, 2020 • edited by jenkins-beats-ci bot Loading

💚 Build Succeeded

Build stats

Test stats 🧪

Steps errors

Terraform Apply on x-pack/metricbeat/module/aws

Terraform Apply on x-pack/metricbeat/module/aws

💚 Flaky test report

Test stats 🧪

chrisronline commented Oct 19, 2020

sayden commented Oct 28, 2020

chrisronline commented Oct 29, 2020

sayden commented Nov 3, 2020

elasticmachine commented Nov 6, 2020

🐛 Flaky test report

Test stats 🧪

Genuine test errors

ycombinator commented Nov 9, 2020

chrisronline commented Nov 9, 2020

ycombinator commented Nov 9, 2020

chrisronline commented Nov 10, 2020

ycombinator commented Nov 11, 2020

chrisronline commented Nov 11, 2020

chrisronline commented Nov 11, 2020

sayden commented Nov 11, 2020

ycombinator Dec 10, 2020

Choose a reason for hiding this comment

sayden Dec 10, 2020

Choose a reason for hiding this comment

ycombinator left a comment

Choose a reason for hiding this comment

ycombinator left a comment

Choose a reason for hiding this comment

elasticmachine commented Sep 29, 2020 •

edited by jenkins-beats-ci bot

Loading

`Terraform Apply on x-pack/metricbeat/module/aws`

`Terraform Apply on x-pack/metricbeat/module/aws`