Make elasticsearch/index_recovery metricset work for Stack Monitoring without xpack.enabled flag #20810

sayden · 2020-08-26T20:41:49Z

WIP

Missing steps:

metricset fields.yml
elasticsearch/fields.yml aliases
Manual testing in Kibana

elasticmachine · 2020-08-26T20:58:22Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: Pull request #20810 updated
Start Time: 2020-12-01T14:22:41.882+0000
Duration: 78 min 34 sec

Test stats 🧪

Test	Results
Failed	0
Passed	2246
Skipped	526
Total	2772

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	2246
Skipped	526
Total	2772

elasticmachine · 2020-10-22T10:25:16Z

Pinging @elastic/stack-monitoring (Stack monitoring)

elasticmachine · 2020-10-22T10:25:17Z

Pinging @elastic/integrations-services (Team:Services)

chrisronline

LGTM!

andresrc · 2020-10-29T17:28:05Z

@ycombinator I think this is ready for your final review, thanks!

ycombinator · 2020-10-30T03:54:08Z

metricbeat/module/elasticsearch/index_recovery/_meta/data.json

        },
        "index": {
-            "name": ".monitoring-es-6-2018.11.20",


We are removing this field, which would be a breaking change. We should add it back.

Never mind, I see this has been updated now.

ycombinator · 2020-10-30T04:00:00Z

metricbeat/module/elasticsearch/index_recovery/_meta/data.json

-                    "name": "es1_1"
-                },
-                "type": "EMPTY_STORE"
+                "shards": [


Hmmm, this is a tricky change.

This appears to be a breaking change too. Not so much defining a new elasticsearch.index.recovery.shards field because that's an additive change. But more that we're taking away the elasticsearch.index.recovery.id, elasticsearch.index.recovery.primary, etc. fields.

I think the reason you had to do this is because previously each document produced by the index_recovery metricset represented a single shard, whereas what Stack Monitoring expects each document to contain an array of shards.

@chrisronline what do you need the data type of this elasticsearch.index.recovery.shards field to be for queries to work correctly: object or nested? Depending on that we might need to figure out how we handle this breaking change.

We don't perform any aggregations across this data, so object works fine.

Okay, that's good news.

I'm still concerned about the breaking change in document structure here, for any existing users of the elasticsearch/index_recovery metricset (non-xpack).

@chrisronline How much work would it be to make the UI work on the current (before this PR) structure of elasticsearch/index_recovery metricset (non-xpack) documents where one document == one shard? Keep in mind that this would be code we'd get to clean up in 8.0 as we could introduce a breaking change in document structure in this metricset at that point.

This should be possible. We only access this data from a single code path and adding any kind of funky logic in there is fine, knowing it can go away in 8.0.

Thanks so much @chrisronline ❤️ . Essentially you'll need logic to check for the new structure (1 document == 1 shard) that will be produced by this metricset when its run without the xpack.enabled flag but fall back to the current logic of expecting multiple shard inner documents in a single type: index_recovery document. Hope that makes sense.

In that case, @sayden, we should keep the 1 document == 1 shard structure that is currently (prior to this PR) being produced by this metricset. We can add more fields to that document, if needed for Stack Monitoring, of course.

@ycombinator Yes, that's my understanding. Should be fine

Yeah! Ok! Sounds good! I'll revert to the OSS original structure of 1 doc == 1 shard.

ycombinator · 2020-10-30T04:00:29Z

@sayden I don't see a fields.yml file in this PR.

sayden · 2020-11-03T17:38:04Z

That's because there's no "type": "index_recovery" in the monitoring mapping so the original fields.yml was untouched. I can add the fields though

ycombinator · 2020-11-05T04:27:07Z

Hey @sayden, could you update the PR description please? It's not clear if this PR is still WIP and there are things you are still working on or if it's ready for me to review.

Also, there are failing tests that look related, so I'm assuming you need to do more on this PR before it's ready for me to review again.

In general, for each of these PRs, can I suggest a bit of process please?

Please keep the PR descriptions up to date at all times. Since there are so many such PRs open at the same time, it's hard to keep track of their states in my head. Being able to just look at a PR's description to know where things stand would be nice.
Whenever a PR is ready for my review (code review), please request my review by adding me to the Reviewers section in the top right of the PR:
If I'm already a reviewer and the PR is ready for a re-review, please request my review again by clicking the "re-request review" icon next to my name:

Otherwise right now I'm having to periodically look at each PR and try to guess what state it's in, and whether it's awaiting my review. Thanks!

# Conflicts: # metricbeat/docs/fields.asciidoc # metricbeat/module/elasticsearch/fields.go # metricbeat/module/elasticsearch/node_stats/_meta/fields.yml

elasticmachine · 2020-11-05T18:50:20Z

🐛 Flaky test report

❕ There are test failures but not known flaky tests.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	2
Passed	2248
Skipped	500
Total	2750

Genuine test errors

💔 There are test failures but not known flaky tests, most likely a genuine test failure.

Name: Build&Test / metricbeat-goIntegTest / TestXPackEnabled – elasticsearch
Name: Build&Test / metricbeat-goIntegTest / TestXPackEnabled/index_recovery – elasticsearch

metricbeat/module/elasticsearch/elasticsearch_integration_test.go

…without xpack.enabled flag (elastic#20615)

ycombinator

Code changes LGTM but tests are failing in CI and test failures look relevant.

…csearch/index_recovery_xpack_flag # Conflicts: # metricbeat/module/elasticsearch/fields.go

…csearch/index_recovery_xpack_flag

…csearch/index_recovery_xpack_flag # Conflicts: # metricbeat/module/elasticsearch/fields.go

…csearch/index_recovery_xpack_flag

…csearch/index_recovery_xpack_flag # Conflicts: # metricbeat/module/elasticsearch/fields.go

… without xpack.enabled flag (elastic#20810)

sayden requested a review from ycombinator August 26, 2020 20:41

sayden self-assigned this Aug 26, 2020

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Aug 26, 2020

sayden added Metricbeat Metricbeat Team:Services (Deprecated) Label for the former Integrations-Services team labels Aug 27, 2020

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Aug 27, 2020

ycombinator added the Feature:Stack Monitoring label Sep 11, 2020

sayden marked this pull request as ready for review October 22, 2020 10:25

chrisronline approved these changes Oct 29, 2020

View reviewed changes

ycombinator reviewed Oct 30, 2020

View reviewed changes

sayden force-pushed the feature/mb/elasticsearch/index_recovery_xpack_flag branch from cab7374 to b169e66 Compare November 4, 2020 22:35

sayden added 2 commits November 5, 2020 12:15

Merge branch 'master' into feature-stack-monitoring-mb-ecs

b1903d4

# Conflicts: # metricbeat/docs/fields.asciidoc # metricbeat/module/elasticsearch/fields.go # metricbeat/module/elasticsearch/node_stats/_meta/fields.yml

Make update after rebasing and fixing conflicts in fields.yml

abc4f69

sayden force-pushed the feature/mb/elasticsearch/index_recovery_xpack_flag branch from b169e66 to 02be3b3 Compare November 5, 2020 17:45

sayden commented Nov 6, 2020

View reviewed changes

metricbeat/module/elasticsearch/elasticsearch_integration_test.go Show resolved Hide resolved

sayden added 5 commits November 10, 2020 12:02

Make elasticsearch/index_summary metricset work for Stack Monitoring …

58e2a7d

…without xpack.enabled flag (elastic#20615)

Force push new atomic commit. 1 shard == 1 event

dbaefcd

Fix test that was failing

aa574dd

Add missing field

f9e65e2

Rebase and make update

5f8560d

sayden force-pushed the feature/mb/elasticsearch/index_recovery_xpack_flag branch from b822954 to 5f8560d Compare November 10, 2020 16:51

sayden added 2 commits November 10, 2020 17:54

Add comment to test

6c00189

Add missing fields to mapping

1f26899

ycombinator reviewed Nov 12, 2020

View reviewed changes

sayden force-pushed the feature-stack-monitoring-mb-ecs branch from 58e2a7d to 9de5959 Compare November 12, 2020 11:44

sayden added 14 commits November 19, 2020 11:27

Merge branch 'feature-stack-monitoring-mb-ecs' into feature/mb/elasti…

a1a5209

…csearch/index_recovery_xpack_flag # Conflicts: # metricbeat/module/elasticsearch/fields.go

Rebase and update fields

083ecf8

Skip broken test

bb16dc2

Create TestData test

384faa4

Maybe fix lint?

6739ec4

Merge branch 'feature-stack-monitoring-mb-ecs' into feature/mb/elasti…

f164738

…csearch/index_recovery_xpack_flag

Fix testdata test

f198cc0

Merge branch 'feature-stack-monitoring-mb-ecs' into feature/mb/elasti…

77c3fa3

…csearch/index_recovery_xpack_flag

Merge branch 'feature-stack-monitoring-mb-ecs' into feature/mb/elasti…

e9fc22f

…csearch/index_recovery_xpack_flag # Conflicts: # metricbeat/module/elasticsearch/fields.go

Fix fields.go

36332e3

Merge branch 'feature-stack-monitoring-mb-ecs' into feature/mb/elasti…

4d110c3

…csearch/index_recovery_xpack_flag

Merge branch 'feature-stack-monitoring-mb-ecs' into feature/mb/elasti…

d57b430

…csearch/index_recovery_xpack_flag # Conflicts: # metricbeat/module/elasticsearch/fields.go

Modify fields.go

5bb0551

Fix linting

eca8d7b

sayden merged commit 74491e9 into elastic:feature-stack-monitoring-mb-ecs Dec 1, 2020

sayden mentioned this pull request Apr 27, 2021

[Metricbeat] Remove xpack enabled flag on ES, Logstash, Beats and Kibana #24427

Merged

6 tasks

leweafan pushed a commit to leweafan/beats that referenced this pull request Apr 28, 2023

Make elasticsearch/index_recovery metricset work for Stack Monitoring…

3861294

… without xpack.enabled flag (elastic#20810)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make elasticsearch/index_recovery metricset work for Stack Monitoring without xpack.enabled flag #20810

Make elasticsearch/index_recovery metricset work for Stack Monitoring without xpack.enabled flag #20810

sayden commented Aug 26, 2020

elasticmachine commented Aug 26, 2020 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

Test stats 🧪

elasticmachine commented Oct 22, 2020

elasticmachine commented Oct 22, 2020

chrisronline left a comment

andresrc commented Oct 29, 2020

ycombinator Oct 30, 2020

ycombinator Nov 12, 2020

ycombinator Nov 12, 2020

ycombinator Oct 30, 2020

chrisronline Oct 30, 2020

ycombinator Oct 30, 2020

chrisronline Oct 30, 2020

ycombinator Oct 30, 2020

chrisronline Nov 2, 2020

sayden Nov 3, 2020

ycombinator commented Oct 30, 2020

sayden commented Nov 3, 2020 •

edited

Loading

ycombinator commented Nov 5, 2020 •

edited

Loading

elasticmachine commented Nov 5, 2020 •

edited by jenkins-beats-ci bot

Loading

Test stats 🧪

Genuine test errors

ycombinator left a comment

Make elasticsearch/index_recovery metricset work for Stack Monitoring without xpack.enabled flag #20810

Make elasticsearch/index_recovery metricset work for Stack Monitoring without xpack.enabled flag #20810

Conversation

sayden commented Aug 26, 2020

elasticmachine commented Aug 26, 2020 • edited by jenkins-beats-ci bot Loading

💚 Build Succeeded

Build stats

Test stats 🧪

💚 Flaky test report

Test stats 🧪

elasticmachine commented Oct 22, 2020

elasticmachine commented Oct 22, 2020

chrisronline left a comment

Choose a reason for hiding this comment

andresrc commented Oct 29, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ycombinator commented Oct 30, 2020

sayden commented Nov 3, 2020 • edited Loading

ycombinator commented Nov 5, 2020 • edited Loading

elasticmachine commented Nov 5, 2020 • edited by jenkins-beats-ci bot Loading

🐛 Flaky test report

Test stats 🧪

Genuine test errors

ycombinator left a comment

Choose a reason for hiding this comment

elasticmachine commented Aug 26, 2020 •

edited by jenkins-beats-ci bot

Loading

sayden commented Nov 3, 2020 •

edited

Loading

ycombinator commented Nov 5, 2020 •

edited

Loading

elasticmachine commented Nov 5, 2020 •

edited by jenkins-beats-ci bot

Loading