Make elasticsearch/enrich metricset work for Stack Monitoring without xpack.enabled flag #20821

sayden · 2020-08-27T11:37:06Z

Ready to test in Kibana. data.json is included to help troubleshooting and it has been generating testing the Metricbeat binary directly with Elasticsearch

elasticmachine · 2020-08-27T11:37:08Z

Pinging @elastic/integrations-services (Team:Services)

elasticmachine · 2020-08-27T11:52:51Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: [Pull request #20821 updated]
Start Time: 2020-11-18T22:16:37.327+0000
Duration: 74 min 39 sec

Test stats 🧪

Test	Results
Failed	0
Passed	2250
Skipped	502
Total	2752

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	2250
Skipped	502
Total	2752

elasticmachine · 2020-09-11T00:58:05Z

Pinging @elastic/stack-monitoring (Stack monitoring)

sayden · 2020-10-28T11:31:49Z

I added some fields into fields.yml but they weren't used before anyways. It's just for completeness.

chrisronline

We do not currently use this data in the Stack Monitoring UI. I'll approve the PR but I'm not sure how we want to handle this. I'm also fine with not doing this work since we don't use the data.

andresrc · 2020-10-29T17:27:34Z

I would say that given that we have already done it, let's merge it and we can decide what to do as part of the migration.

Thanks,

@ycombinator I think this is ready for your final review, thanks!

metricbeat/module/elasticsearch/enrich/data.go

metricbeat/module/elasticsearch/enrich/_meta/fields.yml

metricbeat/module/elasticsearch/enrich/data.go

metricbeat/module/elasticsearch/enrich/_meta/fields.yml

# Conflicts: # metricbeat/docs/fields.asciidoc # metricbeat/module/elasticsearch/fields.go # metricbeat/module/elasticsearch/node_stats/_meta/fields.yml

elasticmachine · 2020-11-05T21:41:56Z

🐛 Flaky test report

❕ There are test failures but not known flaky tests.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	2
Passed	2248
Skipped	500
Total	2750

Genuine test errors

💔 There are test failures but not known flaky tests, most likely a genuine test failure.

Name: Build&Test / x-pack/metricbeat-build / test_migration – x-pack.metricbeat.tests.system.test_xpack_base.Test
Name: Build&Test / x-pack/metricbeat-build / test_template – x-pack.metricbeat.tests.system.test_xpack_base.Test

metricbeat/module/elasticsearch/enrich/_meta/data.json

metricbeat/module/elasticsearch/enrich/data.go

ycombinator

Left a couple of comments.

…without xpack.enabled flag (elastic#20615)

ycombinator · 2020-11-10T23:04:02Z

metricbeat/module/elasticsearch/enrich/_meta/data.json

-            "name": "docker-cluster"
-        },
-        "enrich": {
-            "executed_searches": {


If I'm following this correctly, we are introducing a breaking change here, as we're going from elasticsearch.enrich.executed_searches to elasticsearch.enrich.coordinator_stats.executed_searches. Same applies to other fields as well. We should try to prevent this, maybe by introducing field aliases from the old to the new fields?

ycombinator · 2020-11-10T23:05:51Z

Does there need to be a fields.yml in this PR?

sayden · 2020-11-11T20:29:47Z

Does there need to be a fields.yml in this PR?

Yes. I have been struggling to run the "fields are documented" test in local until today and now I could manage to add all the missing fields in the mapping

ycombinator · 2020-11-12T23:54:49Z

CI failures are looking better now. CI is still failing but the failures look unrelated to this PR. Maybe rebasing the feature branch on master and then this PR's branch on the feature branch might help get this PR green?
Not sure where this stands: Make elasticsearch/enrich metricset work for Stack Monitoring without xpack.enabled flag #20821 (comment)
There are some merge conflicts that need resolving.

…csearch/enrich_xpack_flag # Conflicts: # metricbeat/docs/fields.asciidoc # metricbeat/module/elasticsearch/fields.go # metricbeat/module/elasticsearch/node_stats/_meta/fields.yml

… xpack.enabled flag (elastic#20821)

sayden added Metricbeat Metricbeat Team:Services (Deprecated) Label for the former Integrations-Services team labels Aug 27, 2020

sayden requested a review from ycombinator August 27, 2020 11:37

sayden self-assigned this Aug 27, 2020

botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Aug 27, 2020

ycombinator added the Feature:Stack Monitoring label Sep 11, 2020

chrisronline approved these changes Oct 29, 2020

View reviewed changes

ycombinator reviewed Oct 30, 2020

View reviewed changes

metricbeat/module/elasticsearch/enrich/data.go Outdated Show resolved Hide resolved

ycombinator reviewed Oct 30, 2020

View reviewed changes

metricbeat/module/elasticsearch/enrich/_meta/fields.yml Outdated Show resolved Hide resolved

ycombinator reviewed Oct 30, 2020

View reviewed changes

metricbeat/module/elasticsearch/enrich/data.go Outdated Show resolved Hide resolved

sayden force-pushed the feature/mb/elasticsearch/enrich_xpack_flag branch from 76a10b6 to 6cbc265 Compare November 3, 2020 16:22

sayden commented Nov 3, 2020

View reviewed changes

metricbeat/module/elasticsearch/enrich/_meta/fields.yml Outdated Show resolved Hide resolved

sayden added 2 commits November 5, 2020 12:15

Merge branch 'master' into feature-stack-monitoring-mb-ecs

b1903d4

# Conflicts: # metricbeat/docs/fields.asciidoc # metricbeat/module/elasticsearch/fields.go # metricbeat/module/elasticsearch/node_stats/_meta/fields.yml

Make update after rebasing and fixing conflicts in fields.yml

abc4f69

sayden force-pushed the feature/mb/elasticsearch/enrich_xpack_flag branch from 6cbc265 to 9e195af Compare November 5, 2020 20:36

sayden requested a review from ycombinator November 5, 2020 20:36

ycombinator reviewed Nov 8, 2020

View reviewed changes

metricbeat/module/elasticsearch/enrich/_meta/data.json Show resolved Hide resolved

ycombinator reviewed Nov 8, 2020

View reviewed changes

metricbeat/module/elasticsearch/enrich/data.go Outdated Show resolved Hide resolved

ycombinator reviewed Nov 8, 2020

View reviewed changes

Make elasticsearch/index_summary metricset work for Stack Monitoring …

58e2a7d

…without xpack.enabled flag (elastic#20615)

sayden force-pushed the feature/mb/elasticsearch/enrich_xpack_flag branch from 9e195af to a55741a Compare November 10, 2020 16:40

sayden added 2 commits November 10, 2020 17:40

Atomic commit

ca1051f

Update fields.yml and data.json

01ae994

sayden added 2 commits November 10, 2020 17:41

Adress comments

9951561

Fix conflict and revert "count" fields to "total"

001c44c

sayden force-pushed the feature/mb/elasticsearch/enrich_xpack_flag branch from a55741a to 001c44c Compare November 10, 2020 16:43

ycombinator reviewed Nov 10, 2020

View reviewed changes

sayden added 3 commits November 11, 2020 16:24

Add missing fields

7b0b0a7

Fix failing test?

ffd869e

Actually fix failing test

e359e96

sayden force-pushed the feature-stack-monitoring-mb-ecs branch from 58e2a7d to 9de5959 Compare November 12, 2020 11:44

sayden added 2 commits November 18, 2020 21:55

Merge branch 'feature-stack-monitoring-mb-ecs' into feature/mb/elasti…

d8449b6

…csearch/enrich_xpack_flag # Conflicts: # metricbeat/docs/fields.asciidoc # metricbeat/module/elasticsearch/fields.go # metricbeat/module/elasticsearch/node_stats/_meta/fields.yml

Fix mapping

56f45ac

sayden merged commit 1b5d63f into elastic:feature-stack-monitoring-mb-ecs Nov 19, 2020

sayden mentioned this pull request Apr 27, 2021

[Metricbeat] Remove xpack enabled flag on ES, Logstash, Beats and Kibana #24427

Merged

6 tasks

leweafan pushed a commit to leweafan/beats that referenced this pull request Apr 28, 2023

Make elasticsearch/enrich metricset work for Stack Monitoring without…

cbc0582

… xpack.enabled flag (elastic#20821)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make elasticsearch/enrich metricset work for Stack Monitoring without xpack.enabled flag #20821

Make elasticsearch/enrich metricset work for Stack Monitoring without xpack.enabled flag #20821

sayden commented Aug 27, 2020 •

edited

Loading

elasticmachine commented Aug 27, 2020

elasticmachine commented Aug 27, 2020 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

Test stats 🧪

elasticmachine commented Sep 11, 2020

sayden commented Oct 28, 2020

chrisronline left a comment

andresrc commented Oct 29, 2020

elasticmachine commented Nov 5, 2020 •

edited by jenkins-beats-ci bot

Loading

Test stats 🧪

Genuine test errors

ycombinator left a comment

ycombinator Nov 10, 2020

ycombinator Nov 12, 2020

ycombinator commented Nov 10, 2020

sayden commented Nov 11, 2020

ycombinator commented Nov 12, 2020

Make elasticsearch/enrich metricset work for Stack Monitoring without xpack.enabled flag #20821

Make elasticsearch/enrich metricset work for Stack Monitoring without xpack.enabled flag #20821

Conversation

sayden commented Aug 27, 2020 • edited Loading

elasticmachine commented Aug 27, 2020

elasticmachine commented Aug 27, 2020 • edited by jenkins-beats-ci bot Loading

💚 Build Succeeded

Build stats

Test stats 🧪

💚 Flaky test report

Test stats 🧪

elasticmachine commented Sep 11, 2020

sayden commented Oct 28, 2020

chrisronline left a comment

Choose a reason for hiding this comment

andresrc commented Oct 29, 2020

elasticmachine commented Nov 5, 2020 • edited by jenkins-beats-ci bot Loading

🐛 Flaky test report

Test stats 🧪

Genuine test errors

ycombinator left a comment

Choose a reason for hiding this comment

ycombinator Nov 10, 2020

Choose a reason for hiding this comment

ycombinator Nov 12, 2020

Choose a reason for hiding this comment

ycombinator commented Nov 10, 2020

sayden commented Nov 11, 2020

ycombinator commented Nov 12, 2020

sayden commented Aug 27, 2020 •

edited

Loading

elasticmachine commented Aug 27, 2020 •

edited by jenkins-beats-ci bot

Loading

elasticmachine commented Nov 5, 2020 •

edited by jenkins-beats-ci bot

Loading