[elasticsearch] Align mappings with metricbeat #3928

klacabane · 2022-08-01T20:56:10Z

Summary

Closes #3914

Align the elasticsearch data streams' mappings with the metricbeat ones. Methodology described in this issue

Testing

build the package elastic-package build
start a 8.5.0 stack elastic-package stack up -v --version 8.5.0-SNAPSHOT
include the package to the existing policy pulled by the running Agent
- navigate to https://localhost:5601/app/integrations/detail/elasticsearch-0.3.0/overview
- click "Add elasticsearch"
- disable "Collect Elasticsearch logs"
- edit "Collect Elasticsearch metrics" defaults to

add the integration to the existing "Elastic-Agent" policy
open Stack Monitoring and verify that every views load as expected (besides the ones that load Logs, these will be handled in [Stack Monitoring] Align integration packages ingest pipelines with filebeat kibana#137112)

Known issues

Cgroup CPU Utilization data does not show up by default and we need to update the es service in the elastic-package/docker-compose.yml with cpu limits. Add the following:

deploy:
    resources:
        limits:
            cpus: 1.0

I/O Operations Rate is not populated - this is because we're missing the io_stats values from the node_stats API. some infos in this slack thread

klacabane · 2022-08-01T20:56:54Z

packages/elasticsearch/data_stream/cluster_stats/manifest.yml

+elasticsearch:
+  index_template:
+    mappings:
+      dynamic: false


cluster_stats is the offending metricset that caused the initial issue in #3918 so I'd be inclined to close it at the same time, unless we want to review all metricsets and apply dynamic: false where necessary

I ended up disabling dynamic mappings on every data streams

elasticmachine · 2022-08-01T21:07:35Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2022-08-03T13:35:31.340+0000
Duration: 15 min 3 sec

Test stats 🧪

Test	Results
Failed	0
Passed	31
Skipped	0
Total	31

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

elasticmachine · 2022-08-01T22:47:23Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	100.0% (`0/0`)	💚
Files	100.0% (`0/0`)	💚 2.868
Classes	100.0% (`0/0`)	💚 2.868
Methods	49.057% (`26/53`)	👎 -40.192
Lines	100.0% (`0/0`)	💚 9.393
Conditionals	100.0% (`0/0`)	💚

miltonhultgren

Tested locally and didn't run into any errors, LGTM 👍🏼

miltonhultgren · 2022-08-04T08:20:34Z

packages/elasticsearch/data_stream/ccr/manifest.yml

@@ -1,6 +1,10 @@
 type: metrics
 title: Elasticsearch ccr metrics
 release: experimental
+elasticsearch:


Why is it elasticsearch here? What does that represent?

That's the section allowing more granular control over the assets installed, more options in the package-spec

Ah I see, thanks!

miltonhultgren · 2022-08-04T08:22:01Z

packages/elasticsearch/data_stream/node_stats/fields/fields.yml

@@ -261,10 +261,10 @@
                - name: control_group
                  type: keyword
                - name: limit.bytes
-                  type: long
+                  type: keyword


Should we add some ignore_above here?

this didn't surface as a diff surprisingly but I'll add that where needed

It might not be in the original mappings at all but maybe it's needed, I see it in some places and some places not so I never know

crespocarlos

I was thinking about creating tests for these data streams, but given the fact that we'll deprecate this package, I'm not sure of it's worth the effort. Definitely something to take into consideration for the new integration packages

crespocarlos · 2022-08-04T14:56:53Z

packages/elasticsearch/manifest.yml

@@ -13,7 +13,7 @@ format_version: 1.0.0
 license: basic
 categories: ["elastic_stack", "datastore"]
 conditions:
-  kibana.version: ^7.15.0
+  kibana.version: ^8.5.0


Should we increment the package semver too? if so, I would also add an entry to the changelog file

let's keep that for #3917 so that we can pack all the changes in a single changelog entry

klacabane · 2022-08-04T15:21:30Z

What kind of tests ? Since we have the SM ui as an official consumer of these streams I thought integration tests in Kibana would cover everything we need. PO will be a different story though

crespocarlos · 2022-08-04T15:37:13Z

When I wrote the comment I had pipeline test in mind, but here we only have data streams, so it's not the case. But digging into integration packages doc it seems that it's possible to create tests for data streams too https://github.com/elastic/elastic-package/blob/main/docs/howto/system_testing.md. I've never tried it, so I don't know what the possibilities are.

klacabane · 2022-08-04T15:51:41Z

I'll merge this now as the ignore_above task may take a while, followup here #3959

klacabane added 6 commits August 1, 2022 22:30

align ecs fields

fe67816

add index_stats property group

cc779c9

remove unused version index_recovery property

e74a67e

align node_stats fields type

f7d56b2

turn dynamic off for cluster_stats mappings

f4c3179

bump kibana minimum version

d17cf5c

klacabane added v8.5.0 Team:Infra Monitoring UI - DEPRECATED Label for the Infrastructure Monitoring UI team. - DEPRECATED - Use Team:obs-ux-infra_services labels Aug 1, 2022

klacabane requested a review from a team as a code owner August 1, 2022 20:56

klacabane self-assigned this Aug 1, 2022

klacabane commented Aug 1, 2022

View reviewed changes

lint

8ef350e

klacabane added 2 commits August 3, 2022 15:34

disable dynamic mappings for every data streams

bfec760

add missing shard property

f856c85

miltonhultgren self-requested a review August 4, 2022 12:03

miltonhultgren approved these changes Aug 4, 2022

View reviewed changes

crespocarlos approved these changes Aug 4, 2022

View reviewed changes

klacabane merged commit 09500cc into elastic:main Aug 4, 2022

klacabane mentioned this pull request Aug 31, 2022

Elasticsearch integration: missing index_stats alias on index mappings elastic/kibana#119984

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[elasticsearch] Align mappings with metricbeat #3928

[elasticsearch] Align mappings with metricbeat #3928

klacabane commented Aug 1, 2022 •

edited

klacabane Aug 1, 2022 •

edited

klacabane Aug 3, 2022

elasticmachine commented Aug 1, 2022 •

edited

Build stats

Test stats 🧪

elasticmachine commented Aug 1, 2022 •

edited

miltonhultgren left a comment

miltonhultgren Aug 4, 2022

klacabane Aug 4, 2022

miltonhultgren Aug 4, 2022

miltonhultgren Aug 4, 2022

klacabane Aug 4, 2022

miltonhultgren Aug 4, 2022

crespocarlos left a comment

crespocarlos Aug 4, 2022

klacabane Aug 4, 2022

klacabane commented Aug 4, 2022 •

edited

crespocarlos commented Aug 4, 2022 •

edited

klacabane commented Aug 4, 2022

[elasticsearch] Align mappings with metricbeat #3928

[elasticsearch] Align mappings with metricbeat #3928

Conversation

klacabane commented Aug 1, 2022 • edited

Summary

Testing

Known issues

klacabane Aug 1, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elasticmachine commented Aug 1, 2022 • edited

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

elasticmachine commented Aug 1, 2022 • edited

🌐 Coverage report

miltonhultgren left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

crespocarlos left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

klacabane commented Aug 4, 2022 • edited

crespocarlos commented Aug 4, 2022 • edited

klacabane commented Aug 4, 2022

klacabane commented Aug 1, 2022 •

edited

klacabane Aug 1, 2022 •

edited

elasticmachine commented Aug 1, 2022 •

edited

elasticmachine commented Aug 1, 2022 •

edited

klacabane commented Aug 4, 2022 •

edited

crespocarlos commented Aug 4, 2022 •

edited