Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding default templates for Metricbeat ECS data #81744

Merged

Conversation

jbaiera
Copy link
Member

@jbaiera jbaiera commented Dec 14, 2021

This PR adds four new templates that are automatically installed from the Monitoring plugin.

In 8.x, Metricbeat will be writing its data in ECS compliant format, even when used with xpack mode enabled (stack monitoring). In order to continue to support the legacy data format, new mappings have been created with the new ECS fields for indexing data, and alias fields for the legacy format which point to the corresponding ECS fields.

New templates added:

  • Template .monitoring-beats-mb matching on .monitoring-beats-8-* defined in monitoring-beats-mb.json
  • Template .monitoring-es-mb matching on .monitoring-es-8-* defined in monitoring-es-mb.json
  • Template .monitoring-kibana-mb matching on .monitoring-kibana-8-* defined in monitoring-kibana-mb.json
  • Template .monitoring-logstash-mb matching on .monitoring-logstash-8-* defined in monitoring-logstash-mb.json

All templates are being created with internal version 8000099

@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-data-management (Team:Data Management)

@jbaiera
Copy link
Member Author

jbaiera commented Dec 14, 2021

@elasticmachine update branch

@jbaiera
Copy link
Member Author

jbaiera commented Dec 15, 2021

@elasticmachine run elasticsearch-ci/part-2

Copy link
Contributor

@matschaffer matschaffer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some read-through observations, still don't quite have my head wrapped around all this.

docs/reference/monitoring/indices.asciidoc Outdated Show resolved Hide resolved
@@ -0,0 +1,2198 @@
{
"index_patterns": [".monitoring-beats-${xpack.stack.monitoring.template.version}-*"],
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the same pattern as https://github.com/elastic/elasticsearch/blob/master/x-pack/plugin/core/src/main/resources/monitoring-es.json#L2 - are we planning to introduce a new variable to set the mb version to 10?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently the xpack.stack.monitoring.template.version is set to version 8 for the metricbeat indices. We talked it over a bit more and agreed that 8 was indeed the more appropriate version number to put here. The monitoring-es.json file uses the xpack.monitoring.template.version property (doesn't contain stack)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah... xpack.monitoring.template.version vs xpack.stack.monitoring.template.version - I missed the distinction. Thanks!

Co-authored-by: Mat Schaffer <mat@schaffer.me>
@jbaiera
Copy link
Member Author

jbaiera commented Dec 16, 2021

@elasticmachine update branch

@jbaiera jbaiera merged commit 159b1ce into elastic:master Dec 16, 2021
@jbaiera jbaiera deleted the stack-monitoring-add-metricbeat-8-templates branch December 16, 2021 21:14
jbaiera added a commit to jbaiera/elasticsearch that referenced this pull request Dec 16, 2021
This PR adds four new templates that are automatically installed from the Monitoring plugin.

In 8.x, Metricbeat will be writing its data in ECS compliant format, even when used with xpack 
mode enabled (stack monitoring). In order to continue to support the legacy data format, new 
mappings have been created with the new ECS fields for indexing data, and alias fields for the 
legacy format which point to the corresponding ECS fields.

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: Mat Schaffer <mat@schaffer.me>
jbaiera added a commit that referenced this pull request Dec 17, 2021
This PR adds four new templates that are automatically installed from the Monitoring plugin.

In 8.x, Metricbeat will be writing its data in ECS compliant format, even when used with xpack 
mode enabled (stack monitoring). In order to continue to support the legacy data format, new 
mappings have been created with the new ECS fields for indexing data, and alias fields for the 
legacy format which point to the corresponding ECS fields.

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: Mat Schaffer <mat@schaffer.me>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants