Migrate to ecs@mappings template

[zmoog]

This issue tracks the activities needed to achieve the following goals in the Agent integrations ecosystem:

• Stop using the legacy mappings
• Start using the modern ecs@mappings from Elasticsearch

### Tasks
- [ ] https://github.com/zmoog/public-notes/issues/78
- [ ] https://github.com/elastic/package-spec/pull/726
- [ ] https://github.com/elastic/elastic-package/pull/1711
- [ ] https://github.com/elastic/integrations/pull/9268

[zmoog]

I spent some time trying to learn what we have now and what we need to do in Step 3.

What we have now

I see elastic-package, when users set dependencies.ecs.import_mappings to true, can:

• Import the ECS schema to validate the fields.
• Embed a "legacy" version of the ECS dynamic mappings.

What we need to do in the step 3

My current understanding is we should:

• Continue using ECS schema to validate fields
• Stop embedding the "legacy" ECS dynamic mappings
• Start leveraging the ecs@mappings component template when it is available for integrations.

Questions

We currently use dependencies.ecs.reference to understand which ECS version the package targets. However, once we add ecs@mappings to the integration component template, it will be available out-of-the-box from ES. Should we add some mechanism to the package-spec to avoid this from happening automatically, and allow developers to test their packages before opt-in to ecs@mappings?

[jsoriano]

Continue using ECS schema to validate fields

Yes, we need to continue doing this in elastic-package. What we can maybe change now is to somehow obtain the same mapping Elasticsearch is using, maybe copying it directly from Elasticsearch repo (from here?).

• Stop embedding the "legacy" ECS dynamic mappings
• Start leveraging the ecs@mappings component template when it is available for integrations.

I wonder if this could be handled by Fleet. The logic could be like this:

• If the manifest contains dynamic templates prefixed by _embedded_ecs- (under elasticsearch.index_template.mappings.dynamic_templates)
  • These dynamic templates are not installed.
  • The ecs@mappings component template is added.

Or maybe Fleet can just always add the ecs@mappings component template, and ignore dynamic mappings prefixed by _embedded_ecs-. @kpollich wdyt?

Eventually elastic-package should stop including the _embedded-ecs- dynamic mappings in any case, at least for recent versions of Kibana.

Should we add some mechanism to the package-spec to avoid this from happening automatically, and allow developers to test their packages before opt-in to ecs@mappings?

As ecs@mappings is now it looks like a safer default for dynamic mappings in our use cases. Do we want this to continue being opt-in? Would it make sense to install it in the data streams of all packages?

[zmoog]

Continue using ECS schema to validate fields

Yes, we need to continue doing this in elastic-package. What we can maybe change now is to somehow obtain the same mapping Elasticsearch is using, maybe copying it directly from Elasticsearch repo (from here?).

These are the ecs@mappings dynamic templates.

With "ECS schema" I was referring to https://github.com/elastic/ecs/blob/main/generated/ecs/ecs_nested.yml: Isn't elastic-package using ecs_nested.yml to validate the fields that the integration defines in the fields.yml and other .yml files?

Or maybe Fleet can just always add the ecs@mappings component template, and ignore dynamic mappings prefixed by _embedded_ecs-

Here is the related change in Kibana to always add the ecs@mappings component template: elastic/kibana#174855

Eventually elastic-package should stop including the _embedded-ecs- dynamic mappings in any case, at least for recent versions of Kibana.

Yep, we should stop including the _embedded-ecs- dynamic templates starting from the Kibana version that will begin adding the ecs@mappings to the integration index template.

On new stack versions, adding ecs@mappings will be the responsibility of Elasticsearch and Kibana, not elastic-package.

Currently, there are ~40 security-focused integrations currently leveraging import_mappings: true.

As ecs@mappings is now it looks like a safer default for dynamic mappings in our use cases. Do we want this to continue being opt-in? Would it make sense to install it in the data streams of all packages?

Yep, the consensus seems to gravitate towards installing ecs@mappings in the data streams of all packages.

We were wondering if we need to provide an opt-out mechanism for not adding ecs@mappings to the index template. This can help if an existing integration runs into an issue with ecs@mappings.

[zmoog]

I am trying to summarize the possible following actions and some questions on aspects that probably need more work:

• Remove the Legacy ECS Mappings from elastic-package.
• Validate the fields that the integration defines in the fields.yml and other .yml files using the "ECS schema" at https://github.com/elastic/ecs/blob/main/generated/ecs/ecs_nested.yml
• On new stack versions:
  • Adding ecs@mappings will be the responsibility of Elasticsearch and Kibana, not elastic-package thanks to Add reference to ecs@mappings for each index template kibana#174855
  •  installing ecs@mappings in the data streams of all packages.
    • ❓Question: should we provide a mechanism to opt-out from including ecs@mappigns in the integration's index template (requires a change in Fleet)
• ❓Question: should we identify the minimum ECS version supported by the integration using dependencies.ecs.reference, or something like the min stack version at conditions.kibana.version?

cc @P1llus

[felixbarny]

❓Question: should we provide a mechanism to opt-out from including ecs@mappigns in the integration's index template (requires a change in Fleet)

I think it's a good and prudent idea to add an opt-out mechanism here. While we don't anticipate issues, and ecs@mappings only applys to fields that are not specifically mapped in the fields yml, it seems like it doesn't hurt to have an escape hatch if we do notice an unforeseen issue in a specific integration. It would give us a quicker path to resolve such issues by creating a new version of an integration that opts-out of importing ecs@mappings.

❓Question: should we identify the minimum ECS version supported by the integration using dependencies.ecs.reference, or something like the min stack version at conditions.kibana.version?

The flag to specify the minimum stack version (conditions.kibana.version) still makes sense. Integrations that rely on ecs@mappings, and don't add a specific fields.yml will need a min stack version of 8.13. However, integrations can't pin the ecs mappings to a specific version anymore. They'll just import the current version. Therefore, we'll need to ensure that the changes in ecs@mappings are strictly additive.

[zmoog]

@jsoriano, is dependencies.ecs.reference currently used to get field definitions in the ecs.yml file AND download the ECS schema from https://github.com/elastic/ecs, right?

@P1llus, do you think we may need to migrate the ~40 security-focused integrations currently using legacy ecs@mappings embedded by elastic-package?

[P1llus]

@jsoriano, is dependencies.ecs.reference currently used to get field definitions in the ecs.yml file AND download the ECS schema from https://github.com/elastic/ecs, right?

@P1llus, do you think we may need to migrate the ~40 security-focused integrations currently using legacy ecs@mappings embedded by elastic-package?

I would say that if we plan on removing the option in build.yml + deprecating the legacy template (which we should), then we do need to remove that line from each build.yml file and bump the package versions for any integration using it. We also need to update the min version of those packages to the version where this new template is actually available and working for integrations, so that there is no conflict.
If we dont do that then we will simply be missing all ECS mapping if users are on new integration versions but older stack versions.

[zmoog]

@P1llus @jsoriano, we plan to:

• Remove dependencies.ecs.import_mappings: true from integration using it, and set 8.13 as min stack version to switch from legacy to modern ecs@mappings.
• Deprecate dependencies.ecs.import_mappings.
• Document how to migrate to modern ecs@mappings.
• Share how to adopt modern ecs@mappings at the following integrations office hours meeting.

TIL, the index templates get an update during stack upgrades; I didn't expect this, but the index template for Azure the Blob Storage integration received the ecs@mappings from 8.12 to 8.13. See zmoog/public-notes#78 for more.

[jsoriano]

• Remove dependencies.ecs.import_mappings: true from integration using it, and set 8.13 as min stack version to switch from legacy to modern ecs@mappings.

If we make Fleet to ignore mappings prefixed with _embedded-ecs- when ecs@mappins is used, do we need changes in packages?

[zmoog]

• Remove dependencies.ecs.import_mappings: true from integration using it, and set 8.13 as min stack version to switch from legacy to modern ecs@mappings.

If we make Fleet to ignore mappings prefixed with _embedded-ecs- when ecs@mappings is used, do we need changes in packages?

If we ignore dynamic mappings prefixed with _embedded-ecs- ¹ when ecs@mappings is used (starting from 8.13) I think we don't need to update the packages.

Footnotes

1. Coming from https://github.com/elastic/elastic-package/blob/main/internal/builder/_static/ecs_mappings.yaml ↩

[P1llus]

We do have tooling to modify these packages though, I dont see why we would want to leave the settings there when the amount of work to remove it is so small?

[jsoriano]

We do have tooling to modify these packages though, I dont see why we would want to leave the settings there when the amount of work to remove it is so small?

My concern is more on needing to increase the kibana version to 8.13. If we keep the current approach we keep support for more stack versions.