[EPM] Use index templates v2

[skh]

Summary

Implements the alpha1 / 7.8 parts of #63950

• handles v1 and v2 prebuilt index templates in the base package
• only generates v2 index templates for package datasets

This should not affect how we update packages, as long as we don't use index template composition for generated v2 index templates.

How to test this

• Run this PR against our default package registry, with a fresh elasticsearch database (e.g. by restarting yarn es snapshot).
• Kibana should start without errors, login should work without errors.
• Verify in the Kibana dev console or with curl that after login, GET _cat/templates correctly lists all packages from the base package that was installed during setup (that is, event-mappings, event-settings, logs-mappings, logs-settings, metrics-mappings and metrics-settings.

---

• Then run Use index templates v2 in package 'base' package-registry#392 locally and run this PR with xpack.ingestManager.epm.registryUrl set to http://localhost:8080, against another fresh elasticsearch database.
• Kibana should start without errors, login should work without errors.
• Verify in the Kibana dev console or with curl that after login, GET _index_template and GET _component_template correctly return all v2 index templates and component templates that were added / changed in Use index templates v2 in package 'base' package-registry#392.

---

• Now verify (running against either package registry) that GET _index_template correctly returns a v2 index template for every dataset from the system package.
• Add a datasource for the nginx package, thereby triggering the installation of this package. This should work without error too. Again, verify with GET _index_template that index templates have been installed for all datasets of the package.
• In the UI, delete the newly created nginx-1 datasource, then uninstall the nginx integration. Both operation should complete without errors.
• Verify with GET _index_template that the nginx index templates have been removed.
• Try to break it in any other way!

[elasticmachine]

Pinging @elastic/ingest-management (Team:Ingest Management)

[ruflin]

I tested this with elastic/package-registry#392 and works as expected.

@skh At the moment all the mappings for metrics-system.cpu are directly in this template. As a next step I would suggest to also use components here so we have metrics-system.cpu-mapping component, this will make it natural if we add later metrics-system.cpu-settings.

[skh]

As a next step I would suggest to also use components here so we have metrics-system.cpu-mapping component, this will make it natural if we add later metrics-system.cpu-settings.

For alpha 1?

[skh]

@elasticmachine merge upstream

[ruflin]

@skh Alpha 1 question: No.

[skh]

@ruflin See the 7.9 / beta1 section of #63950.

[skh]

I did forget to handle index template deletion, 351a051 fixes that.

To test this:

• after creating a datasource for nginx, verify that the index templates have been installed correctly with GET _index_template.
• In the UI, delete the newly created nginx-1 datasource, then uninstall the nginx integration. Both operation should complete without errors.
• Verify with GET _index_template that the nginx index templates have been removed.

[skh]

@ruflin thanks for the approval!
@neptunian Your review would be much appreciated. I'm not entirely sure how to test the upgrade scenario.

[kibanamachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request
• Commit: 7b27d50

History

• 💚 Build #43654 succeeded 090abcc
• 💔 Build #43644 failed 3f2be6c
• 💔 Build #43432 failed eeef087
• 💔 Build #43394 failed ea4fdfb
• 💔 Build #43362 failed fab7137

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

[ruflin]

Also tested the deleting part for nginx and templates are removed again on uninstall.