diff --git a/reference/fleet/_snippets/change-index-template-priority.md b/reference/fleet/_snippets/change-index-template-priority.md new file mode 100644 index 0000000000..d8fe3460ea --- /dev/null +++ b/reference/fleet/_snippets/change-index-template-priority.md @@ -0,0 +1,8 @@ +:::{dropdown} How to change an index template's priority +1. Go to the **Index Management** page using the navigation menu or the [global search field](/explore-analyze/find-and-organize/find-apps-and-objects.md), and open the **Index Templates** tab. +2. Find and select the index template you want to change, then select **Manage** → **Edit**. +3. On the **Logistics** step, update the **Priority** value. +4. Continue to the **Review** step, then save the template. + +For more about creating and managing index templates, refer to [Index templates](/manage-data/data-store/templates.md#index-templates). +::: diff --git a/reference/fleet/data-streams-ilm-tutorial.md b/reference/fleet/data-streams-ilm-tutorial.md index 5f85d319a4..e279f6f2a0 100644 --- a/reference/fleet/data-streams-ilm-tutorial.md +++ b/reference/fleet/data-streams-ilm-tutorial.md @@ -30,4 +30,7 @@ How you apply an ILM policy depends on your use case. Choose a scenario for the * **[Scenario 1](/reference/fleet/data-streams-scenario1.md)**: You want to apply an ILM policy to all logs or metrics data streams across all namespaces. * **[Scenario 2](/reference/fleet/data-streams-scenario2.md)**: You want to apply an ILM policy to selected data streams in an integration. * **[Scenario 3](/reference/fleet/data-streams-scenario3.md)**: You want to apply an ILM policy for data streams in a selected namespace in an integration. -* **[Scenario 4](/reference/fleet/data-streams-scenario4.md)**: You want to apply an ILM policy to all data streams in a custom integration using a `@custom` component template. {applies_to}`stack: ga 9.1` \ No newline at end of file + + {applies_to}`stack: ga 9.5+` For shared settings across every data stream in a namespace, refer to [Customize data streams with namespace index templates](/reference/fleet/data-streams-namespace-custom.md). + +* **[Scenario 4](/reference/fleet/data-streams-scenario4.md)**: You want to apply an ILM policy to all data streams in a custom integration using a `@custom` component template. {applies_to}`stack: ga 9.1` diff --git a/reference/fleet/data-streams-namespace-custom.md b/reference/fleet/data-streams-namespace-custom.md new file mode 100644 index 0000000000..1acd56d6e0 --- /dev/null +++ b/reference/fleet/data-streams-namespace-custom.md @@ -0,0 +1,154 @@ +--- +navigation_title: Customize data streams with namespace index templates +description: Enable namespace index templates so you can apply shared settings and mappings across a Fleet integration's data streams in one namespace. +applies_to: + stack: ga 9.5+ + serverless: ga +products: + - id: fleet + - id: elastic-agent +--- + +# Customize data streams with namespace index templates [data-streams-namespace-custom] + +Namespace index templates are {{fleet}}-managed [index templates](/manage-data/data-store/templates.md#index-templates) scoped to one namespace of an installed integration. When you opt a namespace in, {{fleet}} creates one of these templates for each data stream defined by the integration, and each template references a `@custom` component template that you create and manage. + +Use this feature when you want the same custom settings or mappings on every data stream in a specific namespace for a given integration, instead of editing each data stream's `@custom` component template separately. + +## How namespace index templates work [data-streams-namespace-custom-how] + +When you enable namespace index templates for a namespace on an installed integration, {{fleet}} creates one namespace index template for each data stream defined by the integration. The templates are named using the following pattern: + +```text +-@namespace. +``` + +For example, enabling the `production` namespace for the System integration creates templates such as `logs-system.application@namespace.production`. + +Each namespace index template is a copy of the integration's base data stream index template, but differs in these ways: + +* The `index_patterns` value is scoped to that namespace (for example, `logs-system.application-production*`). +* The priority is the base template priority plus 50: `250` for most integrations (base `200`), or `200` for integrations with `dataset_is_prefix: true` data streams (base `150`). Because this is higher than the base template priority, {{es}} applies the namespace index template for matching data streams. If you change this priority manually, {{fleet}} restores the default the next time the integration's templates are reinstalled or resynchronized. +* The `composed_of` list includes a `@custom` component template (for example, `production@custom`) after the type-level and package-level `@custom` templates, and before the data stream-level `@custom` template. + +Example `composed_of` list for `logs-system.application@namespace.production`: + +```json +[ + "logs@mappings", + "logs@settings", + "logs-system.application@package", + "logs@custom", + "system@custom", + "production@custom", + "logs-system.application@custom", + "ecs@mappings", + ".fleet_globals-1", + ".fleet_agent_id_verification-1" +] +``` + +Later component templates in this list take precedence when the same setting or mapping key appears more than once. From highest to lowest precedence among the `@custom` templates: `logs-system.application@custom`, then `production@custom`, then `system@custom`, then `logs@custom`. + +::::{note} +{{fleet}} does not create the `@custom` component template. You create and manage that template yourself. Until you create it, the reference in `composed_of` has no effect. The same `@custom` template is shared by namespace name across every integration that has that namespace opted in. +:::: + +The namespaces you opt in for an integration are the same across the whole cluster. The index templates and component templates {{fleet}} creates are also shared across the cluster. + +If your {{kib}} space restricts which namespaces are allowed, you can only enable namespace index templates for namespaces that match those restrictions. + +## Enable namespace index templates in the UI [data-streams-namespace-custom-ui] + +You can enable namespace index templates from the integration settings, or while creating or editing an integration policy. + +:::{dropdown} From the integration's Settings tab +:open: + +1. Find **Integrations** in the navigation menu or use the [global search field](/explore-analyze/find-and-organize/find-apps-and-objects.md). +2. Open the **Installed integrations** tab, then select the integration. +3. Open the **Settings** tab. +4. In the **Namespace index templates** section, add the namespace you want to customize. +5. Save your changes. +::: + +:::{dropdown} From the integration policy editor +:open: + +1. Find **Integrations** in the navigation menu or use the [global search field](/explore-analyze/find-and-organize/find-apps-and-objects.md). +2. Select the integration you want, then create a new integration policy or select an existing integration policy from the **Integration policies** tab. +3. In the **Integration settings** section, expand **Advanced options**. +4. In the **Namespace** field, enter the namespace you want to customize. +5. Turn on **Use dedicated index templates for this namespace**. +6. Save the policy. +::: + +After you save, {{fleet}} creates the namespace index templates asynchronously. You can confirm they exist on the integration's **Assets** tab, or in the **Index Management** → **Index Templates** list by searching for `@namespace.`. + +::::{important} +Namespace index templates are shared across all integration policies for that integration and namespace. Enabling or turning them off for one policy updates the opt-in list for the entire integration. +:::: + +## Create the namespace `@custom` component template [data-streams-namespace-custom-component] + +1. Find **Index Management** in the navigation menu or use the [global search field](/explore-analyze/find-and-organize/find-apps-and-objects.md), and open the **Component Templates** tab. +2. Select **Create component template**. +3. Name the template using the pattern `@custom` (for example, `production@custom`). +4. Add the index settings, mappings, or aliases you want applied to every data stream in that namespace for opted-in integrations. +5. Create the component template. +6. Roll over each affected data stream so new backing indices pick up the changes. For example: + + ```console + POST logs-system.application-production/_rollover + ``` + +## Enable namespace index templates with the API [data-streams-namespace-custom-api] + +You can also manage opt-in with the {{fleet}} package APIs. Template create and delete operations run asynchronously after the request succeeds. + +To enable one or more namespaces for a single installed package: + +```console +PUT kbn:/api/fleet/epm/packages/system +{ + "namespace_customization_enabled_for": ["production"] +} +``` + +::::{note} +For a single package, `namespace_customization_enabled_for` replaces the full opt-in list. To keep existing namespaces, include them in the array. To turn a namespace off, omit it from the array, or pass an empty array to clear the list. +:::: + +To enable namespaces or remove them from the opt-in list across multiple packages in one request: + +```console +POST kbn:/api/fleet/epm/packages/_bulk_namespace_customization +{ + "packages": ["system", "nginx", "apache"], + "enable": ["production"], + "disable": ["staging"] +} +``` + +The bulk endpoint is additive and subtractive: `enable` adds namespaces to each package's opt-in list, and `disable` removes them. It does not replace the full list. + +## Turn off namespace index templates [data-streams-namespace-custom-disable] + +You can remove a namespace from the opt-in list in either of these ways: + +* On the integration **Settings** tab, remove the namespace from **Namespaces with dedicated index templates**, then save. +* In the integration policy editor, turn off **Use dedicated index templates for this namespace**, then save. + +When you remove a namespace from the opt-in list, {{fleet}} deletes the corresponding namespace index templates. The `@custom` component template is left in place so you can reuse it later. + +## Overlapping index templates [data-streams-namespace-custom-conflicts] + +If you previously duplicated a base data stream index template and gave the copy a higher priority, that copy can overlap with the {{fleet}}-managed namespace index template. When you opt a namespace in, {{fleet}} warns you about overlapping templates and what will happen for each one. + +For details and resolution steps, refer to [Resolve overlapping index templates for namespace customization](/reference/fleet/data-streams-template-conflicts.md). + +## Related pages [data-streams-namespace-custom-related] + +* [{{agent}} data streams for {{fleet}}](/reference/fleet/data-streams.md) +* [Edit the {{es}} index template](/reference/fleet/data-streams.md#data-streams-index-templates-edit) +* [Resolve overlapping index templates for namespace customization](/reference/fleet/data-streams-template-conflicts.md) diff --git a/reference/fleet/data-streams-scenario3.md b/reference/fleet/data-streams-scenario3.md index cf073e21c7..8a64dffb4c 100644 --- a/reference/fleet/data-streams-scenario3.md +++ b/reference/fleet/data-streams-scenario3.md @@ -78,15 +78,15 @@ metrics-system.network-production@custom ## Step 3: Clone and modify the existing index template [data-streams-ilm-three] -Now that you’ve created a component template, you need to create an index template to apply the changes to the correct data stream. The easiest way to do this is to duplicate and modify the integration’s existing index template. - -::::{warning} -* If you duplicate an index template, do not change or remove any managed properties. This may result in problems when upgrading. Cloning the index template of an integration package involves some risk as any changes made to the original index template are not propagated to the cloned version when you upgrade versions. -* These steps assume that you want to have a namespace specific ILM policy, which requires index template cloning. Cloning the index template of an integration package involves some risk because any changes made to the original index template as part of package upgrades are not propagated to the cloned version. Check out [Cloning the index template of an integration package](/reference/fleet/integrations-assets-best-practices.md#assets-restrictions-cloning-index-template) for details. -* If you want to change the ILM policy, the number of shards, or other settings for the data streams of one or more integrations, but **the changes do not need to be specific to a given namespace**, use a `@custom` component template, as described in [Scenario 1](/reference/fleet/data-streams-scenario1.md) and [Scenario 2](/reference/fleet/data-streams-scenario2.md), to avoid the problems mentioned earlier. Check out the [ILM](/reference/fleet/data-streams.md#data-streams-ilm) section for details. +Now that you’ve created a component template, you need an index template scoped to the correct data stream and namespace. This scenario duplicates and modifies the integration’s existing index template. +::::{important} +Duplicating an integration index template is risky: don't change or remove managed properties, and remember that package upgrades don't update your copy. For details, refer to [Cloning the index template of an integration package](/reference/fleet/integrations-assets-best-practices.md#assets-restrictions-cloning-index-template). :::: +If your changes don't need to be namespace-specific, use a `@custom` component template instead, as described in [Scenario 1](/reference/fleet/data-streams-scenario1.md) and [Scenario 2](/reference/fleet/data-streams-scenario2.md). For more about {{ilm-init}} with {{fleet}} data streams, refer to [Index lifecycle management ({{ilm-init}})](/reference/fleet/data-streams.md#data-streams-ilm). + +{applies_to}`stack: ga 9.5+` If you want the same settings for every data stream in a namespace (not only one data stream), you can use [namespace index templates](/reference/fleet/data-streams-namespace-custom.md) instead of duplicating the integration index template. 1. Go to the **Index Management** page using the navigation menu or the [global search field](/explore-analyze/find-and-organize/find-apps-and-objects.md), and open the **Index Templates** tab. 2. Find the index template you want to clone. The index template will have the `` and `` in its name, but not the ``. In this case, it’s `metrics-system.network`. diff --git a/reference/fleet/data-streams-template-conflicts.md b/reference/fleet/data-streams-template-conflicts.md new file mode 100644 index 0000000000..53e59b5e54 --- /dev/null +++ b/reference/fleet/data-streams-template-conflicts.md @@ -0,0 +1,96 @@ +--- +navigation_title: Resolve overlapping index templates +description: Resolve overlapping index templates when you enable namespace index templates for a Fleet integration. +applies_to: + stack: ga 9.5+ + serverless: ga +products: + - id: fleet + - id: elastic-agent +--- + +# Resolve overlapping index templates for namespace customization [data-streams-template-conflicts] + +Learn how to interpret and resolve warnings about overlapping index templates when you enable [namespace index templates](/reference/fleet/data-streams-namespace-custom.md) for a {{fleet}} integration. This page helps you decide whether to keep your existing customizations, switch to {{fleet}}-managed namespace index templates, or combine both approaches. + +## When overlapping templates appear [data-streams-template-conflicts-when] + +If you previously duplicated an integration's base data stream index template (priority `200`), narrowed the index pattern to one namespace, and raised the copy's priority (often to `250` or higher), that copy can overlap with a {{fleet}}-managed namespace index template. For example, you might have copied `logs-nginx.error` into `logs-nginx.error-production-custom` with pattern `logs-nginx.error-production*` and priority `280`. + +When you enable namespace index templates for that same namespace, {{fleet}} tries to create a managed template such as `logs-nginx.error@namespace.production` at priority `250` for the same index pattern. {{es}} applies only one winning index template per index creation, so those templates overlap. The result depends on the existing template's priority, as described in [How {{fleet}} detects and reports overlaps](#data-streams-template-conflicts-detect). + +The same can happen with any pre-existing index template whose index patterns match the namespace's data streams, not only copies with a custom suffix in the name. + +## How {{fleet}} detects and reports overlaps [data-streams-template-conflicts-detect] + +When you enable namespace index templates for a namespace on an integration, {{fleet}} runs a preflight check for each `(data stream, namespace)` pair. The check uses the {{es}} simulate index API (`POST _index_template/_simulate_index/`) to see which existing template would win for that data stream, then compares priorities with the planned namespace index template. + +::::{note} +{{fleet}} sets the namespace index template priority to the base template priority plus 50: `250` for most integrations (base `200`), or `200` for integrations with `dataset_is_prefix: true` data streams (base `150`). The examples on this page use `250`; if your integration uses `dataset_is_prefix` data streams, read `200` instead. A user with the `manage_index_templates` privilege can change this priority with the {{es}} index template API, but {{fleet}} restores the default whenever the integration's templates are reinstalled or resynchronized. +:::: + +{{fleet}} reports overlaps in the UI before changes are applied, and in API responses as warnings. Each overlap falls into one of these cases: + +| Situation | What happens | +| --- | --- | +| Existing template priority is higher than `250` | {{fleet}} creates the namespace index template, but {{es}} applies the existing template for new data streams. The namespace index template's `composed_of` chain (including `@custom`) is not used. | +| Existing template priority is exactly `250` | {{fleet}} cannot create the namespace index template. The existing template continues to govern new data streams for that pair until you change priorities and enable namespace index templates again. | +| Existing template priority is lower than `250` | {{fleet}} creates the namespace index template, and {{es}} applies it for new data streams. The existing template no longer affects newly created indices in that namespace. | + +::::{note} +Index templates are applied when backing indices are created. Existing indices are not rewritten when you enable or turn off namespace index templates. +:::: + +{{es}} does not merge overlapping index templates. The highest-priority matching template wins as a whole. Component templates are merged only within that winning template's `composed_of` list. + +## Choose a resolution path [data-streams-template-conflicts-resolve] + +Pick the path that matches your goal. + +### Use {{fleet}}-managed namespace index templates [data-streams-template-conflicts-use-fleet] + +Use this path when you want `@custom` and related {{fleet}} namespace features to apply. + +1. Identify overlapping templates in **Index Management** → **Index Templates**. Look for templates whose index patterns match the same data streams as the planned `@namespace.` templates. +2. If any overlapping template has priority `250`, lower that priority (for example to `200` or less) or remove the template before enabling. Same-priority templates block creation of the {{fleet}}-managed namespace index template. + + ::::{include} _snippets/change-index-template-priority.md + :::: +3. Enable namespace index templates for the namespace. For UI and API steps, refer to [Customize data streams with namespace index templates](/reference/fleet/data-streams-namespace-custom.md). +4. Confirm the managed templates exist (for example, search index templates for `@namespace.`). +5. Create or update the `@custom` component template with the settings you still need. Copy useful settings from your older custom index templates into `@custom` or into the data stream `@custom` templates when they should stay data-stream-specific. +6. Optionally delete or further lower the priority of unused custom index templates that no longer win, if you don't need them as a fallback. +7. Roll over each affected data stream so new backing indices use the managed templates. + +::::{tip} +If you enabled namespace index templates while a priority-`250` overlap blocked some templates, turn them off for that namespace, adjust those priorities, then enable them again so {{fleet}} can create the missing namespace index templates. +:::: + +### Keep your existing custom index templates [data-streams-template-conflicts-keep-custom] + +Use this path when your custom index templates already do what you need. + +1. Cancel enabling if the warning appears in the UI, or leave the namespace out of **Namespaces with dedicated index templates**. +2. If you already enabled namespace index templates for the namespace, remove it from **Namespaces with dedicated index templates** on the integration's **Settings** tab, or turn off **Use dedicated index templates for this namespace** in the integration policy editor. {{fleet}} deletes the managed namespace index templates; your existing templates remain. +3. Continue managing customization through your existing index templates and component templates. + +### Combine both approaches [data-streams-template-conflicts-mixed] + +Use this path only when you understand which template should win for each data stream. + +1. Review every overlapping template's priority against the namespace index template priority (`250` for most integrations; refer to the preceding note). +2. For data streams where your custom template must win, set its priority higher than the namespace index template priority. For data streams where the {{fleet}}-managed namespace index template must win, set your custom template's priority lower (but not equal, which blocks creation), or remove the template. + + ::::{include} _snippets/change-index-template-priority.md + :::: +3. Enable namespace index templates and read the warning list to confirm the expected winners. +4. Verify a sample of new backing indices (settings, mappings, and lifecycle) after rollover. + +Leaving overlapping templates at priority `250` or higher means some or all namespace index template customization might be unused or incomplete, even after you enable the feature. + +## Related pages [data-streams-template-conflicts-related] + +* [Customize data streams with namespace index templates](/reference/fleet/data-streams-namespace-custom.md) +* [Edit the {{es}} index template](/reference/fleet/data-streams.md#data-streams-index-templates-edit) +* [Best practices for copying an integration index template](/reference/fleet/integrations-assets-best-practices.md#assets-restrictions-cloning-index-template) +* [Tutorials: Customize data retention policies](/reference/fleet/data-streams-ilm-tutorial.md) diff --git a/reference/fleet/data-streams.md b/reference/fleet/data-streams.md index b44a3fab84..27b07ee0f4 100644 --- a/reference/fleet/data-streams.md +++ b/reference/fleet/data-streams.md @@ -92,12 +92,15 @@ Custom index mappings may conflict with the mappings defined by the integration :::: -When you install an integration, {{fleet}} creates two default `@custom` component templates: +When you install an integration, {{fleet}} creates default `@custom` component templates that you can edit: -* A `@custom` component template allowing customization across all documents of a given data stream type, named following the pattern: `@custom`. -* A `@custom` component template for each data stream, named following the pattern: `@custom`. +* A type-level `@custom` component template for all documents of a given data stream type, named `@custom` (for example, `logs@custom`). +* A data stream-level `@custom` component template for each data stream, named `@custom` (for example, `logs-system.application@custom`). +* {applies_to}`stack: ga 9.1+` A package-level `@custom` component template for the integration, named `@custom` (for example, `system@custom`). -The `@custom` component template specific to a data stream has higher precedence over the data stream type `@custom` component template. +{applies_to}`stack: ga 9.5+` {applies_to}`serverless: ga` You can also create a `@custom` component template and enable [namespace index templates](/reference/fleet/data-streams-namespace-custom.md) so {{fleet}} applies that template to every data stream in the namespace. + +In the component template chain, later entries take precedence. Among `@custom` templates, precedence from highest to lowest is: data stream, namespace (when present), package (when present), then data stream type. You can edit a `@custom` component template to customize your {{es}} indices: diff --git a/reference/fleet/integrations-assets-best-practices.md b/reference/fleet/integrations-assets-best-practices.md index 5c9ee2e16f..d020cb3bdd 100644 --- a/reference/fleet/integrations-assets-best-practices.md +++ b/reference/fleet/integrations-assets-best-practices.md @@ -89,3 +89,5 @@ Additionally, cloning index templates to add or inject additional component temp If you want to change the ILM Policy, the number of shards, or other settings for the data streams of one or more integrations, but the changes do not need to be specific to a given namespace, it’s highly recommended to use the `package@custom` component templates, as described in [Scenario 1](/reference/fleet/data-streams-scenario1.md) and [Scenario 2](/reference/fleet/data-streams-scenario2.md) of the Customize data retention policies tutorial, so as to avoid the problems mentioned above. If you want to change these settings for the data streams in one or more integrations and the changes **need to be namespace specific**, then you can do so following the steps in [Scenario 3](/reference/fleet/data-streams-scenario3.md) of the Customize data retention policies tutorial, but be aware of the restrictions mentioned above. + +{applies_to}`stack: ga 9.5+` As an alternative that does not duplicate the integration index template, you can use [namespace index templates](/reference/fleet/data-streams-namespace-custom.md) for every data stream in the namespace. diff --git a/reference/fleet/toc.yml b/reference/fleet/toc.yml index ca15aa78cf..39f329b42f 100644 --- a/reference/fleet/toc.yml +++ b/reference/fleet/toc.yml @@ -159,6 +159,8 @@ toc: - file: data-streams-scenario3.md - file: data-streams-scenario4.md - file: data-streams-pipeline-tutorial.md + - file: data-streams-namespace-custom.md + - file: data-streams-template-conflicts.md - file: data-streams-advanced-features.md - file: alerting-rule-templates.md - file: agent-command-reference.md