diff --git a/docs/images/tutorial-ilm-custom-policy.png b/docs/images/tutorial-ilm-custom-policy.png new file mode 100644 index 00000000000000..03b67829f605ca Binary files /dev/null and b/docs/images/tutorial-ilm-custom-policy.png differ diff --git a/docs/images/tutorial-ilm-delete-phase-creation.png b/docs/images/tutorial-ilm-delete-phase-creation.png new file mode 100644 index 00000000000000..91a55733c284e9 Binary files /dev/null and b/docs/images/tutorial-ilm-delete-phase-creation.png differ diff --git a/docs/images/tutorial-ilm-delete-rollover.png b/docs/images/tutorial-ilm-delete-rollover.png new file mode 100644 index 00000000000000..ba021ecc2ac5c5 Binary files /dev/null and b/docs/images/tutorial-ilm-delete-rollover.png differ diff --git a/docs/images/tutorial-ilm-hotphaserollover-default.png b/docs/images/tutorial-ilm-hotphaserollover-default.png new file mode 100644 index 00000000000000..a9088c63d885dd Binary files /dev/null and b/docs/images/tutorial-ilm-hotphaserollover-default.png differ diff --git a/docs/images/tutorial-ilm-modify-default-warm-phase-rollover.png b/docs/images/tutorial-ilm-modify-default-warm-phase-rollover.png new file mode 100644 index 00000000000000..c6f1e9b40e9777 Binary files /dev/null and b/docs/images/tutorial-ilm-modify-default-warm-phase-rollover.png differ diff --git a/docs/management/index-lifecycle-policies/example-index-lifecycle-policy.asciidoc b/docs/management/index-lifecycle-policies/example-index-lifecycle-policy.asciidoc index f68708f1b6394b..28766015e2766b 100644 --- a/docs/management/index-lifecycle-policies/example-index-lifecycle-policy.asciidoc +++ b/docs/management/index-lifecycle-policies/example-index-lifecycle-policy.asciidoc @@ -1,23 +1,179 @@ [role="xpack"] + [[example-using-index-lifecycle-policy]] -=== Example of using an index lifecycle policy +=== Tutorial: Use {ilm-init} to manage {filebeat} time-based indices + +With {ilm} ({ilm-init}), you can create policies that perform actions automatically +on indices as they age and grow. {ilm-init} policies help you to manage +performance, resilience, and retention of your data during its lifecycle. This tutorial shows +you how to use {kib}’s *Index Lifecycle Policies* to modify and create {ilm-init} +policies. You can learn more about all of the actions, benefits, and lifecycle +phases in the {ref}/index-lifecycle-management.html[{ilm-init} overview]. + + +[discrete] +[[example-using-index-lifecycle-policy-scenario]] +==== Scenario + +You’re tasked with sending syslog files to an {es} cluster. This +log data has the following data retention guidelines: + +* Keep logs on hot data nodes for 30 days +* Roll over to a new index if the size reaches 50GB +* After 30 days: +** Move the logs to warm data nodes +** Set {ref}/glossary.html#glossary-replica-shard[replica shards] to 1 +** {ref}/indices-forcemerge.html[Force merge] multiple index segments to free up the space used by deleted documents +* Delete logs after 90 days + + +[discrete] +[[example-using-index-lifecycle-policy-prerequisites]] +==== Prerequisites + +To complete this tutorial, you'll need: + +* An {es} cluster with hot and warm nodes configured for shard allocation +awareness. If you’re using {cloud}/ec-getting-started-templates-hot-warm.html[{ess}], +choose the hot-warm architecture deployment template. + ++ +For a self-managed cluster, add node attributes as described for {ref}/shard-allocation-filtering.html[shard allocation filtering] +to label data nodes as hot or warm. This step is required to migrate shards between +nodes configured with specific hardware for the hot or warm phases. ++ +For example, you can set this in your `elasticsearch.yml` for each data node: ++ +[source,yaml] +-------------------------------------------------------------------------------- +node.attr.data: "warm" +-------------------------------------------------------------------------------- + +* A server with {filebeat} installed and configured to send logs to the `elasticsearch` +output as described in {filebeat-ref}/filebeat-getting-started.html[Getting Started with {filebeat}]. + +[discrete] +[[example-using-index-lifecycle-policy-view-fb-ilm-policy]] +==== View the {filebeat} {ilm-init} policy + +{filebeat} includes a default {ilm-init} policy that enables rollover. {ilm-init} +is enabled automatically if you’re using the default `filebeat.yml` and index template. + +To view the default policy in {kib}, go to *Management > Index Lifecycle Policies*, +search for _filebeat_, and choose the _filebeat-version_ policy. + +This policy initiates the rollover action when the index size reaches 50GB or +becomes 30 days old. + +[role="screenshot"] +image::images/tutorial-ilm-hotphaserollover-default.png["Default policy"] + + +[float] +==== Modify the policy + +The default policy is enough to prevent the creation of many tiny daily indices. +You can modify the policy to meet more complex requirements. + +. Activate the warm phase. + ++ +. Set either of the following options to control when the index moves to the warm phase: + +** Provide a value for *Timing for warm phase*. Setting this to *15* keeps the +indices on hot nodes for a range of 15-45 days, depending on when the initial +rollover occurred. + +** Enable *Move to warm phase on rollover*. The index might move to the warm phase +more quickly than intended if it reaches the *Maximum index size* before the +the *Maximum age*. + +. In the *Select a node attribute to control shard allocation* dropdown, select +*data:warm(2)* to migrate shards to warm data nodes. + +. Change *Number of replicas* to *1*. + +. Enable *Force merge data* and set *Number of segments* to *1*. ++ +NOTE: When rollover is enabled in the hot phase, action timing in the other phases +is based on the rollover date. + ++ +[role="screenshot"] +image::images/tutorial-ilm-modify-default-warm-phase-rollover.png["Modify to add warm phase"] + +. Activate the delete phase and set *Timing for delete phase* to *90* days. ++ +[role="screenshot"] +image::images/tutorial-ilm-delete-rollover.png["Add a delete phase"] + +[float] +==== Create a custom policy + +If meeting a specific retention time period is most important, you can create a +custom policy. For this option, you will use {filebeat} daily indices without +rollover. + +. Create a custom policy in {kib}, go to *Management > Index Lifecycle Policies > +Create Policy*. + +. Activate the warm phase and configure it as follows: ++ +|=== +|*Setting* |*Value* + +|Timing for warm phase +|30 days from index creation + +|Node attribute +|`data:warm` + +|Number of replicas +|1 + +|Force merge data +|enable + +|Number of segments +|1 +|=== + ++ +[role="screenshot"] +image::images/tutorial-ilm-custom-policy.png["Modify the custom policy to add a warm phase"] + -A common use case for managing index lifecycle policies is when you’re using -{beats-ref}/beats-reference.html[Beats] to continually send time-series data, -such as metrics and log data, to {es}. When you create the Beats packages, an -index template is installed. The template includes a default policy to apply -when new indices are created. ++ +. Activate the delete phase and set the timing. ++ +|=== +|*Setting* |*Value* +|Timing for delete phase +|90 +|=== -You can edit the policy in {kib}'s *Index Lifecycle Policies*. For example, you might: ++ +[role="screenshot"] +image::images/tutorial-ilm-delete-phase-creation.png["Delete phase"] -* Rollover the index when it reaches 50 GB in size or is 30 days old. These -settings are the default for the Beats lifecycle policy. This avoids -having 1000s of tiny indices. When a rollover occurs, a new “hot” index is -created and added to the index alias. +. Configure the index to use the new policy in *{kib} > Management > Index Lifecycle +Policies* -* Move the index into the warm phase, shrink the index down to a single shard, -and force merge to a single segment. +.. Find your {ilm-init} policy. +.. Click the *Actions* link next to your policy name. +.. Choose *Add policy to index template*. +.. Select your {filebeat} index template name from the *Index template* list. For example, `filebeat-7.5.x`. +.. Click *Add Policy* to save the changes. -* After 60 days, move the index into the cold phase and onto less expensive hardware. ++ +NOTE: If you initially used the default {filebeat} {ilm-init} policy, you will +see a notice that the template already has a policy associated with it. Confirm +that you want to overwrite that configuration. -* Delete the index after 90 days. ++ ++ +TIP: When you change the policy associated with the index template, the active +index will continue to use the policy it was associated with at index creation +unless you manually update it. The next new index will use the updated policy. +For more reasons that your {ilm-init} policy changes might be delayed, see +{ref}/update-lifecycle-policy.html#update-lifecycle-policy[Update Lifecycle Policy].