Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DOCS] Add ILM tutorial #59502

Merged
merged 8 commits into from
Apr 13, 2020
Merged

[DOCS] Add ILM tutorial #59502

merged 8 commits into from
Apr 13, 2020

Conversation

mellieA
Copy link

@mellieA mellieA commented Mar 6, 2020

Summary

Added a walkthrough using ILM with filebeat indices

Closes #29363
[skip-ci]

Preview the tutorial

@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-docs (Team:Docs)

@mellieA
Copy link
Author

mellieA commented Mar 6, 2020

This is still a work in progress, I know I need to check on at least the following things:

  • Does a custom policy requires more Filebeat changes
  • Add a step to associate a custom policy with the index template
  • Change the custom policy screenshot

=== Tutorial: Use ILM to manage Filebeat time-based indices

With index lifecycle management (ILM), you can create policies to define actions
that will be performed automatically against indices as they age. In this tutorial,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

that are automatically performed

node.attr.data: "warm"
--------------------------------------------------------------------------------

* A server with Filebeat installed and configured to send logs to the `elasticsearch`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add ending period

[float]
==== View the filebeat ILM policy

Filebeat includes a default ILM policy which enables rollover. Index lifecycle
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

which > that

The default policy is enough to prevent the creation of many tiny daily indices.
You can modify the policy to meet more complex requirements.

* Activate the warm phase.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Change bullet points to numbered steps.

  1. Activate the warm phase.
  2. Set either of the following:
    bullet list
  3. In Select a node attribute ...

* Activate the warm phase.

+
You have a choice to make, you can’t meet the requirements exactly with rollover
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Make this step 2 instead of a substep:

Set either of the following:

more quickly than intended if it reaches the *Maximum index size* condition before
*Maximum age*.

** Provide a value for *Timing for warm phase*. Setting this to *15* will keep the
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd put this first because it is the default.

indices on hot nodes for a range of 15-45 days, depending on when the initial
rollover occurred.

* *Select a node attribute to control shard allocation*. For this example,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In Select a node attribute to control shard allocation, set data:warm(2) to migrate shards to warm data nodes.

[role="screenshot"]
image::images/tutorial-ilm-modify-default-warm-phase-rollover.png["Modify to add warm phase"]

* Activate the delete phase.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Make this one step.

Activate the delete phase and set the Timing to 60 days.

To create a custom policy in {kib}, go to *Management > Index Lifecycle Policies > Create Policy*.


* *Warm phase*
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Combine.

  1. Activate the warm phase and configure it as follows.

* After 60 days, move the index into the cold phase and onto less expensive hardware.
* *Delete phase*
+
Activate the delete phase and set the timing.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. Activate the delete phase and set the timing.

@mellieA
Copy link
Author

mellieA commented Mar 16, 2020

@gchaps @DanRoscigno I made the changes recommended above and pushed a new commit.

I'm still waiting to see how my custom policy behaves when it rolls over in 3 days.

@mellieA
Copy link
Author

mellieA commented Apr 8, 2020

@gchaps @DanRoscigno I verified that ILM behaves the way I expected with this new policy. I think this is ready for another review!

to label data nodes as hot or warm. This step is required to migrate shards between
nodes using shard allocation awareness.
+
For example, you can set this in `elasticsearch.yml` for each data node:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
For example, you can set this in `elasticsearch.yml` for each data node:
For example, you can set this in your `elasticsearch.yml` for each data node:

can create a custom policy. For this option, you will use Filebeat daily indices
without rollover.

To create a custom policy in {kib}, go to *Management > Index Lifecycle Policies >
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this be step 1?


* Move the index into the warm phase, shrink the index down to a single shard,
and force merge to a single segment.
. Configure the index to use the new policy in *{kib} > Management > Index Lifecycle
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is hard to parse. Can you make it a series of substeps?
3. Configure the index to use the new policy.
a. In Index Lifecycle Policies, select your policy.
b....
c....

@gchaps gchaps requested a review from jrodewig April 8, 2020 22:05
[[example-using-index-lifecycle-policy]]
=== Example of using an index lifecycle policy
=== Tutorial: Use ILM to manage Filebeat time-based indices
Copy link
Contributor

@jrodewig jrodewig Apr 9, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We maintain a list of attributes here: https://github.com/elastic/docs/blob/master/shared/attributes.asciidoc

Try to use those when possible. They ensure we can make docs-wide term changes easily.

Suggested change
=== Tutorial: Use ILM to manage Filebeat time-based indices
=== Tutorial: Use {ilm-init} to manage {filebeat} time-based indices

=== Example of using an index lifecycle policy
=== Tutorial: Use ILM to manage Filebeat time-based indices

With index lifecycle management (ILM), you can create policies to define actions
Copy link
Contributor

@jrodewig jrodewig Apr 9, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
With index lifecycle management (ILM), you can create policies to define actions
With {ilm} ({ilm-init}), you can create policies to define actions

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm replacing an existing page, so I was leaving the old anchor in case it was referenced elsewhere. Do you still suggest to change it?

Comment on lines 6 to 8
With index lifecycle management (ILM), you can create policies to define actions
that are automatically performed against indices as they age. In this tutorial,
you will use {kib}’s *Index Lifecycle Policies* to modify and create ILM policies.
Copy link
Contributor

@jrodewig jrodewig Apr 9, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With index lifecycle management (ILM), you can create policies to define actions that are automatically performed against indices as they age.

This implies that ILM only works based on age, but we can also take actions based on index size or number of docs. More importantly, this doesn't really address the why of ILM. It helps you automatically manage your indices for performance and retention. You may want to consider reusing a sentence from https://www.elastic.co/guide/en/elasticsearch/reference/master/overview-index-lifecycle-management.html.

In this tutorial, you will use Kibana’s Index Lifecycle Policies to modify and create ILM policies.

Nit: I'd reword to avoid future tense if possible. Something like:

This tutorial shows you how to use use {kib}'s *Index Lifecycle Policies* to
create and change {ilm-init} policies.

The bolded Index Lifecycle Policies seemed unnecessary to me, but maybe that's just a Kibana docs style.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, it's a Kibana docs style.

Comment on lines 11 to 12
[float]
==== Scenario
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should use [discrete] instead of [float] when possible. I'd also add an explicit anchor here.

Suggested change
[float]
==== Scenario
[discrete]
[[example-using-index-lifecycle-policy-scenario]]
==== Scenario

Comment on lines 14 to 15
You’ve been tasked with sending syslog files to an {es} cluster. This
project has the following data retention guidelines for log data:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
You’ve been tasked with sending syslog files to an {es} cluster. This
project has the following data retention guidelines for log data:
You’re tasked with sending syslog files to an {es} cluster. This
log data has the following data retention guidelines:

You’ve been tasked with sending syslog files to an {es} cluster. This
project has the following data retention guidelines for log data:

* Keep logs on hot data nodes for 30 days
Copy link
Contributor

@jrodewig jrodewig Apr 9, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the first mention of hot-warm architecture.

If readers should be familiar with those concepts, maybe add a link to a conceptual doc in the introduction?

project has the following data retention guidelines for log data:

* Keep logs on hot data nodes for 30 days
* Rollover to a new index if the size reaches 50GB
Copy link
Contributor

@jrodewig jrodewig Apr 9, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"Roll over" should be two words when used as a verb. One word as a noun.

Suggested change
* Rollover to a new index if the size reaches 50GB
* Roll over any index containing log data to a new index when it reaches 50GB

Comment on lines 21 to 22
** Set replicas to 1
** Force merge indices to a single segment
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

** Set replicas to 1
** Force merge indices to a single segment

I think these two items need more context/explanation. I imagine many users may not be familiar with the concept of replicas, force merges, or segments. At the least, provide a link.

@mellieA mellieA deleted the ilm-beats-tutorial branch April 13, 2020 22:32
mellieA pushed a commit to mellieA/kibana that referenced this pull request Apr 13, 2020
* Add tutorial for ILM with filebeat

* Change screenshots and add additional steps

* Update screenshots, add numbered steps, and other minor edits

* Incorporate feedback: update links, formatting, and minor edits

* Move tip inline with list

* Apply suggestions from code review

Co-Authored-By: James Rodewig <james.rodewig@elastic.co>

* Move TIP inline . . . again

* Put TIP inline

Co-authored-by: James Rodewig <james.rodewig@elastic.co>
mellieA pushed a commit to mellieA/kibana that referenced this pull request Apr 13, 2020
* Add tutorial for ILM with filebeat

* Change screenshots and add additional steps

* Update screenshots, add numbered steps, and other minor edits

* Incorporate feedback: update links, formatting, and minor edits

* Move tip inline with list

* Apply suggestions from code review

Co-Authored-By: James Rodewig <james.rodewig@elastic.co>

* Move TIP inline . . . again

* Put TIP inline

Co-authored-by: James Rodewig <james.rodewig@elastic.co>
mellieA pushed a commit to mellieA/kibana that referenced this pull request Apr 13, 2020
* Add tutorial for ILM with filebeat

* Change screenshots and add additional steps

* Update screenshots, add numbered steps, and other minor edits

* Incorporate feedback: update links, formatting, and minor edits

* Move tip inline with list

* Apply suggestions from code review

Co-Authored-By: James Rodewig <james.rodewig@elastic.co>

* Move TIP inline . . . again

* Put TIP inline

Co-authored-by: James Rodewig <james.rodewig@elastic.co>
mellieA pushed a commit to mellieA/kibana that referenced this pull request Apr 13, 2020
* Add tutorial for ILM with filebeat

* Change screenshots and add additional steps

* Update screenshots, add numbered steps, and other minor edits

* Incorporate feedback: update links, formatting, and minor edits

* Move tip inline with list

* Apply suggestions from code review

Co-Authored-By: James Rodewig <james.rodewig@elastic.co>

* Move TIP inline . . . again

* Put TIP inline

Co-authored-by: James Rodewig <james.rodewig@elastic.co>
mellieA pushed a commit to mellieA/kibana that referenced this pull request Apr 13, 2020
* Add tutorial for ILM with filebeat

* Change screenshots and add additional steps

* Update screenshots, add numbered steps, and other minor edits

* Incorporate feedback: update links, formatting, and minor edits

* Move tip inline with list

* Apply suggestions from code review

Co-Authored-By: James Rodewig <james.rodewig@elastic.co>

* Move TIP inline . . . again

* Put TIP inline

Co-authored-by: James Rodewig <james.rodewig@elastic.co>
mellieA pushed a commit to mellieA/kibana that referenced this pull request Apr 13, 2020
* Add tutorial for ILM with filebeat

* Change screenshots and add additional steps

* Update screenshots, add numbered steps, and other minor edits

* Incorporate feedback: update links, formatting, and minor edits

* Move tip inline with list

* Apply suggestions from code review

Co-Authored-By: James Rodewig <james.rodewig@elastic.co>

* Move TIP inline . . . again

* Put TIP inline

Co-authored-by: James Rodewig <james.rodewig@elastic.co>
mellieA pushed a commit to mellieA/kibana that referenced this pull request Apr 13, 2020
* Add tutorial for ILM with filebeat

* Change screenshots and add additional steps

* Update screenshots, add numbered steps, and other minor edits

* Incorporate feedback: update links, formatting, and minor edits

* Move tip inline with list

* Apply suggestions from code review

Co-Authored-By: James Rodewig <james.rodewig@elastic.co>

* Move TIP inline . . . again

* Put TIP inline

Co-authored-by: James Rodewig <james.rodewig@elastic.co>
mellieA pushed a commit to mellieA/kibana that referenced this pull request Apr 13, 2020
* Add tutorial for ILM with filebeat

* Change screenshots and add additional steps

* Update screenshots, add numbered steps, and other minor edits

* Incorporate feedback: update links, formatting, and minor edits

* Move tip inline with list

* Apply suggestions from code review

Co-Authored-By: James Rodewig <james.rodewig@elastic.co>

* Move TIP inline . . . again

* Put TIP inline

Co-authored-by: James Rodewig <james.rodewig@elastic.co>
mellieA pushed a commit that referenced this pull request Apr 13, 2020
* Add tutorial for ILM with filebeat

* Change screenshots and add additional steps

* Update screenshots, add numbered steps, and other minor edits

* Incorporate feedback: update links, formatting, and minor edits

* Move tip inline with list

* Apply suggestions from code review

Co-Authored-By: James Rodewig <james.rodewig@elastic.co>

* Move TIP inline . . . again

* Put TIP inline

Co-authored-by: James Rodewig <james.rodewig@elastic.co>

Co-authored-by: James Rodewig <james.rodewig@elastic.co>
mellieA pushed a commit that referenced this pull request Apr 13, 2020
* Add tutorial for ILM with filebeat

* Change screenshots and add additional steps

* Update screenshots, add numbered steps, and other minor edits

* Incorporate feedback: update links, formatting, and minor edits

* Move tip inline with list

* Apply suggestions from code review

Co-Authored-By: James Rodewig <james.rodewig@elastic.co>

* Move TIP inline . . . again

* Put TIP inline

Co-authored-by: James Rodewig <james.rodewig@elastic.co>

Co-authored-by: James Rodewig <james.rodewig@elastic.co>
mellieA pushed a commit that referenced this pull request Apr 13, 2020
* Add tutorial for ILM with filebeat

* Change screenshots and add additional steps

* Update screenshots, add numbered steps, and other minor edits

* Incorporate feedback: update links, formatting, and minor edits

* Move tip inline with list

* Apply suggestions from code review

Co-Authored-By: James Rodewig <james.rodewig@elastic.co>

* Move TIP inline . . . again

* Put TIP inline

Co-authored-by: James Rodewig <james.rodewig@elastic.co>

Co-authored-by: James Rodewig <james.rodewig@elastic.co>
gmmorris added a commit to gmmorris/kibana that referenced this pull request Apr 14, 2020
* master: (132 commits)
  document code splitting for client code (elastic#62593)
  Escape single quotes surrounded by double quotes (elastic#63229)
  [Endpoint] Update cli mapping to match endpoint package (elastic#63372)
  update in-app links to metricbeat configuration docs (elastic#63295)
  investigation notes field (documentation / metadata) (elastic#63386)
  [Maps] fix bug where toggling Scaling type does not re-fetch data (elastic#63326)
  [Alerting] set correct parameter for unauthented email action (elastic#63086)
  [Telemetry] force staging urls in tests (elastic#63356)
  Migrate legacy maps service to NP & update refs (elastic#60942)
  Fix task manager query to return tasks to retry (elastic#63360)
  [Endpoint] Policy list support for URL pagination state (elastic#63291)
  [Canvas] Migrate saved object mappings and migrations to Kibana Platform (elastic#58891)
  [DOCS] Add ILM tutorial (elastic#59502)
  [Maps] Add SOURCE_TYPES enumeration (elastic#62975)
  [Maps] update geospatial filters to use geo_shape query for geo_point fields (elastic#62966)
  Move away from npStart for embeddables in canvas (elastic#62680)
  Use MapInput type from Maps plugin (elastic#61539)
  Update to pagination for workpad and templates (elastic#62050)
  [SIEM] Fix AlertsTable id (elastic#63368)
  Consistent terminology around cypress test data (elastic#63279)
  ...
gmmorris added a commit to gmmorris/kibana that referenced this pull request Apr 14, 2020
* master:
  document code splitting for client code (elastic#62593)
  Escape single quotes surrounded by double quotes (elastic#63229)
  [Endpoint] Update cli mapping to match endpoint package (elastic#63372)
  update in-app links to metricbeat configuration docs (elastic#63295)
  investigation notes field (documentation / metadata) (elastic#63386)
  [Maps] fix bug where toggling Scaling type does not re-fetch data (elastic#63326)
  [Alerting] set correct parameter for unauthented email action (elastic#63086)
  [Telemetry] force staging urls in tests (elastic#63356)
  Migrate legacy maps service to NP & update refs (elastic#60942)
  Fix task manager query to return tasks to retry (elastic#63360)
  [Endpoint] Policy list support for URL pagination state (elastic#63291)
  [Canvas] Migrate saved object mappings and migrations to Kibana Platform (elastic#58891)
  [DOCS] Add ILM tutorial (elastic#59502)
  [Maps] Add SOURCE_TYPES enumeration (elastic#62975)
  [Maps] update geospatial filters to use geo_shape query for geo_point fields (elastic#62966)
  Move away from npStart for embeddables in canvas (elastic#62680)
mellieA pushed a commit that referenced this pull request Apr 15, 2020
* [DOCS] Add ILM tutorial (#59502)

* Add tutorial for ILM with filebeat

* Change screenshots and add additional steps

* Update screenshots, add numbered steps, and other minor edits

* Incorporate feedback: update links, formatting, and minor edits

* Move tip inline with list

* Apply suggestions from code review

Co-Authored-By: James Rodewig <james.rodewig@elastic.co>

* Move TIP inline . . . again

* Put TIP inline

Co-authored-by: James Rodewig <james.rodewig@elastic.co>

* Update ILM overview link for 7.0

Co-authored-by: James Rodewig <james.rodewig@elastic.co>
mellieA pushed a commit that referenced this pull request Apr 15, 2020
* [DOCS] Add ILM tutorial (#59502)

* Add tutorial for ILM with filebeat

* Change screenshots and add additional steps

* Update screenshots, add numbered steps, and other minor edits

* Incorporate feedback: update links, formatting, and minor edits

* Move tip inline with list

* Apply suggestions from code review

Co-Authored-By: James Rodewig <james.rodewig@elastic.co>

* Move TIP inline . . . again

* Put TIP inline

Co-authored-by: James Rodewig <james.rodewig@elastic.co>

* Update ILM overview link for 7.1

Co-authored-by: James Rodewig <james.rodewig@elastic.co>
mellieA pushed a commit that referenced this pull request Apr 15, 2020
* [DOCS] Add ILM tutorial (#59502)

* Add tutorial for ILM with filebeat

* Change screenshots and add additional steps

* Update screenshots, add numbered steps, and other minor edits

* Incorporate feedback: update links, formatting, and minor edits

* Move tip inline with list

* Apply suggestions from code review

Co-Authored-By: James Rodewig <james.rodewig@elastic.co>

* Move TIP inline . . . again

* Put TIP inline

Co-authored-by: James Rodewig <james.rodewig@elastic.co>

* Update ILM overview link for 7.2

Co-authored-by: James Rodewig <james.rodewig@elastic.co>
mellieA pushed a commit that referenced this pull request Apr 15, 2020
* [DOCS] Add ILM tutorial (#59502)

* Add tutorial for ILM with filebeat

* Change screenshots and add additional steps

* Update screenshots, add numbered steps, and other minor edits

* Incorporate feedback: update links, formatting, and minor edits

* Move tip inline with list

* Apply suggestions from code review

Co-Authored-By: James Rodewig <james.rodewig@elastic.co>

* Move TIP inline . . . again

* Put TIP inline

Co-authored-by: James Rodewig <james.rodewig@elastic.co>

* Update ILM overview link for 7.3

Co-authored-by: James Rodewig <james.rodewig@elastic.co>
mellieA pushed a commit that referenced this pull request Apr 15, 2020
* [DOCS] Add ILM tutorial (#59502)

* Add tutorial for ILM with filebeat

* Change screenshots and add additional steps

* Update screenshots, add numbered steps, and other minor edits

* Incorporate feedback: update links, formatting, and minor edits

* Move tip inline with list

* Apply suggestions from code review

Co-Authored-By: James Rodewig <james.rodewig@elastic.co>

* Move TIP inline . . . again

* Put TIP inline

Co-authored-by: James Rodewig <james.rodewig@elastic.co>

* Update ILM link for 7.5

Co-authored-by: James Rodewig <james.rodewig@elastic.co>
mellieA pushed a commit that referenced this pull request Apr 15, 2020
* [DOCS] Add ILM tutorial (#59502)

* Add tutorial for ILM with filebeat

* Change screenshots and add additional steps

* Update screenshots, add numbered steps, and other minor edits

* Incorporate feedback: update links, formatting, and minor edits

* Move tip inline with list

* Apply suggestions from code review

Co-Authored-By: James Rodewig <james.rodewig@elastic.co>

* Move TIP inline . . . again

* Put TIP inline

Co-authored-by: James Rodewig <james.rodewig@elastic.co>

* Update ILM overview link for 7.5

Co-authored-by: James Rodewig <james.rodewig@elastic.co>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
wayneseymour pushed a commit that referenced this pull request Apr 15, 2020
* Add tutorial for ILM with filebeat

* Change screenshots and add additional steps

* Update screenshots, add numbered steps, and other minor edits

* Incorporate feedback: update links, formatting, and minor edits

* Move tip inline with list

* Apply suggestions from code review

Co-Authored-By: James Rodewig <james.rodewig@elastic.co>

* Move TIP inline . . . again

* Put TIP inline

Co-authored-by: James Rodewig <james.rodewig@elastic.co>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Docs improvement for using ILM with Filebeat and Logstash
5 participants