[alerting] add user facing doc on event log ILM policy #92736

pmuellr · 2021-02-24T23:07:15Z

resolves #82435

doc preview: https://kibana_92736.docs-preview.app.elstc.co/guide/en/kibana/master/alerting-production-considerations.html

Summary

I added this to the alert/actions setting doc, as that seemed like the best
place for now. Just provided a brief description, name of the policy, mentioned
we create it but never modify it, provided the default values, and mentioned
it could be updated by customers for their environment. Not sure we want to
provide more info than that.

Checklist

Delete any items that are not applicable to this PR.

Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
Documentation was added for features that require explanation or tutorials
~~Unit or functional tests were updated or added to match the most common scenarios~~
~~Any UI touched in this PR is usable by keyboard only (learn more about keyboard accessibility)~~
~~Any UI touched in this PR does not create any new axe failures (run axe in browser: FF, Chrome)~~
~~If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the docker list~~
~~This renders correctly on smaller devices using a responsive layout. (You can test this in your browser)~~
~~This was checked for cross-browser compatibility~~

For maintainers

~~This was checked for breaking API changes and was labeled appropriately~~

resolves elastic#82435 I added this to the alert/actions setting doc, as that seemed like the best place for now. Just provided a brief description, name of the policy, mentioned we create it but never modify it, provided the default values, and mentioned it could be updated by customers for their environment. Not sure we want to provide more info than that.

elasticmachine · 2021-03-01T22:19:58Z

Pinging @elastic/kibana-alerting-services (Team:Alerting Services)

gchaps

The new text doesn't seem to fit in the settings doc. Is there another place it might go?

gchaps · 2021-03-01T22:57:10Z

docs/settings/alert-action-settings.asciidoc

+[[event-log-ilm]]
+==== Event log index lifecycle managment
+
+Alerts and actions log their activity in a set of "event log" indices.  These indices are configured with an index lifecycle managment (ILM) policy which can be customized.  The default policy will perform a rollover of the index when it reaches 50GB in size, or after 30 days.  Indices over 90 days old will be deleted.


Suggested change

Alerts and actions log their activity in a set of "event log" indices. These indices are configured with an index lifecycle managment (ILM) policy which can be customized. The default policy will perform a rollover of the index when it reaches 50GB in size, or after 30 days. Indices over 90 days old will be deleted.

Alerts and actions log activity in a set of "event log" indices. These indices are configured with an index lifecycle management (ILM) policy, which you can customize. The default policy rolls over the index when it reaches 50GB, or after 30 days. Indices over 90 days old are deleted.

gchaps · 2021-03-01T23:01:07Z

docs/settings/alert-action-settings.asciidoc

+
+Alerts and actions log their activity in a set of "event log" indices.  These indices are configured with an index lifecycle managment (ILM) policy which can be customized.  The default policy will perform a rollover of the index when it reaches 50GB in size, or after 30 days.  Indices over 90 days old will be deleted.
+
+The name of the index policy is `kibana-event-log-policy`.  It will be created when Kibana starts, if it doesn't exist, otherwise it will never be modified by Kibana.  It can be customized for your environment.


Suggested change

The name of the index policy is `kibana-event-log-policy`. It will be created when Kibana starts, if it doesn't exist, otherwise it will never be modified by Kibana. It can be customized for your environment.

The name of the index policy is `kibana-event-log-policy`. {kib} creates the index policy on startup, if it doesn't already exist. The index policy can be customized for your environment, but {kib} never modifies the index policy after creating it.

I didn't understand the part about "otherwise it will never be modified by Kibana" so my last sentence is a guess.

The idea is that we create the ILM policy if it doesn't already exist, with some default values. But we never modify it after creating it - there is no UI in Kibana to explicitly modify it (outside of the general ui for all ILM policies that Kibana has). We also never programmatically modify it.

So a customer can make modifications to the policy, to keep data for a shorter or longer period of time, etc. And not worry that alerting will somehow "overwrite" their settings while it's running, or when upgrading versions, etc. We just create it when needed, and then never directly access it (even for reading!) after that.

So the wording "You can customize the index policy prior to startup, but not after." is exactly wrong! hehe

Customers should NOT "pre-create" the index policy, because then we won't create one, but we actually DO need to create it to set all the index patterns, aliases, etc. And customers may modify the ILM policy at any time after it's created. Note again, the policy is created at Kibana startup time; this was added in ~7.9 time frame, so anyone upgrading from 7.9 or higher will already have one. Brand new deployments will have one created when Kibana starts up.

new suggested wording from @gchaps:

Kibana creates the index policy on startup, if it doesn't already exist. The index policy can be customized for your environment, but Kibana never modifies the index policy after creating it.

I updated the suggestion.

gchaps · 2021-03-01T23:10:11Z

docs/settings/alert-action-settings.asciidoc

+
+The name of the index policy is `kibana-event-log-policy`.  It will be created when Kibana starts, if it doesn't exist, otherwise it will never be modified by Kibana.  It can be customized for your environment.
+
+The event log is current used to display historic data on alert and action activity, and for diagnostic purposes.


Suggested change

The event log is current used to display historic data on alert and action activity, and for diagnostic purposes.

The event log displays historic data on alert and action activity, for diagnostic purposes.

The wording on this is not quite right, as the event log doesn't "display" anything itself. It just contains the data that Kibana uses to display historic information in the UI.

I think we can also drop the "diagnostic purposes" part of this - my bad. I have some separate "how to use the event log for diagnostic purposes" info I'm going to be putting in the event log README.md for now (not quite ready for "production"). I was hoping to drop a hint that "there's some interesting stuff in this index!", but that's just teasing without more info.

Here's a new take:

Because the documents in the event log are used to display historic alert and action activity in {kib}, care should be taken when using the ILM delete phase. For example, if the rolled over indices are deleted after 2 weeks, no historic data will be shown in {kib} that is more than 2 weeks old.

Nice! Suggest changing "care should be taken when" to "take care when".

pmuellr · 2021-03-02T16:04:28Z

The new text doesn't seem to fit in the settings doc. Is there another place it might go?

Yeah, it's certainly not a kibana.yml type setting, but the ILM policy for this set of indices is a "setting" in general. I think it should be kept with other alerting topics, otherwise I feel like no one would ever see it.

I considered putting it in "Production considerations" as well, and that would be fine. https://www.elastic.co/guide/en/kibana/7.12/alerting-production-considerations.html

Open for any other thoughts though.

gchaps · 2021-03-02T17:03:37Z

I like the idea of moving the content to production considerations.

ymao1

LGTM! Just a question about header size

ymao1 · 2021-03-04T00:28:38Z

docs/user/alerting/alerting-production-considerations.asciidoc

+
+[float]
+[[event-log-ilm]]
+==== Event log index lifecycle managment


Should this be a sibling to Deployment considerations instead of a sub-heading?

Yeah, I think you're right. Will change that.

mikecote

Changes LGTM. The location of these docs will probably have to change when other plugins start using the event log?

mikecote · 2021-03-04T15:35:49Z

docs/user/alerting/alerting-production-considerations.asciidoc

+
+The name of the index policy is `kibana-event-log-policy`.  {kib} creates the index policy on startup, if it doesn't already exist.  The index policy can be customized for your environment, but {kib} never modifies the index policy after creating it.
+
+Because the documents in the event log are used to display historic alert and action activity in {kib}, take care when using the ILM delete phase.  For example, if the rolled over indices are deleted after 2 weeks, no historic data will be shown in {kib} that is more than 2 weeks old.


I'm not sure what it means take care when using the ILM delete phase. Did you mean be careful when modifying the ILM delete phase? or cautious?

cc @gchaps

could be phrased better, for sure. Maybe something like:

Because Kibana uses the documents to display historic data, you should set the delete phase longer than you would like the historic data to be shown. For example, if you would like to see one month's worth of historic data, you should set the delete phase to at least one month.

That suggestion looks good to me 👍

pmuellr · 2021-03-04T21:44:57Z

The location of these docs will probably have to change when other plugins start using the event log?

Yes, I think it would go somewhere under the top-level Config section at the top, but not sure where. I originally put it in the alert settings page under Config, which wasn't great since it's not strictly "config" (and was still under alerting anyway).

Still open to move it somewhere else. This seems like the best place for now, otherwise it's hard to imagine anyone finding it - though if we do move it, we could add a link to the new location in the alerting page I have it in now.

mikecote · 2021-03-05T01:25:05Z

Still open to move it somewhere else. This seems like the best place for now, otherwise it's hard to imagine anyone finding it - though if we do move it, we could add a link to the new location in the alerting page I have it in now.

Yeah, I'm good waiting until the event log isn't explicit for alerting before moving the docs 👍

pmuellr · 2021-03-24T23:46:30Z

This PR is so stale, the single file that had content added to it moved!

from:

docs/user/alerting/alerting-production-considerations.asciidoc

to

docs/user/production-considerations/alerting-production-considerations.asciidoc

So the backport to 7.11 is going to require a tiny bit of extra work, that hardly seems worth it considering it's the 7.11 release, heh. Going to skip that for now; LMK if there's concern about backports - technically this goes all the way back to 7.8.

resolves elastic#82435 Just provided a brief description, name of the policy, mentioned we create it but never modify it, provided the default values, and mentioned it could be updated by customers for their environment. Not sure we want to provide more info than that.

pmuellr · 2021-03-25T12:40:03Z

Even the backport to 7.12 had conflicts, as we've made alerts => rules type changes. Rather than screw that up in the backport, I'll just not backport there either.

resolves #82435 Just provided a brief description, name of the policy, mentioned we create it but never modify it, provided the default values, and mentioned it could be updated by customers for their environment. Not sure we want to provide more info than that.

pmuellr force-pushed the event-log/ilm-user-doc branch from 2a662bd to a19c0d9 Compare March 1, 2021 22:16

pmuellr added Feature:EventLog Team:ResponseOps Label for the ResponseOps team (formerly the Cases and Alerting teams) Feature:Alerting release_note:skip Skip the PR/issue when compiling release notes v7.11.0 v7.12.0 v7.13.0 v8.0.0 and removed Feature:Alerting labels Mar 1, 2021

pmuellr marked this pull request as ready for review March 1, 2021 22:19

pmuellr requested a review from gchaps March 1, 2021 22:20

gchaps reviewed Mar 1, 2021

View reviewed changes

move to perf considerations, tweak some text

2640130

pmuellr requested a review from a team as a code owner March 3, 2021 15:41

Merge branch 'master' into event-log/ilm-user-doc

118fd9f

ymao1 approved these changes Mar 4, 2021

View reviewed changes

mikecote self-requested a review March 4, 2021 15:31

mikecote approved these changes Mar 4, 2021

View reviewed changes

LeeDr added the documentation label Mar 18, 2021

pmuellr added 2 commits March 24, 2021 12:11

final changes from review

0458d2f

move the event log ilm doc to the newly moved file

f018d6a

pmuellr removed the v7.11.0 label Mar 25, 2021

pmuellr added v7.12.1 and removed v7.12.0 labels Mar 25, 2021

pmuellr merged commit 9d47330 into elastic:master Mar 25, 2021

pmuellr mentioned this pull request Mar 25, 2021

[7.x] [alerting] add user facing doc on event log ILM policy (#92736) #95404

Merged

pmuellr removed the v7.12.1 label Mar 25, 2021

pmuellr added the backported label Mar 25, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[alerting] add user facing doc on event log ILM policy #92736

[alerting] add user facing doc on event log ILM policy #92736

pmuellr commented Feb 24, 2021 •

edited

Loading

elasticmachine commented Mar 1, 2021

gchaps left a comment

gchaps Mar 1, 2021

gchaps Mar 1, 2021 •

edited

Loading

gchaps Mar 1, 2021

pmuellr Mar 2, 2021

pmuellr Mar 2, 2021

pmuellr Mar 2, 2021

gchaps Mar 2, 2021

gchaps Mar 1, 2021

pmuellr Mar 2, 2021

gchaps Mar 2, 2021

pmuellr commented Mar 2, 2021

gchaps commented Mar 2, 2021

ymao1 left a comment

ymao1 Mar 4, 2021

pmuellr Mar 4, 2021

mikecote left a comment

mikecote Mar 4, 2021

pmuellr Mar 4, 2021

mikecote Mar 5, 2021

pmuellr commented Mar 4, 2021

mikecote commented Mar 5, 2021

pmuellr commented Mar 24, 2021

pmuellr commented Mar 25, 2021

	Alerts and actions log their activity in a set of "event log" indices. These indices are configured with an index lifecycle managment (ILM) policy which can be customized. The default policy will perform a rollover of the index when it reaches 50GB in size, or after 30 days. Indices over 90 days old will be deleted.
	Alerts and actions log activity in a set of "event log" indices. These indices are configured with an index lifecycle management (ILM) policy, which you can customize. The default policy rolls over the index when it reaches 50GB, or after 30 days. Indices over 90 days old are deleted.


		Alerts and actions log their activity in a set of "event log" indices. These indices are configured with an index lifecycle managment (ILM) policy which can be customized. The default policy will perform a rollover of the index when it reaches 50GB in size, or after 30 days. Indices over 90 days old will be deleted.

		The name of the index policy is `kibana-event-log-policy`. It will be created when Kibana starts, if it doesn't exist, otherwise it will never be modified by Kibana. It can be customized for your environment.


		The name of the index policy is `kibana-event-log-policy`. It will be created when Kibana starts, if it doesn't exist, otherwise it will never be modified by Kibana. It can be customized for your environment.

		The event log is current used to display historic data on alert and action activity, and for diagnostic purposes.

	The event log is current used to display historic data on alert and action activity, and for diagnostic purposes.
	The event log displays historic data on alert and action activity, for diagnostic purposes.


		The name of the index policy is `kibana-event-log-policy`. {kib} creates the index policy on startup, if it doesn't already exist. The index policy can be customized for your environment, but {kib} never modifies the index policy after creating it.

		Because the documents in the event log are used to display historic alert and action activity in {kib}, take care when using the ILM delete phase. For example, if the rolled over indices are deleted after 2 weeks, no historic data will be shown in {kib} that is more than 2 weeks old.

[alerting] add user facing doc on event log ILM policy #92736

[alerting] add user facing doc on event log ILM policy #92736

Conversation

pmuellr commented Feb 24, 2021 • edited Loading

Summary

Checklist

For maintainers

elasticmachine commented Mar 1, 2021

gchaps left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gchaps Mar 1, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pmuellr commented Mar 2, 2021

gchaps commented Mar 2, 2021

ymao1 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mikecote left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pmuellr commented Mar 4, 2021

mikecote commented Mar 5, 2021

pmuellr commented Mar 24, 2021

pmuellr commented Mar 25, 2021

pmuellr commented Feb 24, 2021 •

edited

Loading

gchaps Mar 1, 2021 •

edited

Loading