Skip to content

Document ILM waits for tsds end_time to lapse in some actions #100204

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Oct 4, 2023
Merged

Document ILM waits for tsds end_time to lapse in some actions #100204

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
5b04938
Document ILM waits for tsds end_time to lapse in some actions
andreidan Oct 3, 2023
297d24d
Docs update
andreidan Oct 4, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 11 additions & 2 deletions docs/reference/data-streams/downsampling-ilm.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,8 +37,17 @@ To enable downsampling, add a <<ilm-downsample,Downsample action>> and set
which you want to aggregate the original time series data.

In this example, an ILM policy is configured for the `hot` phase. The downsample
takes place after the initial index rollover, which for demonstration
purposes is set to run after five minutes.
takes place after the index is rolled over and the <<index-time-series-end-time, index time series end time>>
has lapsed as the source index is still expected to receive major writes until then.
{ilm-cap} will not proceed with any action that expects the index to not receive
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

receive major writes? After end_time is passed a backing index can still accept writes, but the expectation is that the volume of writes will be lower.

writes anymore until the <<index-time-series-end-time, index's end time>> has
passed. The {ilm-cap} actions that wait on the end time before proceeding are:
- <<ilm-delete>>
- <<ilm-downsample>>
- <<ilm-forcemerge>>
- <<ilm-readonly>>
- <<ilm-searchable-snapshot>>
- <<ilm-shrink>>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

and ilm-delete? (when other pr is merged)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

++ done, thanks Martijn


[source,console]
----
Expand Down
4 changes: 4 additions & 0 deletions docs/reference/data-streams/tsds-index-settings.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,10 @@ days). Only indices with an `index.mode` of `time_series` support this setting.
For more information, refer to <<tsds-look-ahead-time>>. Additionally this setting
can not be less than `time_series.poll_interval` cluster setting.

NOTE: Increasing the `look_ahead_time` will also increase the amount of time {ilm-cap}
waits before being able to proceed with executing the actions that expect the
index to not receive any writes anymore. For more information, refer to <<time-bound-indices>>.

[[index-look-back-time]]
`index.look_back_time`::
(<<_static_index_settings,Static>>, <<time-units,time units>>)
Expand Down
15 changes: 11 additions & 4 deletions docs/reference/data-streams/tsds.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -218,10 +218,17 @@ if the index isn't the most recent backing index.

image::images/data-streams/time-bound-indices.svg[align="center"]

TIP: Some {ilm-init} actions, such as <<ilm-forcemerge,`forcemerge`>>,
<<ilm-shrink,`shrink`>>, and <<ilm-searchable-snapshot,`searchable_snapshot`>>,
make a backing index read-only. You cannot add documents to read-only indices.
Keep this in mind when defining the index lifecycle policy for your TSDS.
TIP: Some {ilm-init} actions mark the source index as read-only, or expect the index
to not be actively written anymore in order to provide good performance. These actions are:
- <<ilm-delete>>
- <<ilm-downsample>>
- <<ilm-forcemerge>>
- <<ilm-readonly>>
- <<ilm-searchable-snapshot>>
- <<ilm-shrink>>
{ilm-cap} will **not** proceed with executing these actions until the upper time-bound
for accepting writes, represented by the <<index-time-series-end-time,`index.time_series.end_time`>>
index setting, has lapsed.

If no backing index can accept a document's `@timestamp` value, {es} rejects the
document.
Expand Down