Enable TSDB downsampling ILM configuration #138748

Dosant · 2022-08-12T15:04:07Z

Summary

Close #130437

Enables TSDB downsampling action in ILM configuration.
Actions can be set in the hot, warm, cold phases.

It is hidden in warm and cold phases when searchable snapshots are enabled in hot phase (just like shrink, readonly, force merge actions).

Users need to specify an interval:

Intervals have the following validation:

they must be greater than any previous interval and a multiple of that interval.
- For example, on the warm phase, the user includes a 3-hour interval; if they choose to add a rollup action on the cold phase, the interval must be a multiple of 3h. It cannot be 5h, for example.
- Another example: a user can have a rollup on the hot phase of 2d and then no rollup action on the warm phase, and then a rollup action on the cold phase, but it must be a multiple of 2d, for example, it cannot be 1d nor can it be 3d.

How to test

(What I tested)

Create an ILM policy with Hot phase only. Enabled rollover, disable "use recommended settings" and set "maximum documents to 1
Enable new "Downsample" action, set interval to 1 hour.
Save policy
Then create a data stream template using Dev tools: https://github.com/csoulios/downsampling-demo/blob/main/downsample-ilm.es#L26-L169. Currently, it isn't possible to create this template using UI because of [Mappings Editor] allow time series related properties #139922. Make sure you are attaching your ilm policy to the template. You can also do it through UI
Create data stream by injesting data using Dev tools: https://github.com/csoulios/downsampling-demo/blob/1e66a13bccc7d70aae7763bcc9c4499ec8b3fd02/downsample-ilm.es#L174-L194 . You will have to change timestamp in the data to the current hour
Inspect indices and data streams. You should see them changed during the next hour. Inspect data of the downsampled stream in Discover

Note

Rollup of rollups is not yet implemented on es side. We should test everything end to end when it is ready. We wouldn't need to change ILM screen when it ready so we can merge without waiting for that.
Ideally we should have a es docs link about ILM and downsampling to point to

Release Notes

Enables time series downsampling action in ILM configuration

Dosant · 2022-09-06T10:00:03Z

@elasticmachine merge upstream

…m-rollup

Dosant · 2022-09-07T12:21:42Z

@elasticmachine merge upstream

…m-rollup

elasticmachine · 2022-09-09T13:24:36Z

Pinging @elastic/platform-deployment-management (Team:Deployment Management)

elasticmachine · 2022-09-09T13:24:37Z

Pinging @elastic/kibana-app-services (Team:AppServicesUx)

jloleysens

Great work @Dosant !

I had two observations I wanted to get y'alls thoughts on (both non-blockers):

First: @yuliacech (and @Dosant) as I was interacting with this beast of a form I thought one thing that could be really useful is to add the ability to filter out actions - maybe like a global fuzzy matcher so that if I type "down" then all the advanced options will auto expand and show only matching actions. This increases the chances that you'll be able to see the continuity between all relevant config more easily. (I'm thinking of this as a future enhancement of course). Similar to the UX you get when searching for settings in VS Code:

Second: When I enable the downsampling action for the first time the field starts in an errored state.

I'm guessing this is because the form is not in a pristine state anymore (you need to enable the action). Would it be possible to have a default value here?

Otherwise, this PR LGTM tested locally.

x-pack/plugins/index_lifecycle_management/common/types/policies.ts

jloleysens · 2022-09-12T08:34:33Z

...ugins/index_lifecycle_management/public/application/sections/edit_policy/form/validations.ts

+          return {
+            message: i18n.translate(
+              'xpack.indexLifecycleMgmt.editPolicy.downsamplePreviousIntervalWarmPhaseError',
+              {
+                defaultMessage:
+                  'Must be greater than and a multiple of the hot phase value ({value})',
+                values: {
+                  value: intervalValues.hot.esFormat,
+                },
+              }
+            ),
+          };


Do we want to provide a pre-filled number for downsampling here? One thing that is tricky about this form is how big it is and I feel like scrolling between phases you can easily get distracted by all the shiny things.

One alternative I thought of was providing a list of multiples in the error message like: (ex. 10 days, 20 days...)

Might not be worth the effort.

Do we want to provide a pre-filled number for downsampling here?

As I understand, this isn't a low effort because this wouldn't be just a static value and have to depend on the previous step:
e.g., if someone configured a 1-day interval in a hot phase and then enabled a warm phase, then we should prefill with e.g., 2 days. As I understand, such support for "dynamic" initial values needs to be implemented first

Also really not sure what could be good defaults here.

One alternative I thought of was providing a list of multiples in the error message like: (ex. 10 days, 20 days...)

Could be!

cc @ghudgins

lets worry about optimizations and better defaults in subsequent PRs. hard to say what the rule should be without spending some time with it / hearing from users

Dosant · 2022-09-12T10:53:30Z

@elasticmachine merge upstream

Dosant · 2022-09-12T12:02:49Z

@jloleysens,

Second: When I enable the downsampling action for the first time the field starts in an errored state.
I'm guessing this is because the form is not in a pristine state anymore (you need to enable the action). Would it be possible to have a default value here?

This was because I validated the interval for the current and next phases when downsample action was toggled. I fixed this by validating only the next phases when it is toggled.
just FYI: We need this for the following scenario:

Have hot downsample action enabled and set 1d
Have warm downsample action enabled and set 1d -> see the error
Disable downsample on hot action (need to trigger validation of warm, so that error resets)

note: there is the following edge case:

Have the warm downsample action enabled and set 1d
Have the cold downsample action enabled and set 1d -> see the error
Disable the warm step completely -> error on cold downsample action is still there until triggering change or saving.

To fix this, we need to validate downsample actions when steps are toggled. But I thought this is an overkill for this edge case.

yuliacech · 2022-09-12T12:28:18Z

Hey @jloleysens, thanks a lot for this suggestion, I think that is a great idea for the ILM form, it's becoming scarier with each PR 😆 Do you mind opening an issue for that so we don't forget?

Thanks a lot for working on this feature @Dosant! I had a quick look and the validation worked correctly for me. I agree with JL that the interval input should not be marked as error as soon as the user toggles the Downsampling action. I think there is some similar logic in the MinAge field. The field is required if the phase is enabled but it's nor marked as error straight away. Also there is similar validation logic in this field, because the MinAge can't be smaller than in the previous phase. Also is the Downsampling action also available in the Frozen phase?

A suggestion I have for a follow up PR: since downsampling turns the index read-only automatically, I think we could hide the ReadOnly action in the phase if there is a downsampling action enabled in it or in any of the previous phases.

Could you please add the Downsampling action to this text in the Hot phase?

Dosant · 2022-09-12T12:33:53Z

@yuliacech,

I agree with JL that the interval input should not be marked as error as soon as the user toggles the Downsampling action. I think there is some similar logic in the MinAge field. The field is required if the phase is enabled but it's nor marked as error straight away.

this should be fixed 👍 #138748 (comment)

Also is the Downsampling action also available in the Frozen phase?

No! only hot,warm,cold

A suggestion I have for a follow-up PR: since downsampling turns the index read-only automatically, I think we could hide the ReadOnly action in the phase if there is a downsampling action enabled in it or in any of the previous phases.

Good suggestion. I will create an issue and will try in a separate pr #140515

Could you please add the Downsampling action to this text in the Hot phase?

👍

alisonelizabeth

Nice work @Dosant!

I agree with Yulia and JL's comments on looking into ways we can improve the usability of the form in a future iteration.

I left some minor code comments, nothing blocking.

Also is the Downsampling action also available in the Frozen phase?

No! only hot,warm,cold

I noticed in your last comment on the issue, you mentioned it should be available in the frozen phase, but is not yet available in ES. Is this future work?

alisonelizabeth · 2022-09-12T15:19:59Z

...nagement/__jest__/client_integration/edit_policy/form_validation/downsample_interval.test.ts

+
+    // disable warm phase;
+    await actions.togglePhase('warm');
+    // TODO: there is a bug that disabling a phase doesn't trigger downsample validation in other phases,


Is there a separate issue open to address this? Will the validation work as expected on save?

Will the validation work as expected on saving?

Yes! the form state stays in the error state until the validation is triggered again.
More details on the case here: #138748 (comment)

note: there is the following edge case:
Have the warm downsample action enabled and set 1d
Have the cold downsample action enabled and set 1d -> see the error
Disable the warm step completely -> error on cold downsample action is still there > until triggering change or saving.
To fix this, we need to validate downsample actions when steps are toggled. But I thought this is an overkill for this edge case.

Is there a separate issue open to address this?

I wanted to leave a TODO comment to point this out as a known bug/edge case but didn't think this was worth an issue

I'll see if I can fix this with low effort: #140628

alisonelizabeth · 2022-09-12T15:21:33Z

...nagement/__jest__/client_integration/edit_policy/form_validation/downsample_interval.test.ts

+      ['Must be greater than and a multiple of the hot phase value (60m)'],
+      'cold'
+    );
+  }, /* increase a timeout of this test as it could fail on ci*/ 12000);


Out of curiosity - is there any more context you can provide here? Was the test failing without the increased timeout?

It was failing with a 5-second timeout because it was a very long test. initially, I also implemented a frozen phase and tested it here. Since we removed the frozen phase, I also removed a timeout, as I expect it should fit in 5 seconds now.

alisonelizabeth · 2022-09-12T15:34:51Z

...public/application/sections/edit_policy/components/phases/shared_fields/downsample_field.tsx

+            defaultMessage="Roll up documents within a fixed interval to a single summary document. Reduces the index footprint by storing time series data at reduced granularity."
+          />{' '}
+          {/* TODO: add when available*/}
+          {/* <LearnMoreLink docPath={docLinks.links.elasticsearch.ilmDownsample} /> */}


Is this documentation planned, or already in progress? I'm wondering if it makes sense to remove this code until it is ready.

I believe this is planned: #130437 (comment) cc @debadair

alisonelizabeth · 2022-09-12T15:38:00Z

x-pack/plugins/index_lifecycle_management/public/application/sections/edit_policy/constants.ts

+/*
+ * Labels for fixed intervals
+ */
+export const fixedIntervalUnits = [


I see we already have a timeUnits const. Just want to confirm these values should be separate?

micros and nanos are technically not supported for the fixed interval.
btw, I saw this: #137081

I think we can unite these consts if we remove everything smaller then a minute

alisonelizabeth · 2022-09-12T15:43:16Z

...nagement/__jest__/client_integration/edit_policy/form_validation/downsample_interval.test.ts

@@ -0,0 +1,130 @@
+/*


Thanks for adding these tests 👍

…m-rollup

Dosant · 2022-09-13T10:40:18Z

I noticed in your #130437 (comment), you mentioned it should be available in the frozen phase, but is not yet available in ES. Is this future work?

@alisonelizabeth,

There was some back-and-forth :(
Initially, we planned hot,warm,cold.
Then we agreed to add frozen. I added frozen.
Then we agreed not to add frozen (it turned out to be more difficult on es side than originally thought). I removed the frozen.
The last state: keep hot,warm,cold

kibana-ci · 2022-09-13T12:00:56Z

💚 Build Succeeded

Buildkite Build
Commit: aa43923

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
`indexLifecycleManagement`	189	190	+1

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`indexLifecycleManagement`	152.8KB	158.7KB	+5.8KB

History

💚 Build #71250 succeeded 2f7fb75
💚 Build #71224 succeeded decd7bc
💚 Build #71207 succeeded 52628b6
💚 Build #70786 succeeded 8acc8ca
💚 Build #70483 succeeded 1216030

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

alisonelizabeth · 2022-09-13T13:36:34Z

@Dosant I forgot to ask during my review - I noticed you have the release_note:skip label on this. However, this seems like something we might want to add a release note for?

Dosant changed the title ~~ilm rollup wip~~ [wip] Enable TSDB downsampling ILM configuration Aug 24, 2022

sebelga mentioned this pull request Aug 24, 2022

[Form lib] Fix race condition when running validation #139385

Merged

Dosant mentioned this pull request Aug 31, 2022

Enable TSDB downsampling ILM configuration #130437

Closed

enable tsdb downsampling action in ilm

5d7a792

Dosant force-pushed the d/2022-08-04-ilm-rollup branch from 51107de to 5d7a792 Compare September 6, 2022 14:45

Dosant added 2 commits September 7, 2022 12:04

Merge branch 'main' of github.com:elastic/kibana into d/2022-08-04-il…

1094f89

…m-rollup

increase validation test timeout

3bcb2ba

Dosant added the ci:cloud-deploy Create or update a Cloud deployment label Sep 7, 2022

Merge branch 'main' into d/2022-08-04-ilm-rollup

46bf380

Dosant mentioned this pull request Sep 7, 2022

[Index Template] Fix previewing data streams in template editor #140189

Merged

Dosant added 4 commits September 8, 2022 16:23

Merge branch 'main' of github.com:elastic/kibana into d/2022-08-04-il…

0fa4a3e

…m-rollup

remove frozen phase support

1216030

Merge branch 'main' of github.com:elastic/kibana into d/2022-08-04-il…

dc1d06f

…m-rollup

rollup -> downsample

8acc8ca

Dosant added Team:Kibana Management Dev Tools, Index Management, Upgrade Assistant, ILM, Ingest Node Pipelines, and more release_note:skip Skip the PR/issue when compiling release notes Team:AppServicesUx v8.5.0 labels Sep 9, 2022

Dosant marked this pull request as ready for review September 9, 2022 13:24

Dosant requested a review from a team as a code owner September 9, 2022 13:24

Dosant changed the title ~~[wip] Enable TSDB downsampling ILM configuration~~ Enable TSDB downsampling ILM configuration Sep 9, 2022

Dosant requested review from jloleysens, ghudgins, a team and debadair September 9, 2022 13:25

cjcenizal requested a review from alisonelizabeth September 9, 2022 18:27

jloleysens approved these changes Sep 12, 2022

View reviewed changes

kibanamachine and others added 3 commits September 12, 2022 04:53

Merge branch 'main' into d/2022-08-04-ilm-rollup

cd03455

remove resolved todos

52628b6

fix form error state when enabling downsample

decd7bc

adjust label

2f7fb75

Dosant removed the ci:cloud-deploy Create or update a Cloud deployment label Sep 12, 2022

jloleysens mentioned this pull request Sep 12, 2022

Filter for ILM actions in the ILM form #140497

Open

Dosant mentioned this pull request Sep 12, 2022

[ILM] Hide readonly action if downsample enabled #140515

Closed

alisonelizabeth approved these changes Sep 12, 2022

View reviewed changes

Dosant added 2 commits September 13, 2022 12:33

remove test timeout

9239b3a

Merge branch 'main' of github.com:elastic/kibana into d/2022-08-04-il…

aa43923

…m-rollup

Dosant merged commit 678edce into elastic:main Sep 13, 2022

kibanamachine added the backport:skip This commit does not require backporting label Sep 13, 2022

Dosant added release_note:feature Makes this part of the condensed release notes and removed release_note:skip Skip the PR/issue when compiling release notes labels Sep 13, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable TSDB downsampling ILM configuration #138748

Enable TSDB downsampling ILM configuration #138748

Dosant commented Aug 12, 2022 •

edited

Dosant commented Sep 6, 2022

Dosant commented Sep 7, 2022

elasticmachine commented Sep 9, 2022

elasticmachine commented Sep 9, 2022

jloleysens left a comment •

edited

jloleysens Sep 12, 2022

Dosant Sep 12, 2022 •

edited

ghudgins Sep 12, 2022

Dosant commented Sep 12, 2022

Dosant commented Sep 12, 2022

yuliacech commented Sep 12, 2022

Dosant commented Sep 12, 2022 •

edited

alisonelizabeth left a comment

alisonelizabeth Sep 12, 2022

Dosant Sep 13, 2022

Dosant Sep 13, 2022

alisonelizabeth Sep 12, 2022

Dosant Sep 13, 2022 •

edited

alisonelizabeth Sep 12, 2022

Dosant Sep 13, 2022

alisonelizabeth Sep 12, 2022

Dosant Sep 13, 2022

alisonelizabeth Sep 12, 2022

Dosant commented Sep 13, 2022

kibana-ci commented Sep 13, 2022

alisonelizabeth commented Sep 13, 2022

Enable TSDB downsampling ILM configuration #138748

Enable TSDB downsampling ILM configuration #138748

Conversation

Dosant commented Aug 12, 2022 • edited

Summary

How to test

Note

Release Notes

Dosant commented Sep 6, 2022

Dosant commented Sep 7, 2022

elasticmachine commented Sep 9, 2022

elasticmachine commented Sep 9, 2022

jloleysens left a comment • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Dosant Sep 12, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Dosant commented Sep 12, 2022

Dosant commented Sep 12, 2022

yuliacech commented Sep 12, 2022

Dosant commented Sep 12, 2022 • edited

alisonelizabeth left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Dosant Sep 13, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Dosant commented Sep 13, 2022

kibana-ci commented Sep 13, 2022

💚 Build Succeeded

Metrics [docs]

Module Count

Async chunks

History

alisonelizabeth commented Sep 13, 2022

Dosant commented Aug 12, 2022 •

edited

jloleysens left a comment •

edited

Dosant Sep 12, 2022 •

edited

Dosant commented Sep 12, 2022 •

edited

Dosant Sep 13, 2022 •

edited