Add practical tips for downsampling. #3340
Conversation
|
Can we wait for Marci to submit her update, then port these tips to the new structure? |
| ## Practical tips | ||
|
|
||
| Downsampling requires reading and indexing the contents of a backing index. The following guidelines can help you get the most out of it. | ||
|
|
There was a problem hiding this comment.
Do we need a note about rollover? To avoid creating backing indices that are too big..
There was a problem hiding this comment.
I have been going back and forth for this. For ILM it's easy because it's part of the policy, for data stream lifecycle, I would suggest that if we really think that it should be less maybe we should set it to something less. Right?
There was a problem hiding this comment.
You mean, update the default? We can do that at a later point, but what about older versions, or ILM configurations with existing rollover overrides? It could still help to suggest a best practice here.
There was a problem hiding this comment.
Yes, we could update the default, that would apply on all version unless the user chose to overwrite it. I restructure it a bit so we can have ILM focused recommendations. But if we think it should be reduced, we should consider updating the default for DLM as well.
There was a problem hiding this comment.
Let's file a tracking issue for this, so that we don't forget.
It is created on the updated downsampling page. Right? |
🔍 Preview links for changed docs |
Co-authored-by: Kostas Krikellas <131142368+kkrik-es@users.noreply.github.com>
Co-authored-by: Kostas Krikellas <131142368+kkrik-es@users.noreply.github.com>
|
|
||
| ### Choosing the downsampling interval | ||
|
|
||
| When choosing the downsampling interval, you need to consider the original sampling rate of your measurements. Ideally, you would like an interval that would reduce your number of documents by a significant amount. For example, if a sensor sends data every 10 seconds downsampling to 1 minute would reduce the number of documents by 83%, compared to downsampling to 5 minutes by 96%. |
There was a problem hiding this comment.
| When choosing the downsampling interval, you need to consider the original sampling rate of your measurements. Ideally, you would like an interval that would reduce your number of documents by a significant amount. For example, if a sensor sends data every 10 seconds downsampling to 1 minute would reduce the number of documents by 83%, compared to downsampling to 5 minutes by 96%. | |
| When choosing the downsampling interval, you need to consider the original sampling rate of your measurements. Ideally, you would like an interval that would reduce your number of documents by a significant amount. For example, if a sensor sends data every 10 seconds, downsampling to 1 minute would reduce the number of documents by 83%, compared to downsampling to 5 minutes by 96%. |
Co-authored-by: Marci W <333176+marciw@users.noreply.github.com>
|
|
||
| ### Reduce the index size (ILM only) | ||
|
|
||
| When configuring an ILM policy with downsampling, use the [rollover action](elasticsearch://reference/elasticsearch/index-lifecycle-actions/ilm-rollover.md) in the `hot` phase to control index size. Using smaller indices helps to minimize the impact of downsampling on a cluster's performance. |
There was a problem hiding this comment.
I think it's important here to say use and not define. When writing an ILM policy a user needs to define a rollover action no matter what. However, if they are using downsampling, they can consider using this to reduce the size of their index. I want us to be careful to not imply that a user needs to define it only if they are trying to reduce the size. @marciw does this make sense?
There was a problem hiding this comment.
Yes, I see what you mean. we could try "set the rollover action to run in the hot phase"
("use" doesn't seem idiomatic to me here, so i'm hoping we can find an alternative)
|
I am adding @leontyevdv as a reviewer because he recently watched a tutorial on how to configure TSDS and he can tell us how well it reads for a user. |
4 participants
In this PR, we propose to add practical tips for downsampling. For now this includes, a guideline on how to choose the downsampling interval. And then specifically for ILM, an explanation on how downsampling relates with tiers. After elastic/elasticsearch#135834, we should also add here the option to disable force merge.