Batch Cluster State Updates in Datastream Rollover #79782
Assignees
Labels
:Data Management/Data streams
Data streams and their lifecycles
>enhancement
Team:Data Management
Meta label for data/management team
Comments
original-brownbear
added
>enhancement
:Data Management/Data streams
elasticmachine
added
the
Team:Data Management
|
Pinging @elastic/es-data-management (Team:Data Management) |
This was referenced Oct 26, 2021
martijnvg
added a commit
to martijnvg/elasticsearch
that referenced
this issue
Oct 28, 2021
In cases where many indices are managed by ILM, it is likely that rollovers for different indices or data streams happen concurrently. This change allows the cluster state updates that these rollovers generate to be batch. This change also changes the rollover service to not do a reroute and instead perform a single reroute for multiple batched rollovers. Relates to elastic#77466 Closes to elastic#79782
martijnvg
added a commit
that referenced
this issue
Nov 5, 2021
In cases where many indices are managed by ILM, it is likely that rollovers for different indices or data streams happen concurrently. This change allows the cluster state updates that these rollovers generate to be batch. This change also changes the rollover service to not do a reroute and instead perform a single reroute for multiple batched rollovers. Relates to #77466 Closes to #79782
martijnvg
added a commit
to martijnvg/elasticsearch
that referenced
this issue
Nov 5, 2021
In cases where many indices are managed by ILM, it is likely that rollovers for different indices or data streams happen concurrently. This change allows the cluster state updates that these rollovers generate to be batch. This change also changes the rollover service to not do a reroute and instead perform a single reroute for multiple batched rollovers. Relates to elastic#77466 Closes to elastic#79782
elasticsearchmachine
pushed a commit
that referenced
this issue
Nov 5, 2021
In cases where many indices are managed by ILM, it is likely that rollovers for different indices or data streams happen concurrently. This change allows the cluster state updates that these rollovers generate to be batch. This change also changes the rollover service to not do a reroute and instead perform a single reroute for multiple batched rollovers. Relates to #77466 Closes to #79782
|
Implemented via #79945 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
When using a large number of data-streams combined with rollover schedules that could cause the rollovers to collide a significant number of slow rollover CS updates can queue up. They are mainly slow due to having to run a reroute on every index creation, but also just come with the latency associated with any CS update.
This could lead to the normal priority CS updates getting blocked for many minutes which is not ideal.
Batching the cluster state updates in the
TransportRolloverActionwould allow for an efficient batching of reroutes out of the box as well as of the cluster state updates in general speed up the functionality.relates #77466
The text was updated successfully, but these errors were encountered: