Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Prioritize recovery of system indices #61660

Closed
Tracked by #50251
jaymode opened this issue Aug 27, 2020 · 3 comments · Fixed by #62640
Closed
Tracked by #50251

Prioritize recovery of system indices #61660

jaymode opened this issue Aug 27, 2020 · 3 comments · Fixed by #62640
Assignees
@pugnascotia
Labels
:Distributed/Recovery Anything around constructing a new shard, either from a local or a remote source. >enhancement Team:Distributed Meta label for distributed team

Comments

@jaymode
Copy link
Member

jaymode commented Aug 27, 2020

During startup, system indices should be prioritized over the recovery of normal indices/user data so that the stack may be fully operational once user data is recovered.

Going beyond that, system indices themselves may need to be prioritized in terms of recovery operations as some system components may depend on others. We may be able to reuse the order that exists on indices today in terms of prioritizing which system indices get recovered first.
@jaymode jaymode added >enhancement :Distributed/Recovery Anything around constructing a new shard, either from a local or a remote source. labels Aug 27, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-distributed (:Distributed/Recovery)

@elasticmachine elasticmachine added the Team:Distributed Meta label for distributed team label Aug 27, 2020
@jaymode jaymode mentioned this issue Aug 27, 2020
Open
23 tasks
@pugnascotia
Copy link
Contributor

@jaymode Does this issue only seek to change the current recovery order? Do we also need to think about preventing non-system indices from beginning recovery until the system indices have completed?

It looks like PriorityComparator is the right place to adjust the recovery order. We'd probably need to look at implementing a new AllocationDecider sub-class if we need to do recovery blocking.

@jaymode
Copy link
Member Author

jaymode commented Sep 16, 2020

@pugnascotia I think we should allow for concurrent system and non-system index recoveries.

pugnascotia added a commit to pugnascotia/elasticsearch that referenced this issue Sep 18, 2020
@pugnascotia
Prioritise recovery of system index shards
445bf67 
Closes elastic#61660. When ordering shard for recovery, ensure system index
shards are ordered first.
@pugnascotia pugnascotia mentioned this issue Sep 18, 2020
Merged
pugnascotia added a commit that referenced this issue Sep 22, 2020
@pugnascotia
Prioritise recovery of system index shards (#62640)
39a6dec 
Closes #61660. When ordering shard for recovery, ensure system index shards are
ordered first so that their recovery will be started first.

Note that I rewrote PriorityComparatorTests to use IndexMetadata instead of its
local IndexMeta POJO.
pugnascotia added a commit that referenced this issue Sep 22, 2020
@pugnascotia
Prioritise recovery of system index shards (#62640)
3f856d1 
Closes #61660. When ordering shard for recovery, ensure system index shards are
ordered first so that their recovery will be started first.

Note that I rewrote PriorityComparatorTests to use IndexMetadata instead of its
local IndexMeta POJO.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@pugnascotia pugnascotia

Labels
:Distributed/Recovery Anything around constructing a new shard, either from a local or a remote source. >enhancement Team:Distributed Meta label for distributed team
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Prioritise recovery of system index shards pugnascotia/elasticsearch
3 participants
@jaymode @pugnascotia @elasticmachine