Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Remove index routing table of closed indices in mixed versions clusters #38955

Merged
merged 3 commits into from
Feb 18, 2019
Merged

Remove index routing table of closed indices in mixed versions clusters #38955

merged 3 commits into from
Feb 18, 2019

Conversation

tlrx
Copy link
Member

@tlrx tlrx commented Feb 15, 2019

Note: this PR will be merged in the replicated-closed-indices feature branch

This pull request removes the legacy way of closing indices (aka "direct close") in mixed versions clusters, since this backward compatibility logic is not required anymore on master/8.0.0.

It also changes the closing logic so that routing tables of closed indices are removed when the cluster contains a node in version < 8.0.

Relates #33888

@tlrx tlrx added >enhancement :Distributed/Distributed A catch all label for anything in the Distributed Area. If you aren't sure, use this one. labels Feb 15, 2019
@tlrx tlrx requested a review from ywelsch February 15, 2019 13:42
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-distributed

ywelsch
ywelsch approved these changes Feb 15, 2019
View reviewed changes
Copy link
Contributor

@ywelsch ywelsch left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had a quick look, and it looks good. I'm asking for a second pair of eyes here though.

henningandersen
henningandersen approved these changes Feb 18, 2019
View reviewed changes
Copy link
Contributor

@henningandersen henningandersen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Left two minor optional comments.

server/src/test/java/org/elasticsearch/cluster/metadata/MetaDataIndexStateServiceTests.java Show resolved Hide resolved
server/src/test/java/org/elasticsearch/cluster/metadata/MetaDataIndexStateServiceTests.java Show resolved Hide resolved
@tlrx tlrx mentioned this pull request Feb 18, 2019
Closed
50 tasks
@tlrx
Copy link
Member Author

tlrx commented Feb 18, 2019

Unrelated test failure: MlConfigMigratorIT.testConfigIndexIsCreated
@elasticmachine run elasticsearch-ci/2

@tlrx
Copy link
Member Author

tlrx commented Feb 18, 2019

Thanks @ywelsch and @henningandersen !

@tlrx tlrx merged commit c484c66 into elastic:replicated-closed-indices Feb 18, 2019
@tlrx tlrx deleted the rci-remove-index-routing-table-on-pre-8.0 branch February 18, 2019 19:21
@tlrx tlrx mentioned this pull request Feb 22, 2019
Merged
@tlrx tlrx mentioned this pull request Feb 28, 2019
Merged
tlrx added a commit that referenced this pull request Feb 28, 2019
@tlrx
Add support for replicating closed indices (#39499)
309a3e4 
Before this change, closed indexes were simply not replicated. It was therefore 
possible to close an index and then decommission a data node without knowing 
that this data node contained shards of the closed index, potentially leading to 
data loss. Shards of closed indices were not completely taken into account when 
balancing the shards within the cluster, or automatically replicated through shard 
copies, and they were not easily movable from node A to node B using APIs like 
Cluster Reroute without being fully reopened and closed again.

This commit changes the logic executed when closing an index, so that its shards 
are not just removed and forgotten but are instead reinitialized and reallocated on 
data nodes using an engine implementation which does not allow searching or
 indexing, which has a low memory overhead (compared with searchable/indexable 
opened shards) and which allows shards to be recovered from peer or promoted 
as primaries when needed.

This new closing logic is built on top of the new Close Index API introduced in 
6.7.0 (#37359). Some pre-closing sanity checks are executed on the shards before 
closing them, and closing an index on a 8.0 cluster will reinitialize the index shards 
and therefore impact the cluster health.

Some APIs have been adapted to make them work with closed indices:
- Cluster Health API
- Cluster Reroute API
- Cluster Allocation Explain API
- Recovery API
- Cat Indices
- Cat Shards
- Cat Health
- Cat Recovery

This commit contains all the following changes (most recent first):
* c6c42a1 Adapt NoOpEngineTests after #39006
* 3f9993d Wait for shards to be active after closing indices (#38854)
* 5e7a428 Adapt the Cluster Health API to closed indices (#39364)
* 3e61939 Adapt CloseFollowerIndexIT for replicated closed indices (#38767)
* 71f5c34 Recover closed indices after a full cluster restart (#39249)
* 4db7fd9 Adapt the Recovery API for closed indices (#38421)
* 4fd1bb2 Adapt more tests suites to closed indices (#39186)
* 0519016 Add replica to primary promotion test for closed indices (#39110)
* b756f6c Test the Cluster Shard Allocation Explain API with closed indices (#38631)
* c484c66 Remove index routing table of closed indices in mixed versions clusters (#38955)
* 00f1828 Mute CloseFollowerIndexIT.testCloseAndReopenFollowerIndex()
* e845b0a Do not schedule Refresh/Translog/GlobalCheckpoint tasks for closed indices (#38329)
* cf9a015 Adapt testIndexCanChangeCustomDataPath for replicated closed indices (#38327)
* b9becdd Adapt testPendingTasks() for replicated closed indices (#38326)
* 02cc730 Allow shards of closed indices to be replicated as regular shards (#38024)
* e53a9be Fix compilation error in IndexShardIT after merge with master
* cae4155 Relax NoOpEngine constraints (#37413)
* 54d110b [RCI] Adapt NoOpEngine to latest FrozenEngine changes
* c63fd69 [RCI] Add NoOpEngine for closed indices (#33903)

Relates to #33888
tlrx added a commit to tlrx/elasticsearch that referenced this pull request Mar 1, 2019
@tlrx
Remove index routing table of closed indices in mixed versions cluste…
b28acaa 
…rs (elastic#38955)

This pull request removes the legacy way of closing indices (aka "direct close") 
in mixed versions clusters, since this backward compatibility logic is not
 required anymore on master/8.0.0.

It also changes the closing logic so that routing tables of closed indices are
 removed when the cluster contains a node in version < 8.0.

Relates elastic#33888
tlrx added a commit that referenced this pull request Mar 1, 2019
@tlrx
Backport support for replicating closed indices to 7.x (#39506)(#39499)
e005eeb 
Backport support for replicating closed indices (#39499)
    
    Before this change, closed indexes were simply not replicated. It was therefore
    possible to close an index and then decommission a data node without knowing
    that this data node contained shards of the closed index, potentially leading to
    data loss. Shards of closed indices were not completely taken into account when
    balancing the shards within the cluster, or automatically replicated through shard
    copies, and they were not easily movable from node A to node B using APIs like
    Cluster Reroute without being fully reopened and closed again.
    
    This commit changes the logic executed when closing an index, so that its shards
    are not just removed and forgotten but are instead reinitialized and reallocated on
    data nodes using an engine implementation which does not allow searching or
     indexing, which has a low memory overhead (compared with searchable/indexable
    opened shards) and which allows shards to be recovered from peer or promoted
    as primaries when needed.
    
    This new closing logic is built on top of the new Close Index API introduced in
    6.7.0 (#37359). Some pre-closing sanity checks are executed on the shards before
    closing them, and closing an index on a 8.0 cluster will reinitialize the index shards
    and therefore impact the cluster health.
    
    Some APIs have been adapted to make them work with closed indices:
    - Cluster Health API
    - Cluster Reroute API
    - Cluster Allocation Explain API
    - Recovery API
    - Cat Indices
    - Cat Shards
    - Cat Health
    - Cat Recovery
    
    This commit contains all the following changes (most recent first):
    * c6c42a1 Adapt NoOpEngineTests after #39006
    * 3f9993d Wait for shards to be active after closing indices (#38854)
    * 5e7a428 Adapt the Cluster Health API to closed indices (#39364)
    * 3e61939 Adapt CloseFollowerIndexIT for replicated closed indices (#38767)
    * 71f5c34 Recover closed indices after a full cluster restart (#39249)
    * 4db7fd9 Adapt the Recovery API for closed indices (#38421)
    * 4fd1bb2 Adapt more tests suites to closed indices (#39186)
    * 0519016 Add replica to primary promotion test for closed indices (#39110)
    * b756f6c Test the Cluster Shard Allocation Explain API with closed indices (#38631)
    * c484c66 Remove index routing table of closed indices in mixed versions clusters (#38955)
    * 00f1828 Mute CloseFollowerIndexIT.testCloseAndReopenFollowerIndex()
    * e845b0a Do not schedule Refresh/Translog/GlobalCheckpoint tasks for closed indices (#38329)
    * cf9a015 Adapt testIndexCanChangeCustomDataPath for replicated closed indices (#38327)
    * b9becdd Adapt testPendingTasks() for replicated closed indices (#38326)
    * 02cc730 Allow shards of closed indices to be replicated as regular shards (#38024)
    * e53a9be Fix compilation error in IndexShardIT after merge with master
    * cae4155 Relax NoOpEngine constraints (#37413)
    * 54d110b [RCI] Adapt NoOpEngine to latest FrozenEngine changes
    * c63fd69 [RCI] Add NoOpEngine for closed indices (#33903)
    
    Relates to #33888
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ywelsch ywelsch ywelsch approved these changes

@henningandersen henningandersen henningandersen approved these changes

Assignees
No one assigned
Labels
:Distributed/Distributed A catch all label for anything in the Distributed Area. If you aren't sure, use this one. >enhancement
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@tlrx @elasticmachine @ywelsch @henningandersen