Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

IndicesService handles all exceptions during index deletion #22433

Merged

Conversation

Projects
None yet
4 participants
@abeyad
Copy link
Contributor

commented Jan 4, 2017

Previously, we could run into a situation where attempting to delete an
index due to a cluster state update would cause an unhandled exception
to bubble up to the ClusterService and cause the cluster state applier
to fail. The result of this situation is that the cluster state never
gets updated on the ClusterService because the exception happens before
all cluster state appliers have completed and the ClusterService only
updates the cluster state once all cluster state appliers have
successfully completed.

All other methods on IndicesService properly handle all exceptions and
not just IOExceptions, but there were two instances with respect to
index deletion where only IOExceptions where handled by the
IndicesService. If any other exception occurred during these delete
operations, the exception would be bubbled up to the ClusterService,
causing the aforementioned issues.

This commit ensures all methods in IndicesService properly capture all
types of Exceptions, so that the ClusterService manages to update the
cluster state, even in the presence of shard creation/deletion failures.

Note that the lack of updating the cluster state in the presence of such
exceptions can have many unintended consequences, one of them being
the tripping of the assertion in IndicesClusterStateService#removeUnallocatedIndices
where the assumption is that if there is an IndexService to remove with
an unassigned shard, then the index must exist in the cluster state, but if
the cluster state was never updated due to the aforementioned exceptions,
then the cluster state will not have the index in question.

Ali Beyad
IndicesService handles all exceptions during index deletion
Previously, we could run into a situation where attempting to delete an
index due to a cluster state update would cause an unhandled exception
to bubble up to the ClusterService and cause the cluster state applier
to fail.  The result of this situation is that the cluster state never
gets updated on the ClusterService because the exception happens before
all cluster state appliers have completed and the ClusterService only
updates the cluster state once all cluster state appliers have
successfully completed.

All other methods on IndicesService properly handle all exceptions and
not just IOExceptions, but there were two instances with respect to
index deletion where only IOExceptions where handled by the
IndicesService.  If any other exception occurred during these delete
operations, the exception would be bubbled up to the ClusterService,
causing the aforementioned issues.

This commit ensures all methods in IndicesService properly capture all
types of Exceptions, so that the ClusterService manages to update the
cluster state, even in the presence of shard creation/deletion failures.
@ywelsch

ywelsch approved these changes Jan 4, 2017

Copy link
Contributor

left a comment

LGTM

@jasontedor
Copy link
Member

left a comment

LGTM.

@abeyad

This comment has been minimized.

Copy link
Contributor Author

commented Jan 4, 2017

retest this please

@abeyad abeyad merged commit 9d422c1 into elastic:master Jan 4, 2017

2 checks passed

CLA Commit author is a member of Elasticsearch
Details
elasticsearch-ci Build finished.
Details

@abeyad abeyad deleted the abeyad:fix/delete_unassigned_index_exception_handling branch Jan 4, 2017

@abeyad

This comment has been minimized.

Copy link
Contributor Author

commented Jan 4, 2017

thanks @ywelsch and @jasontedor

abeyad pushed a commit that referenced this pull request Jan 4, 2017

Ali Beyad
IndicesService handles all exceptions during index deletion (#22433)
Previously, we could run into a situation where attempting to delete an
index due to a cluster state update would cause an unhandled exception
to bubble up to the ClusterService and cause the cluster state applier
to fail. The result of this situation is that the cluster state never
gets updated on the ClusterService because the exception happens before
all cluster state appliers have completed and the ClusterService only
updates the cluster state once all cluster state appliers have
successfully completed.

All other methods on IndicesService properly handle all exceptions and
not just IOExceptions, but there were two instances with respect to
index deletion where only IOExceptions where handled by the
IndicesService. If any other exception occurred during these delete
operations, the exception would be bubbled up to the ClusterService,
causing the aforementioned issues.

This commit ensures all methods in IndicesService properly capture all
types of Exceptions, so that the ClusterService manages to update the
cluster state, even in the presence of shard creation/deletion failures.

Note that the lack of updating the cluster state in the presence of such
exceptions can have many unintended consequences, one of them being
the tripping of the assertion in IndicesClusterStateService#removeUnallocatedIndices
where the assumption is that if there is an IndexService to remove with
an unassigned shard, then the index must exist in the cluster state, but if
the cluster state was never updated due to the aforementioned exceptions,
then the cluster state will not have the index in question.
@abeyad

This comment has been minimized.

Copy link
Contributor Author

commented Jan 4, 2017

5.x commit: 84596fa

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.