-
Notifications
You must be signed in to change notification settings - Fork 24.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Clean up ShardFollowTasks for deleted indices #44702
Clean up ShardFollowTasks for deleted indices #44702
Conversation
Pinging @elastic/es-distributed |
@elasticmachine run elasticsearch-ci/1 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for picking this up. I left a suggestion.
x-pack/plugin/ccr/src/main/java/org/elasticsearch/xpack/ccr/action/ShardFollowTaskCleaner.java
Show resolved
Hide resolved
Thanks @jasontedor, I applied your feedback. Can you please have another look? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM.
Can you apply this to 6.8 as well? |
Thanks @jasontedor. I'll backport this change down the line to 6.8. |
Deleting a follower index does not delete its ShardFollowTasks, potentially leaving many persistent tasks in the cluster that cannot be allocated on nodes and unnecessary fill the logs. This commit adds a cluster state listener (ShardFollowTaskCleaner) that completes (with a failure) any persistent task that refers to a non existent follower index. I think that this bug has been introduced by elastic#34404: before this change the task would have been completed as failed and removed from the cluster state.
Deleting a follower index does not delete its ShardFollowTasks, potentially leaving many persistent tasks in the cluster that cannot be allocated on nodes and unnecessary fill the logs. This commit adds a cluster state listener (ShardFollowTaskCleaner) that completes (with a failure) any persistent task that refers to a non existent follower index. I think that this bug has been introduced by elastic#34404: before this change the task would have been completed as failed and removed from the cluster state.
…ex (#44801) This commit unmutes and renames the test that failed on CI (#44796) after #44702 has been merged. This test assumes that follow stats still exist after a follower index has been deleted. The follow stats are based on persistent tasks, and since #44702 the persistent tasks of deleted following indices are now automatically cleaned up to avoid to bloat the cluster state. I don't think we should report any follow stats for deleted indices and I don't think that this test makes much sense now the tasks are cleaned up. This is why the test has been renamed. closes #44796
…ex (elastic#44801) This commit unmutes and renames the test that failed on CI (elastic#44796) after elastic#44702 has been merged. This test assumes that follow stats still exist after a follower index has been deleted. The follow stats are based on persistent tasks, and since elastic#44702 the persistent tasks of deleted following indices are now automatically cleaned up to avoid to bloat the cluster state. I don't think we should report any follow stats for deleted indices and I don't think that this test makes much sense now the tasks are cleaned up. This is why the test has been renamed. closes elastic#44796
…ex (elastic#44801) This commit unmutes and renames the test that failed on CI (elastic#44796) after elastic#44702 has been merged. This test assumes that follow stats still exist after a follower index has been deleted. The follow stats are based on persistent tasks, and since elastic#44702 the persistent tasks of deleted following indices are now automatically cleaned up to avoid to bloat the cluster state. I don't think we should report any follow stats for deleted indices and I don't think that this test makes much sense now the tasks are cleaned up. This is why the test has been renamed. closes elastic#44796
…ex (elastic#44801) This commit unmutes and renames the test that failed on CI (elastic#44796) after elastic#44702 has been merged. This test assumes that follow stats still exist after a follower index has been deleted. The follow stats are based on persistent tasks, and since elastic#44702 the persistent tasks of deleted following indices are now automatically cleaned up to avoid to bloat the cluster state. I don't think we should report any follow stats for deleted indices and I don't think that this test makes much sense now the tasks are cleaned up. This is why the test has been renamed. closes elastic#44796
Deleting a follower index does not delete its ShardFollowTasks, potentially leaving many persistent tasks in the cluster that cannot be allocated on nodes and unnecessary fill the logs. This commit adds a cluster state listener (ShardFollowTaskCleaner) that completes (with a failure) any persistent task that refers to a non existent follower index. I think that this bug has been introduced by #34404: before this change the task would have been completed as failed and removed from the cluster state. Backport of #44702 and #44801 on 7.x
Deleting a follower index does not delete its ShardFollowTasks, potentially leaving many persistent tasks in the cluster that cannot be allocated on nodes and unnecessary fill the logs. This commit adds a cluster state listener (ShardFollowTaskCleaner) that completes (with a failure) any persistent task that refers to a non existent follower index. I think that this bug has been introduced by #34404: before this change the task would have been completed as failed and removed from the cluster state. Backport of #44702 and #44801 to 7.3
Deleting a follower index does not delete its ShardFollowTasks, potentially leaving many persistent tasks in the cluster that cannot be allocated on nodes and unnecessary fill the logs. This commit adds a cluster state listener (ShardFollowTaskCleaner) that completes (with a failure) any persistent task that refers to a non existent follower index. I think that this bug has been introduced by #34404: before this change the task would have been completed as failed and removed from the cluster state. Backport of #44702 and #44801 on 6.8
Deleting a follower index does not delete its
ShardFollowTask
s, potentially leaving many persistent tasks in the cluster that cannot be allocated on nodes and unnecessary fill the logs. This pull request adds a cluster state listener (ShardFollowTaskCleaner
) that completes (with a failure) any persistent task that refers to a non existent follower index.I think that this bug has been introduced by #34404: before this change the task would have been completed as failed and removed from the cluster state.