[ILM] Delete step deletes data stream with only one index #105772
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
We seem to have a couple of checks to make sure we delete the data stream when the last index reaches the delete step however, these checks seem a bit contradictory. Namely, the first check makes use if `Index` equality (UUID included) and the second just checks the index name. So if a data stream with just one index (the write index) is restored from snapshot (different UUID) we would've failed the first index equality check and go through the second check `dataStream.getWriteIndex().getName().equals(indexName)` and fail the delete step (in a non-retryable way :( ) because we don't want to delete the write index of a data stream (but we really do if the data stream has only one index) This PR makes 2 changes: 1. use the index name equality everywhere in the step (we already looked up the index abstraction and the parent data stream, so we know for sure the managed index is part of the data stream) 2. do not throw exception when we got here via a write index that is NOT the last index in the data stream but report the exception so we keep retrying this step (i.e. this enables our users to simply execute a manual rollover and the index is deleted by ILM eventually on retry)
andreidan
added
>bug
:Data Management/ILM+SLM
|
Pinging @elastic/es-data-management (Team:Data Management) |
elasticsearchmachine
added
Team:Data Management
|
Hi @andreidan, I've created a changelog YAML for you. |
|
@elasticmachine update branch |
dakrone
approved these changes
Mar 2, 2024
| @@ -41,7 +41,8 @@ public void performDuringNoSnapshot(IndexMetadata indexMetadata, ClusterState cu | |||
|
|
|||
| if (dataStream != null) { | |||
| assert dataStream.getWriteIndex() != null : dataStream.getName() + " has no write index"; | |||
| if (dataStream.getIndices().size() == 1 && dataStream.getIndices().get(0).equals(indexMetadata.getIndex())) { | |||
|
|
|||
| if (dataStream.getIndices().size() == 1 && dataStream.getWriteIndex().getName().equals(indexName)) { | |||
There was a problem hiding this comment.
Can you add a comment about why we use name equality here so it doesn't get accidentally changed back? (I know we have tests, but it's still easy to stop someone from wasting work)
| } | ||
|
|
||
| @Override | ||
| public void onFailure(Exception e) { |
There was a problem hiding this comment.
I think that now that this isn't throwing directly, we need to have a latch or other mechanism to ensure that the onFailure handler was actually invoked. Otherwise if we were to introduce a bug where neither onResponse nor onFailure were called, then we wouldn't hit any asserts and the test would pass (when it shouldn't).
There was a problem hiding this comment.
The execution in this particular case is not async as we're not getting to the callback as part of a client interaction - this is all sync as it's part of the step validation.
I've stubbed the client to fail in case it's being called in this test so that changes to the ILM step structure will yield this test to fail.
There was a problem hiding this comment.
Ah I (later) understood what you meant - opened #105914 to make sure we fail the test if the listener is not called at all
|
@elasticmachine update branch |
andreidan
added
auto-merge
andreidan
added a commit
to andreidan/elasticsearch
that referenced
this pull request
Mar 4, 2024
…5772) We seem to have a couple of checks to make sure we delete the data stream when the last index reaches the delete step however, these checks seem a bit contradictory. Namely, the first check makes use if `Index` equality (UUID included) and the second just checks the index name. So if a data stream with just one index (the write index) is restored from snapshot (different UUID) we would've failed the first index equality check and go through the second check `dataStream.getWriteIndex().getName().equals(indexName)` and fail the delete step (in a non-retryable way :( ) because we don't want to delete the write index of a data stream (but we really do if the data stream has only one index) This PR makes 2 changes: 1. use the index name equality everywhere in the step (we already looked up the index abstraction and the parent data stream, so we know for sure the managed index is part of the data stream) 2. do not throw exception when we got here via a write index that is NOT the last index in the data stream but report the exception so we keep retrying this step (i.e. this enables our users to simply execute a manual rollover and the index is deleted by ILM eventually on retry)
💚 Backport successful
|
elasticsearchmachine
pushed a commit
that referenced
this pull request
Mar 4, 2024
…105897) We seem to have a couple of checks to make sure we delete the data stream when the last index reaches the delete step however, these checks seem a bit contradictory. Namely, the first check makes use if `Index` equality (UUID included) and the second just checks the index name. So if a data stream with just one index (the write index) is restored from snapshot (different UUID) we would've failed the first index equality check and go through the second check `dataStream.getWriteIndex().getName().equals(indexName)` and fail the delete step (in a non-retryable way :( ) because we don't want to delete the write index of a data stream (but we really do if the data stream has only one index) This PR makes 2 changes: 1. use the index name equality everywhere in the step (we already looked up the index abstraction and the parent data stream, so we know for sure the managed index is part of the data stream) 2. do not throw exception when we got here via a write index that is NOT the last index in the data stream but report the exception so we keep retrying this step (i.e. this enables our users to simply execute a manual rollover and the index is deleted by ILM eventually on retry)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We seem to have a couple of checks to make sure we delete the data stream when the last index reaches the delete step however, these checks seem a bit contradictory.
Namely, the first check makes use if
Indexequality (UUID included) and the second just checks the index name.So if a data stream with just one index (the write index) is restored from snapshot (different UUID) we would've failed the first index equality check and go through the second check
dataStream.getWriteIndex().getName().equals(indexName)and fail the delete step (in a non-retryable way :( ) because we don't want to delete the write index of a data stream (but we really do if the data stream has only one index)This PR makes 2 changes: