Remove indexing_complete when removing policy
#36620
Conversation
Leaving `index.lifecycle.indexing_complete` in place when removing the lifecycle policy from an index can cause confusion, as if a new policy is associated with the policy, rollover will be silently skipped. Removing that setting when removing the policy from an index makes associating a new policy with the index more involved, but allows ILM to fail loudly, rather than silently skipping operations which the user may assume are being performed.
gwbrown
added
>bug
:Data Management/ILM+SLM
|
Pinging @elastic/es-core-features |
|
@martijnvg I requested your review here partially to double-check my thinking here: This setting will be used (in combination with other signals) to indicate to ILM that it is safe to de-follow a following index. I believe the only case when removing this setting when removing a policy would have an impact is if the policy is removed from an index after it rolls over (and so is actually finished indexing), but a follower has not yet caught up (and therefore has not yet de-followed the index) before the policy is removed. In this case, the following index will not have this setting set, and so be stuck waiting for this setting to be set. The fix is simply to set this setting manually. Correct? |
...ugin/ilm/src/test/java/org/elasticsearch/xpack/indexlifecycle/IndexLifecycleRunnerTests.java
Show resolved
Hide resolved
This allows ILM to error out properly for indices that have a valid alias, but are not the write index, while still handling `indexing_complete` on old-style aliases and rollover (that is, those which only point to a single index at a time with no explicit write index)
|
@colings86 I've added the test you asked for. In the process of writing the test, I ran across a bug: Instead of moving to the error step at https://github.com/elastic/elasticsearch/pull/36620/files#diff-5de660b10fd8979ff41a9ff8d850aaecR569, execution of the policy for this index would start checking for rollover, because we didn't check that the index was actually the write index for the alias. I've added that check in Because there's now a slight difference in how those are handled, I've tweaked It might be worth pulling those changes out into a separate PR, but this PR won't pass without those changes. This is likely worth discussing when you have time. Also, big thanks to @talevy for helping me figure out the alias gymnastics here! |
...-node/src/test/java/org/elasticsearch/xpack/indexlifecycle/TimeSeriesLifecycleActionsIT.java
Show resolved
Hide resolved
|
@elasticmachine run the gradle build tests 2 please |
Leaving `index.lifecycle.indexing_complete` in place when removing the lifecycle policy from an index can cause confusion, as if a new policy is associated with the policy, rollover will be silently skipped. Removing that setting when removing the policy from an index makes associating a new policy with the index more involved, but allows ILM to fail loudly, rather than silently skipping operations which the user may assume are being performed. * Adjust order of checks in WaitForRolloverReadyStep This allows ILM to error out properly for indices that have a valid alias, but are not the write index, while still handling `indexing_complete` on old-style aliases and rollover (that is, those which only point to a single index at a time with no explicit write index)
Leaving `index.lifecycle.indexing_complete` in place when removing the lifecycle policy from an index can cause confusion, as if a new policy is associated with the policy, rollover will be silently skipped. Removing that setting when removing the policy from an index makes associating a new policy with the index more involved, but allows ILM to fail loudly, rather than silently skipping operations which the user may assume are being performed. * Adjust order of checks in WaitForRolloverReadyStep This allows ILM to error out properly for indices that have a valid alias, but are not the write index, while still handling `indexing_complete` on old-style aliases and rollover (that is, those which only point to a single index at a time with no explicit write index)
Leaving `index.lifecycle.indexing_complete` in place when removing the lifecycle policy from an index can cause confusion, as if a new policy is associated with the policy, rollover will be silently skipped. Removing that setting when removing the policy from an index makes associating a new policy with the index more involved, but allows ILM to fail loudly, rather than silently skipping operations which the user may assume are being performed. * Adjust order of checks in WaitForRolloverReadyStep This allows ILM to error out properly for indices that have a valid alias, but are not the write index, while still handling `indexing_complete` on old-style aliases and rollover (that is, those which only point to a single index at a time with no explicit write index)
Leaving
index.lifecycle.indexing_completein place when removing thelifecycle policy from an index can cause confusion, as if a new policy
is associated with the policy, rollover will be silently skipped.
Removing that setting when removing the policy from an index makes
associating a new policy with the index more involved, but allows ILM to
fail loudly, rather than silently skipping operations which the user may
assume are being performed.
Context: After synchronous discussion earlier today, @talevy, @eskibars, and I decided that failing loudly when we might possibly be able to succeed is better than being silently surprising, especially when the fix is relatively straightforward.