Check again on-going snapshots/restores of indices before closing

[tlrx]

Today we prevent any index that is actively snapshotted or restored to be closed. This verification is done during the execution of the first phase of index closing (ie before blocking the indices).

We should also do this verification again in the last phase of index closing (ie after the shard sanity checks and right before actually changing the index state and the routing table) because a snapshot/restore could sneak in while the shards are verified-before-close.

[elasticmachine]

Pinging @elastic/es-distributed

[tlrx]

@elasticmachine update branch

[original-brownbear]

Spoke about this with @tlrx just now and had one concern:

I wondered if it wouldn't be better to prevent a new snapshot (i.e. just bail because closing is in progress for an index that should be snapshotted) from being started for the affected indices between the two phases of closing.

It seems to me that that would:

1. Be simpler in terms of the state transitions (no redundant failing to close step and subsequent cleanup step on concurrent snapshots)
2. For users running back-to-back snapshots of the whole cluster, a situation could arise where closing an index becomes effectively impossible because a concurrent snapshot start could keep interrupting the close operation over and over (if running the snapshot in a loop ... which some users do apparently).

[ywelsch]

I wondered if it wouldn't be better to prevent a new snapshot (i.e. just bail because closing is in progress for an index that should be snapshotted) from being started for the affected indices between the two phases of closing.

The main problem I see with this is that if an index is not successfully closed (but the closed block still left behind), no new snapshots can be done, as we can't distinguish between "is between the two phases of closing" and "closing failed and left closed block behind". This is trappy as it will put the cluster at risk of data loss, and requires manual intervention to fix. I would rather prefer the original solution.

[original-brownbear]

@ywelsch makes sense to me with the way things are, @tlrx said the same :) Maybe one question just for my own education below (whenever you have a second):

as we can't distinguish between "is between the two phases of closing" and "closing failed and left closed block behind".

Isn't the latter a bug (the fact that we just leave the block behind when closing failed) to be resolved by removing the close block when closing failed? (or do we not want to do that to prevent further indexing to the index?)

[original-brownbear]

LGTM :) just a suggestion on the ex. messages that seem to break our style in some spots.

[ywelsch]

as we can't distinguish between "is between the two phases of closing" and "closing failed and left closed block behind".

Isn't the latter a bug (the fact that we just leave the block behind when closing failed) to be resolved by removing the close block when closing failed? (or do we not want to do that to prevent further indexing to the index?)

There is currently no automated internal retry (we might do that at some point), but the user intent was clearly to close the index, so leaving the block behind should not be surprising (in case the user tries to index into the index, he will get a proper exception message saying the index might be in the process of being closed and that he can remove the block by reopening the index, or that he can issue the close command again). By keeping the block in place, a follow-up close is also more likely to succeed as it can just reuse the block while stepping through the phases. If you would auto-remove the block, two concurrent close operations might be problematic as they would interfere with each other. The close operation is currently idempotent, also in failure cases.

[ywelsch]

I've added the 7.3.0 label here as it's a bugfix.

[tlrx]

@elasticmachine update branch

[tlrx]

Thanks @original-brownbear !