Allow a snapshot to be deleted if a restore for a different snapshot is in progress

[bpiper]

Currently, according to SnapshotsService.deleteSnapshot, a snapshot deletion request is rejected if there is any restore in progress. The reason given is commented thusly:

// don't allow snapshot deletions while a restore is taking place,
// otherwise we could end up deleting a snapshot that is being restored
// and the files the restore depends on would all be gone

This makes perfect sense, however it doesn't necessarily explain why I can't delete snapshot A while snapshot B is being restored (which currently is forbidden). Would it not be feasible to just check if the snapshot we want to delete is among those snapshots with a restore in progress and allow the deletion if it is not, or is there more to it than the comment suggests?

In my use case, I have a 1-to-1 relationship between indices and snapshots, and being able to restore multiple indices in parallel now is great, but as I also need to be able to delete snapshots after I've restored them (for housekeeping purposes), or indeed just delete them along with the index if I no longer need it, not being able to do so while potentially numerous other indices are being restored (something that is driven by users, not me) is a bit of a drag.

[elasticmachine]

Pinging @elastic/es-distributed

[DaveCTurner]

@bpiper to clarify the comment a little, there is a many-to-many relationship between snapshots and the actual objects in a repository. Each snapshot contains many objects (maybe spread across many indices) and each object may belong to more than one snapshot. This means that it's not so simple to determine whether a restore conflicts with a concurrent deletion or not.

Relates the conversation surrounding #27353 (comment), particularly the observation that there remain significant obstacles to doing this correctly.

[bpiper]

Thanks for clarifying @DaveCTurner. I had suspected that might be the case, and I appreciate this is probably stretching the snapshot/restore system a bit beyond its original intent. I think I can live with snapshot deletion failures for now, though if this fruit ever moves down a few branches, it would definitely be great to have a bit more flexibility.