Add ability to cancel long-standing operations

[duarten]

Some operations are executed with no timeout (or equivalently, a minutes-scale timeout). These are mostly internal operations for which we've already done work that doesn't benefit from being thrown away due to a timeout, such as sending view updates.

We should add support for cancelling such operations when they are no longer relevant, for example if the failure detector marks a view replica as unavailable while there are pending view updates.

We can leverage the seastar::abort_source to implement such cancellation. In the particular case of view updates, a possible solution would be to associate a seastar::abort_source with the gms::init_address of the view replica we're updating, and request cancellation from a service::endpoint_lifecycle_subscriber when view replica is marked as unavailable.

[tzach]

@duarten does #2159 counts as such operation?

[duarten]

More or less. This is about individual, internal operations, whereas that issue is about cancelling a higher-level operation, composed of many streaming requests. Right now we can't cancel those individual streaming requests properly (I think), but using the using the mechanisms proposed they would be.

[duarten]

Another example is a view replica being restarted. Gossip can detect this through the change in generation. We should also be able to detect this, and cancel the relevant operations (i.e., pending view updates).

[duarten]

This issue is particularly important for view updates, which have a high timeout. Fixing it helps with memory consumption and decreases shutdown time in some cases. We should give it higher priority.

[nyh]

The three dtests mentioned in #4019 are a good example of why this feature is needed, and can be used to test it. We want the view-write timeout to be as high as possible, so as to allow long view-update backlogs, but because RPC currently may wait this entire duration even if the remote node goes down, a shutdown can hang for this timeout while it is waiting for all pending view writes, or an ongoing view-build step, to complete. Since the tests mentioned in #4019 only wait 2 minutes for the shutdown, they can't currently work with a timeout in storage_proxy::send_to_endpoint() higher than 2 minutes. But with the approach proposed here, we could allow a much longer timeout with the addition that if the remote node goes down (even temporarily), we immediately stop the rpc attempt and instead write a hint.

This feature (stopping instead of waiting for the full timeout) should be optional, in case somebody really needs the current behavior of continuing to wait even if the remote node goes down. But before we do that, we should verify that this waiting even make sense - is the RPC retried after the dead node goes back up? If not, continuing to wait is pointless.

[duarten]

We don't need this for #4019, we need #3966.

[nyh]

@duarten, I'm not sure what concrete proposal #3966 is making. According to @gleb-cloudius (if I understood what he said correctly...) the write that a view build is hung on in #4019 (a write which started when we didn't know the node went down) will never stop until the timeout. So we need a way to tell a write that has already started that it needs to give up the network send immediately. This is this issue, no? Then, indeed, once such a write it stops, it should indeed write to a hint as #3966 suggests, but I assumed this will happen automatically anyway - because when the network send fails, the CL=ANY should try to write a hint.

[duarten]

#3966 is about completing a shutdown in a timely fashion, regardless of nodes being down or just being slow.

We can detect that there is a shutdown in progress without enlisting the gossiper. All we need to do in the storage_proxy is to complete any outstanding writes with an exception, which for the case of view writes (but not only), will result in a hint being stored.

For this, it doesn't matter whether a node is down or just being slow: we can start a shutdown timer (of, say, 1 minute), after which we'll write pending writes as hints.