Allow turning on TCP keepalive for specific sessions/connections

[psarna]

Context: scylladb/scylladb#7781 (comment)

It can be useful for clients to decide that connections created within a session should/should not be subject to TCP timeouts.

[psarna]

Depends on tokio-rs/tokio#3082 - set_keepalive was dropped from Tokio during the 0.2 -> 0.3 transition.

[piodul]

Leaving a comment with similar insight to the one I left under #394:

We could implement a keepalive now without waiting for tokio to support TCP-level keepalives if we take a similar approach as Java and Python drivers do, namely send a cheap query every keepalive period, e.g. select key from system.local where key = 'local'.

[elbaro]

If I want to manually send a cheap query until this is implemented, how can I make sure every connection in the pool is refreshed?

For example, I have 3 nodes, the default connection pool size is 4. If sending every keepalive_period / 12, does the default token-aware round-robin policy will make sure all 12 tcp connections are refreshed?

[psarna]

@elbaro it won't be that easy on Scylla, because our driver is also shard-aware, and we try to send data to a specific shard if possible, and also your queries would have to be hand-crafted to hit proper tokens, so that each connection is used eventually (or make a query that's not token-aware, e.g. an unprepared one, but these ones are more expensive). @havaker is working on an internal solution that allows configuring the keepalive period right now, and the solution will make sure that each connection is kept alive if it's configured. Hopefully we'll be able to give you a branch for testing today/tomorrow.

[piodul]

I'm reopening this. The #395 implemented CQL-level keepalives which is useful but is not the same as TCP-level keepalives. Some drivers expose both (e.g. cpp driver: https://github.com/scylladb/cpp-driver/blob/eb6b70d138752c5e6a487eeaf69e6baa4d6652a4/include/cassandra.h#L2533-L2536).

According to this comment, it is possible to set TCP keepalives on a tokio socket, although with the help of an external crate.