Traffic stability during rolling upgrades

[tnozicka]

This is a tracker for our epic to make sure rolling restarts are graceful and no connection gets handled ungracefully.

Unfortunately, at this point we use services that publish unready endpoints and the traffic gets errors, or gets stuck in contract tables. There are also graceful termination fixes needed for handling ScyllaDB shutdown.

Tasks

Beta Give feedback

1. [WIP] Forward ClusterIP traffic to inter-node communication port before CQL is ready #1614
   approved do-not-merge/work-in-progress kind/bug priority/important-soon size/XXL +5
   
2. Loss of availability below required consistency level during node shutdown scylladb#15899
   P3 area/correctness bug status/regression +4
   
3. During rollout restart operator doesn't wait until previously restarted pod becomes part of a Scylla cluster #1077
   kind/bug lifecycle/stale priority/important-soon +3
   

[scylla-operator-bot]

The Scylla Operator project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 30d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out

/lifecycle stale

[tnozicka]

/remove-lifecycle stale
/triage accepted