During rollout restart operator doesn't wait until previously restarted pod becomes part of a Scylla cluster #1077
Comments
|
We wait for readiness, I wonder if this is the case where it becomes ready without having a quorum. |
tnozicka
added
the
priority/important-soon
|
@vponomaryov do we a run of this with scylla-core latest master ? |
I didn't run it with master. Only with the specified versions in the bug report. |
|
Can we run it at least once with master version ? And then we should try having a dtest reproducer of it. our rolling restart process shouldn't be that brittle. |
The error is expected because a Scylla node always becomes |
|
We can slow down traffic between two nodes |
The 'test_nodetool_flush_and_reshard' test fails due to the following bug: scylladb/scylla-operator#1077 So, skip it.
|
The probability to hit this bug is very high. |
The 'test_nodetool_flush_and_reshard' test is affected by the following
bug:
scylladb/scylla-operator#1077
using non-fast K8S backends such as 'k8s-gke' and 'k8s-local-kind'.
So, skip it on the slow backends.
The 'test_nodetool_flush_and_reshard' test is affected by the following
bug:
scylladb/scylla-operator#1077
using non-fast K8S backends such as 'k8s-gke' and 'k8s-local-kind'.
So, skip it on the slow backends.
|
@tnozicka - I'm unsure if our logic should not be converted to 'ask other nodes what node X status is' instead of 'ask node X what its status is' ... |
Operator's racy bug [1] gets reproduced on slow backends and nodes with small number of cores. So, skip it for the nodes that are not running on the EKS and have less than 14 cores. [1] scylladb/scylla-operator#1077
|
Yeah, currently readiness project ability to talk CONSISTENCY_ONE and we are talking about migrating it project CONSISTENCY_QUORUM. @rzetelskik can you summarize the update of your discoveries we talked about today? |
Operator's racy bug [1] gets reproduced on slow backends and nodes with small number of cores. So, skip it for the nodes that are not running on the EKS and have less than 14 cores. [1] scylladb/scylla-operator#1077
@rzetelskik @tnozicka can one of you update on this one, and shed some light on what's "project CONSISTENCY_QUORUM" in this context, cause I don't understand what does it mean... |
Sure. In an effort to reproduce this I've added a simple coroutine to our E2E test suite that'd just send read queries periodically with quorum consistency and it caused the suite to fail quite consistently, so I haven't yet followed up on why this only kept happening on GKE for you guys, since the issue seems more general anyway. Our readiness probe only checks whether the node considers itself UN. We've hit issues related to it and discussed it many times so that's nothing new. By the time a node declares readiness, the gossip should've settled, although nothing stops it from settling with a split brain, so it's quite easy to hit the following sequence:
At this point you'll get errors from queries with quorum consistency regardless of which node you hit. So we've discussed different approaches to how we could modify the readiness probe or the operator to get closer to what the documentation advises, i.e. all nodes UP before performing a topology change. The current idea suggested by @tnozicka is that each node should ask all of its peers about statuses of their peers and only declare readiness if all of them consider at least N-1 of their peers UN, where N is the total numbers of peers.
I think what Tomas's message meant is that currently the readiness probe only declares that a given node is able to serve queries with consistency ONE (which is only true if that node replicates the data in question) and at that point we were discussing implementing a heuristic where a node would check whether a quorum of its peers is UN. |
Issue description
It happens not only when changing number of cores. The same situation happened on GKE typical rolling restart. just before cdc errors node-1 restarted: ImpactCluster is not operable for CL=QUORUM when RF=3 How frequently does it reproduce?During this run we had only one RollingRestart nemesis and it failed. Observed on GKE. Installation detailsKernel Version: 5.15.0-1023-gke Operator Image: scylladb/scylla-operator:latest Scylla Nodes used in this run: OS / Image: Test: Logs and commands
Logs:
|
|
during testing of operator 1.11, I've hit in multiple cases both on GKE where we see it quite often, and on EKS longevity (in 12h run, and multi tenancy) |
|
thanks, we are having some architectural discussions about what we could do for v1.12, so this is not forgotten but we have limited bandwidth with everything else going on |
|
I ran experiments on GKE with minReadySeconds on StatefulSets - in big shortcut it's a sleep between updating next node, on different setups. Each setup consisted of 3 unbound loaders sending read requests continuously using gocql driver with default config. After cluster was fully up and running, test triggered a rolling restart. Traffic was stopped 5s after restart was completed.
Discrepancy between ClusterIP and PodIP results might suggest that kube-proxy which provides ClusterIP overlay might cause most of the failures. To verify whether it's the case, I repeated the tests on GKE with Dataplane V2 (Cillium managed by Google) where kube-proxy is not present.
Results shows Operator is not able to provide 100% success rate in any setup even when minReadySeconds is high. Although setting it to 60s would help a lot on default ClusterIP configuration while not influencing bootstrap time that much. Simillar results were present when traffic was non-TLS. |
|
@zimnx good job with getting all these results. Were you able to find out the mode that kube-proxy is running in in the default scenario? Given the results I'd assume it's iptables but we need to check. Also, coming back to what the original issue says:
It was over a year ago, so it might have changed, but @vponomaryov do you modify the networking stack in EKS anyhow, or is it vanilla? |
|
|
After my last experiment, I aggregated the same types of errors. Most of the errors were unavailability errors and I found out that in setups where number of total errors is low, almost all errors shows up when a node is tearing down. I looked up Scylla documentation about how nodes should be teared down, and we have a mismatch. Changing PreStopHook to This unfortunately didn't solve traffic disruption on ClusterIP setups where errors were showing more than 1 node being down. This meant there's a split brain in gossip state. Looking at multiple nodetool status of all nodes confirmed it. I initially thought that maybe kube-proxy lags and iptables are not updated fast enough, but I ruled it out as experiments showed Service ClusterIP mappings in iptables are updated right after Pod is recreated with new PodIP. Scylla keeps 4 connections between each shard and node on 7000 port used for inter-node communication. One of them is used for gossip. These connections are lazily initialized so it might be that there're less than 4 connections if one was dropped and there wasn't anyone needing one. If we look at stable 2 node ClusterIP cluster where each node have 1 shard, we can see existing connections and their mapping using conntrack. There're 3 connections from pod-0, to svc-1 with correct NATed address of pod-1, and 4 connections from pod-1 to svc-0 with correct NATed address of pod-0. When pod-1 is being deleted, conntrack shows multiple attempts where pod-0 is trying to reconnect to svc-1 but fails which is expected. Between when pod-1 was deleted, and new one recreated, pod-0 tried to reconnect but traffic was blackholed, meaning this SYN is lost, and this session needs to timeout. After SYN_SENT sessions expired, there were no sessions from pod-0 to svc-1, only pod-1 to svc-0: Eventually pod-0 tried to reconnect and succeeded with correct pod-1 IP: Looks like SYN_SENT sessions are the root cause of our gossip brainsplit as nodes cannot establish session to node being restarted. They report UN, as they are able to talk to others, while others are stuck on connection. Immediately after these stuck session expire, new sessions are established and gossip state synchronizes. On GKE, maximum SYN packets are retransmissed 6 times, meaning from first SYN to failure it can take 127s, and then Scylla needs to reconnect which can take another couple of seconds and then exchange gossip. To solve it we have several options:
|
|
@zimnx thanks for diving into it. I wonder how many more times conntrack is going to bite us. 😛
Just to understand it, could you explain the calculation here?
Remember that such workaround would require running with additional linux capabilities. Looking at the conntrack cleaner daemonset (9a63e01), which we used in our CI to resolve a similar issue in the past (#971), it would require NET_ADMIN. I also wonder if that wouldn't affect stability of otherwise healthy clusters.
I recall you mentioned 180s not being a value high enough to completely rid us of errors, which is quite surprising now given the 127s above. Was that caused by a higher syn_sent conntrack timeout?
It's worth explicitly stating that the issue comes from running kube-proxy depending on netfilter's conntrack. From my understanding at this point we're not sure if this is specific to running kube-proxy in iptables mode, or if that also occurs in ipvs mode. @zimnx have you tried it? Anyway, an approach that gets rid of the issue without us implementing any workarounds is to advise our users to remove the netfilter dependency if they require exposing their nodes on ClusterIPs. Like you said above, you haven't hit the issue while running in GKE with Dataplane V2 (kube-proxy-less Cillium). |
https://www.kernel.org/doc/Documentation/networking/ip-sysctl.txt
Our nodesetup pods are already running as root, so no extra permissions are required.
Removing only SYN_SENT entries to clusterIPs and Scylla ports after configurable and reasonable timeout should only cause more reconnection attempts.
That's what I plan to look into next, maybe there's something else causing disruption.
Nope
I haven't tried with different node teardown logic. It's something I want to tryout later as well. |
Doesn't work, this setting needs to be changed on node, not in container network namespace. |
|
I added setting of conntrack TCP timeout of SYN_SENT entries to our node setup DaemonSet, it solved availability issues on both Cluster and PodIP settings without minReadySeconds.
We no longer see any availability errors meaning Scylla rolls out without loosing Quorum. Since we found configuration where we no longer observe any availability issues, I verified how different Scylla versions behave.
Versions >=5.1 cause request failures during when node is shutting down. Changing
Looks like setting |
|
@zimnx could we make it an optional feature in node setup? The users would then have a choice between running on PodIPs without any issues, using this workaround, or running a kube-proxy-less cluster, which sounds like reasonably many choices. |
|
Issues in scylla regarding loss of availability during node shutdown on newer versions and rpc_client connection timeout |
|
Great job drilling down into this! I don't think we want to get into managing contrack settings for a subset of platforms in node setup DS. It seems fragile, potentially affecting other workloads and quite platform dependent. Scylla timeout seems like more fitting and stable place even if we have to wait a bit. |
|
Are we sure shutdown is not hung, which causes RPC connections not to close properly, from both sides? |
|
@xemul looked at multiple RPC issues recently, may be he has some idea. |
Rollout proceeds, process dies some time after we drain it and kill it without need for ungraceful shutdown. |
|
After couple of days I have doubts whether timeout on rpc_client connection would solve the issue. I think, it would only postpone fixing the real issue. The reason why timeout would help us, is because it would give nodes another attempt to connect to restarted node. New attempt usually means new source port, session would have new ip/port tuple which requires iptables traversal. Effectively using updated ClusterIP mapping. And this is what fixes the issue we hit with ClusterIP clusters.
We can force connection timeout by namespaced Looks like even with configurable timeout, we wouldn't be able to survive rollouts without availability loss, because UN information from single node is not enough. Seems like one of the first idea about asking multiple nodes about status of ourselves would be a good way to solve these availability issues. Together with recommendation to set additional sysctls they should make readiness pass quick enough. |
We won't. |
Describe the bug
If we change the number of cores for Scylla (it will trigger rollout pod restart) then we start facing following racy condition:
First pod gets restarted and if it gets into a cluster with some unexpected delay then operator doesn't wait for it and restarts second pod.
In a cluster of 3 nodes it makes cluster be inoperational not having quorum for several minutes.
It happens on GKE and doesn't happen on EKS because in the latter one case a restarted pod gets into a cluster much faster.
Below are the logs from a run on GKE cluster.
pod-2(the first restarted) logs:pod-1logs (second restarted):So, from the logs above we can see that the time when
pod-1(second restarted) gotshutdownmessage the first onewas not ready yet. The appeared
quorumerror from CDC is the proof for it.To Reproduce
Steps to reproduce the behavior:
scyllaclusterspecExpected behavior
Operator should wait for the previous pod return back to a Scylla cluster before restarting next one.
Logs
K8S logs: https://cloudius-jenkins-test.s3.amazonaws.com/0512c157-c4d7-4d3f-9b44-2bcce9d34de9/20221027_191640/kubernetes-0512c157.tar.gz
DB logs: https://cloudius-jenkins-test.s3.amazonaws.com/0512c157-c4d7-4d3f-9b44-2bcce9d34de9/20221027_191640/db-cluster-0512c157.tar.gz
Environment:
The text was updated successfully, but these errors were encountered: