Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Non-zero shards may have stale values of gossiper application states of a restarted node for some time #3798
Application states of each node are versioned per-node with a pair of generation number (more significant) and value version. Generation number uniquely identifies the life time of the scylla process. Generation number changes after restart. Value versions start from 0 on each restart. When a node gets updates for application states, it merges them with its view on given node. Value updates with older versions are ignored.
Gossiper processes updates only on shard 0, and replicates value updates to other shards. When it sees a value with a new generation, it correclty forgets all previous values. However, non-zero shards don't forget values from previous generations. As a result, replication will fail to override the values on non-zero shards when generation number changes until their value version exceeds the version prior to the restart.
Introduced in 2d5fb9d.
Another incarnation of this issue will result in non-seeds to have their STATUS set to
When restarting a non-seed node, it will do a shadow gossip round before setting its
Note that other nodes will see their STATUS correctly as NORMAL, on all shards, becuase the change to
The problem doesn't happen when restarting a seed, because it doesn't do a shadow gossip round.