-
Notifications
You must be signed in to change notification settings - Fork 3.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
kv: lowering the range size can cause unavailability #46271
Comments
We have some options here: A) Disable all size-based back pressure. In the course of increasing the range size we've explored problems with ranges getting too large. We've now ensured that O(range size) operations no longer buffer anything in RAM and generally carry timeouts proportional to that size.
Anybody have preferences or other ideas? |
How about if we did C, but really crudely - we gossiped something any time a range size is decreased. And then we don't backpressure for 24h after such a change (anywhere in the cluster). |
The problem with this approach and all like it is that it won't survive a node restart. We don't need to gossip anything separately; we could hook the disablement into the update to the zone config itself. D) (from @tbg) Disable backpressure if the range is strictly larger than the size at which backpressure would have started. As in, “don’t backpressure if backpressure wouldn’t have let the range get its current size in the first place.” This option is pretty compelling to me. It seems like generally it should mitigate the problem. |
D) sounds kinda fragile because only some writes are backpressured? So if it's a non-backpressured write happens to take the range over the limit, than the range gets to run wild? |
If we did it such that a single byte would change the behavior then you're right. Perhaps we do something like we backpressure if the current range size would become some amount larger than the current |
There is probably a pathological case here where a loop of abort spans or lease transfers pushes up the range size and then the range gets to run wild but I'm not all that concerned about that. |
This commit increases backpressureByteTolerance in order to make it less fragile in the face of large concurrent writes. This smaller setting was chosen prior to implementing the additional mitigations in cockroachdb#46271. Release note: None
This commit increases backpressureByteTolerance in order to make it less fragile in the face of large concurrent writes. This smaller setting was chosen prior to implementing the additional mitigations in cockroachdb#46271. Release note: None
Describe the problem
When a range (well, most ranges) get too large (2x the
range_max_bytes
as dictated by the zone config), we apply back pressure to writes and block them until a split occurs. In the normal course of business when a split is due to write traffic to the range, this is reasonable. We expect this situation to be quite rate. Once a range exceedsrange_max_bytes
by even a bit, we'd expect a split to happen soon. A common case where this backpressure can occur is when a range is unsplittable because all of its data is due to a single key.There is however, a case where this backpressure is particularly bad: lowering the
range_max_bytes
for a range. Imagine you have a large number of ranges with the new default range size of512 MiB
and you try to lower it to the old default of64 MiB
. At that point, all writes to those ranges will be blocked until the range splits twice!To Reproduce
Create a large number of ranges with the new default range size of
512 MiB
and you try to lower it to the old default of64 MiB
. At that point, all writes to those ranges will be blocked until the range splits twice!Expected behavior
Lowering the
max_range_size
causes graceful splitting.Additional context
Also pretty terrible is that this impacts the time series data. Check out the part of this graph where no time series data was written:
The text was updated successfully, but these errors were encountered: