Skip to content

2.25.0.0-b189

@hulien22 hulien22 tagged this 23 Oct 08:09
Summary:
In the case of network partitions, the master leader may only hear about a new tablet peer from tservers other than the one that is hosting the replica. This results in the tablet being marked in an UNKNOWN raft state.
Due to some improvements to table creation handling, the LB treats these UNKNOWN peers as RUNNING - this causes issues if the tablets are actually BOOTSTRAPPING as they would not count towards the LB's RBS limit, thus leading to scenarios where the LB triggers many more RBSs than it is allowed.

Changing the behaviour here to check tablet consensus state if it the state is UNKNOWN in the load balancer. If the tablet is a PRE_VOTER or PRE_OBSERVER, then treat it as BOOTSTRAPPING - If it is VOTER/OBSERVER, then treat it as RUNNING.
This ensures that these bootstrapping tablets are treated as bootstrapping by the LB and will count towards RBS limits.

Isolating this fix to only the load balancer for now - we would like to change the heartbeat path to not set UNKNOWN state for previously known peer states, but that requires more investigation (see GHI #21840)
Jira: DB-12401

Test Plan:
```
ybd --cxx-test load_balancer-test --gtest_filter "LoadBalancerManyTabletsTest.LimitRbsToUnresponsiveNode"
```
This test fails when we mark bootstrapping peers as UNKNOWN (it will go and try to RBS more than the limit)

Reviewers: asrivastava, zdrudi

Reviewed By: asrivastava, zdrudi

Subscribers: ybase

Differential Revision: https://phorge.dev.yugabyte.com/D37189
Assets 2
Loading