[DocDB] Add TServer with faulty drive to LB blacklist #12768
Assignees
Labels
area/docdb
YugabyteDB core features
kind/bug
This issue is a bug
priority/medium
Medium priority issue
Comments
mikhpolitov
added
area/docdb
mikhpolitov
added
kind/enhancement
mikhpolitov
changed the title
mikhpolitov
added a commit
that referenced
this issue
Jun 17, 2022
Summary: A TServer with faulty drive has fewer tablets than other TServers, which can cause Load Balancer to move tablets to this TServer, but moving tablets to TServer with faulty drive leads to tablet count disbalances on leftover healthy drives. Also, the replaced drive starts with 0 tablets while other drives keep tablets. With this change, the Load Balancer effectively blacklists the TServer. The corresponding platform change ensures that the user is alerted about it and would fix the underlying issue. Once the user fixes the underlying disk issue and brings the TServer back up, then the TServer no longer reports faulty drive issues to the master and as a result the blacklist gets removed. Test Plan: ybd --cxx-test load_balancer_mini_cluster-test Reviewers: skedia, jhe, rthallam Reviewed By: jhe Subscribers: ybase, bogdan Differential Revision: https://phabricator.dev.yugabyte.com/D17680
mikhpolitov
changed the title
mikhpolitov
added a commit
that referenced
this issue
Jun 21, 2022
… blacklist Summary: A TServer with faulty drive has fewer tablets than other TServers, which can cause Load Balancer to move tablets to this TServer, but moving tablets to TServer with faulty drive leads to tablet count disbalances on leftover healthy drives. Also, the replaced drive starts with 0 tablets while other drives keep tablets. With this change, the Load Balancer effectively blacklists the TServer. The corresponding platform change ensures that the user is alerted about it and would fix the underlying issue. Once the user fixes the underlying disk issue and brings the TServer back up, then the TServer no longer reports faulty drive issues to the master and as a result the blacklist gets removed. Original commit: 2549499 / D17680 Partially changes from commit: df9bd67 / D17781 Test Plan: ybd --cxx-test load_balancer_mini_cluster-test Reviewers: skedia, rthallam, sergei, jhe Reviewed By: sergei, jhe Subscribers: sergei, bogdan, ybase Differential Revision: https://phabricator.dev.yugabyte.com/D17775
yugabyte-ci
added
kind/bug
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Jira Link: DB-652
Description
A TServer with fault drive has fewer tablets than other TServers, which can cause Load Balancer to move tablets to this TServer, but moving tablets to TServer with fault drive leads to tablet count disbalances on leftover healthy drives. Also, the replaced drive starts with 0 tablets while other drives keep tablets.
The text was updated successfully, but these errors were encountered: