add mechanisms for rate limiting remote bootstraps #418
Comments
yugabyte-ci
pushed a commit
that referenced
this issue
Aug 24, 2018
Summary: Currently we don't have a mechanism to prioritize network traffic. So if a tablet gets moved during a period of high read/writes ops, our performance will degrade during remote bootstraps. With this change we provide a throttling mechanism that can be controlled with the flag remote_boostrap_rate_limit_bytes_per_sec. This is a per-node global rate for receiving and sending. So no matter how many concurrent remote bootstraps are happening, the max bandwidth utilization for sending or receiving will not exceed the specified limit. For example, if remote_boostrap_rate_limit_bytes_per_sec is 10MB/s, then the maximum bandwidth for sending data to other nodes will not exceed 10MB/s. At the same time, the maximum bandwidth for receiving data in the same node will not exceed 10MB/s. So the maximum bandwidth per node allocated for remote bootstrap traffic will be 2 * remote_boostrap_rate_limit_bytes_per_sec. To disable throttling, set remote_boostrap_rate_limit_bytes_per_sec = 0. Test Plan: new unit tests Reviewers: bharat, venkatesh, mikhail, sergei Reviewed By: bharat, venkatesh, sergei Subscribers: kannan, ybase, bogdan Differential Revision: https://phabricator.dev.yugabyte.com/D4608
yugabyte-ci
pushed a commit
that referenced
this issue
Sep 4, 2018
Summary: Increase default rate from 50 MB/s to 100 MB/s Test Plan: unit tests Reviewers: venkatesh, kannan Reviewed By: kannan Subscribers: ybase Differential Revision: https://phabricator.dev.yugabyte.com/D5425
mbautin
pushed a commit
that referenced
this issue
Jul 11, 2019
…ed to the earlier commit 154810c Original commit message: (#418) ENG-2810: Rate limiter for remote bootstrap Summary: Currently we don't have a mechanism to prioritize network traffic. So if a tablet gets moved during a period of high read/writes ops, our performance will degrade during remote bootstraps. With this change we provide a throttling mechanism that can be controlled with the flag remote_boostrap_rate_limit_bytes_per_sec. This is a per-node global rate for receiving and sending. So no matter how many concurrent remote bootstraps are happening, the max bandwidth utilization for sending or receiving will not exceed the specified limit. For example, if remote_boostrap_rate_limit_bytes_per_sec is 10MB/s, then the maximum bandwidth for sending data to other nodes will not exceed 10MB/s. At the same time, the maximum bandwidth for receiving data in the same node will not exceed 10MB/s. So the maximum bandwidth per node allocated for remote bootstrap traffic will be 2 * remote_boostrap_rate_limit_bytes_per_sec. To disable throttling, set remote_boostrap_rate_limit_bytes_per_sec = 0. Test Plan: new unit tests Reviewers: bharat, venkatesh, mikhail, sergei Reviewed By: bharat, venkatesh, sergei Subscribers: kannan, ybase, bogdan Differential Revision: https://phabricator.dev.yugabyte.com/D4608
mbautin
pushed a commit
to mbautin/yugabyte-db
that referenced
this issue
Jul 16, 2019
Summary: Currently we don't have a mechanism to prioritize network traffic. So if a tablet gets moved during a period of high read/writes ops, our performance will degrade during remote bootstraps. With this change we provide a throttling mechanism that can be controlled with the flag remote_boostrap_rate_limit_bytes_per_sec. This is a per-node global rate for receiving and sending. So no matter how many concurrent remote bootstraps are happening, the max bandwidth utilization for sending or receiving will not exceed the specified limit. For example, if remote_boostrap_rate_limit_bytes_per_sec is 10MB/s, then the maximum bandwidth for sending data to other nodes will not exceed 10MB/s. At the same time, the maximum bandwidth for receiving data in the same node will not exceed 10MB/s. So the maximum bandwidth per node allocated for remote bootstrap traffic will be 2 * remote_boostrap_rate_limit_bytes_per_sec. To disable throttling, set remote_boostrap_rate_limit_bytes_per_sec = 0. Test Plan: new unit tests Reviewers: bharat, venkatesh, mikhail, sergei Reviewed By: bharat, venkatesh, sergei Subscribers: kannan, ybase, bogdan Differential Revision: https://phabricator.dev.yugabyte.com/D4608 Note: This commit provides additional functionality that is logically related to the earlier commit yugabyte@154810c and supersedes the commit yugabyte@e00b14d
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
When a node is added and starts getting new tablets, or when a node dies for an extended period and tablets are moved to other nodes, the controls we have today are for the total number of tablets being "remote bootstrapped" at any given time. [Since this is a background activity, the primary goal is not adversely impact the foreground operations as it is important to bring in the new node gracefully even if it takes longer time.]
Today, each of those remote tablet bootstrap operations go at full throttle. We need a mechanism to rate limit these remote tablet bootstrap operations (similar to rate limiters in place for compaction/flush operations) so that we don't peg network or disk sub-systems.
The text was updated successfully, but these errors were encountered: