Skip to content
This repository has been archived by the owner on Jun 23, 2022. It is now read-only.

fix too many ERR_BUSY logs problem when write throttling enabled #243

Merged
merged 4 commits into from Apr 16, 2019
Merged

fix too many ERR_BUSY logs problem when write throttling enabled #243

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
8b789fd
fix too many ERR_BUSY logs problem when write throttling enabled
qinzuoyan Apr 16, 2019
5bdd61d
fix to code review
qinzuoyan Apr 16, 2019
2aa67ee
Merge branch 'master' into qinzuoyan
qinzuoyan Apr 16, 2019
4d48214
fix to code review
qinzuoyan Apr 16, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
15 changes: 14 additions & 1 deletion src/dist/replication/lib/replica_stub.cpp
View file
Open in desktop
Expand Up @@ -276,6 +276,14 @@ void replica_stub::install_perf_counters()
"recent.write.fail.count",
COUNTER_TYPE_VOLATILE_NUMBER,
"write fail count in the recent period");
_counter_recent_read_busy_count.init_app_counter("eon.replica_stub",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

现在对read应该没有限流吧，暂时可以不加这个counter吧

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ERR_BUSY未必只有在throttling reject的时候才会返回，所以还是加上counter_recent_read_busy_count。

"recent.read.busy.count",
COUNTER_TYPE_VOLATILE_NUMBER,
"read busy count in the recent period");
_counter_recent_write_busy_count.init_app_counter("eon.replica_stub",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

是不是改成_counter_recent_write_throtting_count更合适

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

throttling分为delay和reject，只有reject返回ERR_BUSY，并将该counter加1

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

那是不是改成_counter_recent_write_reject_count更合适？总觉得busy有些笼统

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ERR_BUSY未必只有在throttling reject的时候才会返回。
这样看来，我还是加上counter_recent_read_busy_count吧。

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

嗯，我刚刚看了下代码，rpc busy的时候会返回ERR_BUSY，那还是加上吧

"recent.write.busy.count",
COUNTER_TYPE_VOLATILE_NUMBER,
"write busy count in the recent period");
}

void replica_stub::initialize(bool clear /* = false*/)
Expand Down Expand Up @@ -1332,7 +1340,12 @@ void replica_stub::response_client(gpid id,
partition_status::type status,
error_code error)
{
if (error != ERR_OK) {
if (error == ERR_BUSY) {
if (is_read)
_counter_recent_read_busy_count->increment();
else
_counter_recent_write_busy_count->increment();
} else if (error != ERR_OK) {
if (is_read)
_counter_recent_read_fail_count->increment();
else
Expand Down
2 changes: 2 additions & 0 deletions src/dist/replication/lib/replica_stub.h
View file
Open in desktop
Expand Up @@ -323,6 +323,8 @@ class replica_stub : public serverlet<replica_stub>, public ref_counter

perf_counter_wrapper _counter_recent_read_fail_count;
perf_counter_wrapper _counter_recent_write_fail_count;
perf_counter_wrapper _counter_recent_read_busy_count;
perf_counter_wrapper _counter_recent_write_busy_count;

dsn::task_tracker _tracker;
};
Expand Down