INDY-1602: Dynamic quota control #905
Conversation
Signed-off-by: Sergey Khoroshavin <sergey.khoroshavin@dsr-corporation.com>
…nto node Signed-off-by: Sergey Khoroshavin <sergey.khoroshavin@dsr-corporation.com>
Signed-off-by: Sergey Khoroshavin <sergey.khoroshavin@dsr-corporation.com>
skhoroshavin
force-pushed
the
indy-1602
branch
from
634ffb9
to
eb6f93c
Compare
plenum/server/quota_control.py
Outdated
| max_node_quota: Quota, | ||
| max_client_quota: Quota): | ||
| self._dynamic = dynamic | ||
| self._max_request_queue_size = max_request_queue_size |
There was a problem hiding this comment.
Do we need the prefix max here? quota already means that this is some max limit.
There was a problem hiding this comment.
This is max quota which this strategy will ever allow, so I believe max is appropriate here.
plenum/server/quota_control.py
Outdated
| def client_quota(self) -> Quota: | ||
| if not self._dynamic: | ||
| return self._max_client_quota | ||
| return Quota(count=0, size=0) if self._request_queue_overflow else self._max_client_quota |
There was a problem hiding this comment.
This looks like one of possible strategies. Should we make QuotaControl an abstract class, and have a strategy QuotaControlNoClientMsgsOnRequestsOverflow?
stp_core/config.py
Outdated
| # Zeromq configuration | ||
| # Quotas configuration | ||
| ENABLE_DYNAMIC_QUOTAS = False | ||
| MAX_REQUEST_QUEUE_SIZE = 10000 |
There was a problem hiding this comment.
Isn't it too much? If we have a load of 10 writes per sec, than the client stack will be stopped if we don't order anything for 1000 secs.
plenum/server/node.py
Outdated
| @@ -1098,6 +1107,8 @@ def reset(self): | |||
| self.metrics.add_event(MetricsName.LOOPER_RUN_TIME_SPENT, time.perf_counter() - self.last_prod_started) | |||
| self.last_prod_started = time.perf_counter() | |||
|
|
|||
| self.quota_control.set_request_queue_len(len(self.requests)) | |||
There was a problem hiding this comment.
There is a possible issue here:
As of now we have a problem (INDY-1248) that a request is not removed from the requests queue if at least one instance hasn't yet ordered it. If one of the instances doesn't have a Primary, then the requests queue will grow even if everything is totally fine on master and other instances.
Signed-off-by: Sergey Khoroshavin <sergey.khoroshavin@dsr-corporation.com>
Signed-off-by: Sergey Khoroshavin <sergey.khoroshavin@dsr-corporation.com>
Signed-off-by: Sergey Khoroshavin <sergey.khoroshavin@dsr-corporation.com>
|
(ci) test this please |
Signed-off-by: Sergey Khoroshavin <sergey.khoroshavin@dsr-corporation.com>
| @@ -64,6 +64,7 @@ def is_ordered_by_all(self): | |||
| def __init__(self, instance_count): | |||
| self.instance_count = instance_count | |||
| self._requests = {} | |||
| self._unordered = set() | |||
There was a problem hiding this comment.
Don't we already have such a field? How do we track requests unordered for some time?
There was a problem hiding this comment.
We track them using _requests dictionary. The problem is that it contains requests unordered by any instance, and this list can become quite big if we have some slow backup instance. So I added separate set of requests which are unordered just by master.
No description provided.