This repository has been archived by the owner on Nov 15, 2023. It is now read-only.
Temporarily increase notifications buffer size #5644
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
tomaka
added
the
A2-insubstantial
stanislav-tkach
approved these changes
Apr 15, 2020
mxinden
approved these changes
Apr 15, 2020
bkchr
reviewed
Apr 15, 2020
| @@ -937,7 +937,7 @@ impl Metrics { | |||
| "sub_libp2p_notifications_queues_size", | |||
| "Total size of all the notification queues" | |||
| ), | |||
| buckets: vec![0.0, 1.0, 2.0, 4.0, 8.0, 16.0, 32.0, 64.0, 128.0, 256.0], | |||
| buckets: vec![0.0, 1.0, 2.0, 4.0, 8.0, 16.0, 32.0, 64.0, 128.0, 256.0, 511.0, 512.0], | |||
There was a problem hiding this comment.
Suggested change
| buckets: vec![0.0, 1.0, 2.0, 4.0, 8.0, 16.0, 32.0, 64.0, 128.0, 256.0, 511.0, 512.0], | |
| buckets: vec![0.0, 1.0, 2.0, 4.0, 8.0, 16.0, 32.0, 64.0, 128.0, 256.0, 512.0], |
Why 511?
There was a problem hiding this comment.
We start dropping messages above 512 leaving the length at 512.
Histogram buckets are upper bounds. Thus in the case where we don't have the 511 bucket and the histogram shows 512 we don't know whether the buffer length is
-
between 256 and 512 and thus not dropping packets, or
-
at 512 and thus dropping packets.
Introducing bucket 511 into the picture the Histogram will show 511 or lower if the buffer is not dropping packets and 512 if it is full or dropping packets.
Does that make sense @bkchr?
There was a problem hiding this comment.
Not really, but if that is intentionally, okay :)
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
It seems that the queue size of GrandPa notifications now frequently reaches the 128-256 elements bucket, as seen on this graph:
Since the hard limit is 256, I can't actually know whether notifications are discarded or not. But whether they are or not, let's increase the limit until this is taken care of.
The proper solution is #5481