perf: Refactor onion_client.c do_friends() to reduce network traffic#2078
Merged
Conversation
cba5d15 to
8c0be32
Compare
Codecov Report
@@ Coverage Diff @@
## master #2078 +/- ##
==========================================
+ Coverage 78.15% 78.22% +0.06%
==========================================
Files 117 117
Lines 21516 21527 +11
==========================================
+ Hits 16816 16839 +23
+ Misses 4700 4688 -12
Continue to review full report at Codecov.
|
27de9a5 to
8a7a9d9
Compare
9a05ed5 to
aca376d
Compare
41e0d2c to
8dc9cf4
Compare
83a191e to
5120e59
Compare
5120e59 to
faae422
Compare
iphydf
requested changes
Feb 25, 2022
e294a9b to
c1d1a23
Compare
8d5e25f to
5a75979
Compare
iphydf
requested changes
Mar 3, 2022
This commit simplifies the logic determining how often we send announce request packets to offline peers and how often we attempt to re-populate our nodes lists. It also removes some randomness from packet sending that was intended to reduce network traffic but greatly increased the code complexity and didn't really do all that much to help. We now aggressively search the DHT for offline friends when we first bootstrap, add a new friend, or when a friend goes offline, for a short duration until we've sent a certain number of successful lookup requests. Thereafter, we send one lookup request every 3 minutes for each node associated with the friend (typically between 4 and 8), for each friend. We also no longer spam node requests whenever our list of living nodes drops below the maximum. Instead, we repopulate the nodes list only when it drops to or below half the maximum, or after a 10 minute timeout. The effects of these changes is a decrease in overall tox network traffic by about 30-40% with UDP enabled, and a bit less than that with UDP disabled. The network may also be more reliable overall because everything is significantly more deterministic and less random
We now exponentially reduce the frequency of friend lookups based on how many lookups we've already made for an offline friend
iphydf
approved these changes
Mar 3, 2022
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR simplifies the logic determining how often we
send announce request packets to offline peers and how often
we attempt to re-populate our nodes lists. It also removes
some randomness from packet sending that was intended to
reduce network traffic but greatly increased the code complexity
and didn't really do all that much to help.
We now aggressively search the DHT for offline friends when we first
bootstrap, add a new friend, or when a friend goes offline, and
exponentially increase the lookup interval based on how many
successful lookup requests we've already made, with a max
interval of 60 minutes per node, per friend (with 8 max nodes).
Requests are also now spaced evenly apart for each node
which guarantees at least one lookup per friend, per n seconds,
where n depends on the current timeout.
We also no longer spam node requests whenever our list of living
nodes drops below the maximum. Instead, we repopulate the nodes
list only when it drops to or below half the maximum, or after
a 10 minute timeout.
I ran two identical tox instances simultaneously for 10+ hours using
the netprof branch, and found that these changes reduced total tox
network traffic by between 30% and 50% (there is some randomness
based on the amount of in-traffic you get). Results are below (click
links for full results):
Current master with UDP enabled:
This branch with UDP enabled:
Current master with UDP disabled:
This branch with UDP disabled:
This change is