Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve representative crawler #1803

Merged
merged 3 commits into from Mar 6, 2019

Conversation

@cryptocode
Copy link
Collaborator

commented Mar 5, 2019

Overall changes:

  • Reps are tracked by account instead of endpoint. This thus adds support for multiple reps per host, with an increase of total weight on livenet.
  • Rep crawling is currently partly in peer container and partly in free standing functions. This patch puts all the functionality into a class.

Functional changes compared to v18:

  • Stored peers are crawled at startup. In my tests, this obtains sufficient weight within a couple of seconds instead of up to a minute.
  • Since the rep container is now separate from the peer container, the latter no longer has a last_rep_request index. The rep crawler grabs random peers in addition to iterating known reps periodically. Along with the changes mentioned below, this quickly finds reps on startup, even if there are no stored peers.
  • When total weight < minimum, reps are crawled slightly more aggressively than before (higher frequency, and potentially a few more peers)
  • When total weight >= minimum, reps are crawled at a slightly lower frequency than before (7 vs 4 seconds). This reduces traffic a bit.

These changes appears to improve the process of obtaining peer weight in my tests.

Part of #1112

@cryptocode cryptocode added this to the V19.0 milestone Mar 5, 2019

@cryptocode cryptocode self-assigned this Mar 5, 2019

@cryptocode cryptocode requested review from clemahieu and SergiySW Mar 5, 2019

@cryptocode cryptocode referenced this pull request Mar 5, 2019

Open

Break up and document node source #1112

8 of 13 tasks complete

@zhyatt zhyatt added this to CP1 (2019-03-13) in V19 Mar 5, 2019

@SergiySW
Copy link
Collaborator

left a comment

Looking good

Show resolved Hide resolved nano/node/repcrawler.hpp Outdated
Show resolved Hide resolved nano/node/repcrawler.cpp Outdated
@clemahieu

This comment has been minimized.

Copy link
Collaborator

commented Mar 6, 2019

LGTM after some small nits.

@cryptocode cryptocode merged commit b8cde9f into nanocurrency:master Mar 6, 2019

2 checks passed

continuous-integration/appveyor/pr AppVeyor build succeeded
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details

@cryptocode cryptocode deleted the cryptocode:repcrawler-class branch Mar 6, 2019

@cryptocode cryptocode referenced this pull request Mar 6, 2019

Closed

Improve rep tracking #1756

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.