Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

server: stagger initial reconnects #2885

Merged

Conversation

Projects
None yet
3 participants
@cfromknecht
Copy link
Collaborator

cfromknecht commented Apr 4, 2019

This commit adds optional jitter to our initial reconnection to our
persistent peers. Currently we will attempt reconnections to all peers
simultaneously, which results in large amount of contention as the
number of channels a node has grows.

We resolve this by adding a randomized delay between 0 and 30 seconds
for all persistent peers. This spreads out the load and contention to
resources such as the database, read/write pools, and memory
allocations. On my node, this allows to start up with about 80% of the
memory burst compared to the all-at-once approach.

This also has a second-order effect in better distributing messages sent
at constant intervals, such as pings. This reduces the concurrent jobs
submitted to the read and write pools at any given time, resulting in
better reuse of read/write buffers and fewer bursty allocation and
garbage collection cycles.

The behavior can be enabled via the --stagger-initial-reconnect flag.

@cfromknecht cfromknecht force-pushed the cfromknecht:stagger-initial-reconnect branch 5 times, most recently from be37e7b to 81687ed Apr 4, 2019

Show resolved Hide resolved config.go Outdated

@cfromknecht cfromknecht force-pushed the cfromknecht:stagger-initial-reconnect branch 4 times, most recently from e89aec9 to f3f9c7f Apr 4, 2019

Show resolved Hide resolved config.go Outdated

@cfromknecht cfromknecht force-pushed the cfromknecht:stagger-initial-reconnect branch 4 times, most recently from 1a45f94 to 557dec6 Apr 5, 2019

Show resolved Hide resolved server.go Outdated

@cfromknecht cfromknecht force-pushed the cfromknecht:stagger-initial-reconnect branch 5 times, most recently from e0c4b24 to 172b16c Apr 5, 2019

Show resolved Hide resolved server.go Outdated
server: stagger initial reconnects
This commit adds optional jitter to our initial reconnection to our
persistent peers. Currently we will attempt reconnections to all peers
simultaneously, which results in large amount of contention as the
number of channels a node has grows.

We resolve this by adding a randomized delay between 0 and 30 seconds
for all persistent peers. This spreads out the load and contention to
resources such as the database, read/write pools, and memory
allocations. On my node, this allows to start up with about 80% of the
memory burst compared to the all-at-once approach.

This also has a second-order effect in better distributing messages sent
at constant intervals, such as pings. This reduces the concurrent jobs
submitted to the read and write pools at any given time, resulting in
better reuse of read/write buffers and fewer bursty allocation and
garbage collection cycles.

@cfromknecht cfromknecht force-pushed the cfromknecht:stagger-initial-reconnect branch from 172b16c to cf80476 Apr 5, 2019

Show resolved Hide resolved config.go

@cfromknecht cfromknecht requested a review from joostjager Apr 5, 2019

@joostjager
Copy link
Collaborator

joostjager left a comment

Looks good. Has it been tested on a node with many peers?

@cfromknecht

This comment has been minimized.

Copy link
Collaborator Author

cfromknecht commented Apr 5, 2019

AFAIK it’s been tested on several nodes with 50+ channels, hoping to get feedback from even larger nodes after the next tag

@cfromknecht cfromknecht merged commit 25d2b1b into lightningnetwork:master Apr 5, 2019

2 checks passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
coverage/coveralls First build on stagger-initial-reconnect at 59.581%
Details

@cfromknecht cfromknecht deleted the cfromknecht:stagger-initial-reconnect branch Apr 5, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.