Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Broadcast gossip announcements in sub batches #2985

Merged
merged 3 commits into from May 29, 2019

Conversation

Projects
None yet
3 participants
@johng
Copy link
Contributor

commented Apr 21, 2019

In this PR we split a list of announcements in the gossiper into sub batches. These are broadcasted at regular intervals instead of all at once.

See #2979

Pull Request Checklist

  • If this is your first time contributing, we recommend you read the Code
    Contribution Guidelines
  • All changes are Go version 1.12 compliant
  • The code being submitted is commented according to Code Documentation and Commenting
  • For new code: Code is accompanied by tests which exercise both
    the positive and negative (error paths) conditions (if applicable)
  • For bug fixes: Code is accompanied by new tests which trigger
    the bug being fixed to prevent regressions
  • Any new logging statements use an appropriate subsystem and
    logging level
  • Code has been formatted with go fmt
  • For code and documentation: lines are wrapped at 80 characters
    (the tab character should be counted as 8 characters, not 4, as some IDEs do
    per default)
  • Running make check does not fail any tests
  • Running go vet does not report any issues
  • Running make lint does not report any new issues that did not
    already exist
  • All commits build properly and pass tests. Only in exceptional
    cases it can be justifiable to violate this condition. In that case, the
    reason should be stated in the commit message.
  • Commits have a logical structure according to Ideal Git Commit Structure

@johng johng force-pushed the johng:sub-batch branch from 97fb96f to 623872b Apr 21, 2019

@johng johng changed the title Broadcast gossip announcements into subbatches Broadcast gossip announcements into sub batches Apr 21, 2019

@johng johng changed the title Broadcast gossip announcements into sub batches Broadcast gossip announcements in sub batches Apr 21, 2019

@cfromknecht
Copy link
Collaborator

left a comment

@johng thanks for the PR, great work!! i've completed an initial first pass, and the functionality seems to be correct. apart from one or two reliability comments, the rest are mostly around simplifications and adhering to the style guidelines :)

Show resolved Hide resolved discovery/gossiper.go Outdated
Show resolved Hide resolved discovery/gossiper.go
Show resolved Hide resolved discovery/gossiper.go Outdated
Show resolved Hide resolved discovery/gossiper.go Outdated
Show resolved Hide resolved discovery/gossiper.go Outdated
Show resolved Hide resolved discovery/gossiper.go Outdated
Show resolved Hide resolved discovery/gossiper.go Outdated
Show resolved Hide resolved discovery/gossiper.go
Show resolved Hide resolved discovery/gossiper.go
Show resolved Hide resolved discovery/gossiper_test.go Outdated

@johng johng force-pushed the johng:sub-batch branch from a24f7bf to 82d9924 Apr 24, 2019

@cfromknecht
Copy link
Collaborator

left a comment

@johng thanks for the quick turn around. completed another pass, mostly nits and bringing the PR in line with the style guidelines. appreciate the thorough testing of the batch splits 👍

Show resolved Hide resolved go.sum Outdated
Show resolved Hide resolved discovery/gossiper.go Outdated
Show resolved Hide resolved discovery/gossiper.go Outdated
Show resolved Hide resolved discovery/gossiper.go
Show resolved Hide resolved discovery/gossiper.go Outdated
Show resolved Hide resolved discovery/gossiper.go Outdated
Show resolved Hide resolved discovery/gossiper_test.go Outdated
Show resolved Hide resolved discovery/gossiper.go
Show resolved Hide resolved discovery/gossiper.go Outdated
Show resolved Hide resolved discovery/gossiper_test.go

@johng johng force-pushed the johng:sub-batch branch from 1d2eea6 to d823bc4 Apr 25, 2019

@cfromknecht

This comment has been minimized.

Copy link
Collaborator

commented Apr 26, 2019

Thanks @johng! This looks pretty good to me, would be good to get a second review from @wpaulino or @halseth

@cfromknecht cfromknecht requested review from halseth and wpaulino Apr 26, 2019

@cfromknecht

This comment has been minimized.

Copy link
Collaborator

commented Apr 26, 2019

Gonna start running this patch on my node and see how it fairs

@wpaulino

This comment has been minimized.

Copy link
Collaborator

commented Apr 27, 2019

@johng could you squash the fixup commits and rebase?

Show resolved Hide resolved discovery/gossiper.go Outdated

@johng johng force-pushed the johng:sub-batch branch 2 times, most recently from 6ec82ec to a9d3db7 Apr 27, 2019

@wpaulino
Copy link
Collaborator

left a comment

Mostly just nits from me on commit structure.

@@ -204,6 +205,14 @@ type Config struct {
// activeSyncer due to the current one not completing its state machine
// within the timeout.
ActiveSyncerTimeoutTicker ticker.Ticker

// MinimumBatchSize is the minimum size of each sub batches of gossip

This comment has been minimized.

Copy link
@wpaulino

wpaulino Apr 29, 2019

Collaborator

Nit: s/batches/batch/

// decomposes it into sub batches controlled by the `subBatchSize`.
func splitAnnouncementBatches(subBatchSize int,
announcementBatch []msgWithSenders) [][]msgWithSenders {
var splitAnnouncementBatch [][]msgWithSenders

This comment has been minimized.

Copy link
@wpaulino

wpaulino Apr 29, 2019

Collaborator

Nit: renaming these to annBatch and splitAnnBatch can help gain some more line space.

This comment has been minimized.

Copy link
@johng

johng Apr 30, 2019

Author Contributor

I think I thought the more read able variable name is more important here

This comment has been minimized.

Copy link
@wpaulino

wpaulino May 24, 2019

Collaborator

I think ann is a reasonable abbreviation for announcement but fine with me either way.

Show resolved Hide resolved server.go
Show resolved Hide resolved discovery/gossiper.go Outdated
@@ -3571,15 +3571,76 @@ func TestSplitAnnouncementsCorrectSubBatches(t *testing.T) {
announcementBatch := make([]msgWithSenders, batchSize)

splitAnnouncementBatch := splitAnnouncementBatches(
subBatchSize,
announcementBatch)
subBatchSize, announcementBatch,

This comment has been minimized.

Copy link
@wpaulino

wpaulino Apr 29, 2019

Collaborator

Nit: looks like the TestSplitAnnouncementCorrectSubBatches changes are a fixup of f4a9edc?

AnnSigner: &mockSigner{nodeKeyPriv1},
SubBatchDelay: time.Second * 5,
MinimumBatchSize: 10,
Router: router,

This comment has been minimized.

Copy link
@wpaulino

wpaulino Apr 29, 2019

Collaborator

Nit: this fixup should be applied to f4a9edc since that's where the changes were originally introduced.

@johng johng force-pushed the johng:sub-batch branch from 9c7a220 to b8912f5 Apr 30, 2019

@johng

This comment has been minimized.

Copy link
Contributor Author

commented Apr 30, 2019

@wpaulino following your comments, and since this was a small change for the gossiper I've combined the two testing commits into one so that all commits compile.

@johng johng force-pushed the johng:sub-batch branch from e837211 to 92f324d Apr 30, 2019

@wpaulino wpaulino added this to the 0.7 milestone May 8, 2019

@cfromknecht
Copy link
Collaborator

left a comment

@johng been testing this for a bit on my node, everything appears to be working smoothly! one final comment from me, see inline


ratio := float64(subBatchDelay) / float64(totalDelay)

subBatchSize := int(math.Ceil(float64(batchSize) * ratio))

This comment has been minimized.

Copy link
@cfromknecht

cfromknecht May 13, 2019

Collaborator

i think it'd be preferable to do this computation entirely using integer arithmetic, instead of using floating points and math lib functions. current calculation does ceil(batchSize * subBatchDelay / totalDelay), which is equivalent to using ceiling integer division:

subBatchSize := (batchSize*subBatchDelay + totalDelay - 1) / totalDelay

typically we try to avoid using floats when possible, as behavior the is less consistent across architectures

@cfromknecht

This comment has been minimized.

Copy link
Collaborator

commented May 22, 2019

thanks @johng latest changes lgtm! ready to be squashed

@johng johng force-pushed the johng:sub-batch branch from 4d7c975 to 261eb04 May 22, 2019

@cfromknecht
Copy link
Collaborator

left a comment

Awesome work @johng! Really happy with how this turned out. LGTM 🍻

Will need one more rebase (there were some issues with our build that have now been fixed), if you pull in master it should get a clean run on travis and be merge-ready. Thanks!

@johng johng force-pushed the johng:sub-batch branch from 261eb04 to f6058b4 May 23, 2019

@wpaulino
Copy link
Collaborator

left a comment

LGTM 🎱

A few non-blocking nits left inline -- would prefer if the commits were squashed into one as we generally favor each commit being able to build on its own.

// decomposes it into sub batches controlled by the `subBatchSize`.
func splitAnnouncementBatches(subBatchSize int,
announcementBatch []msgWithSenders) [][]msgWithSenders {
var splitAnnouncementBatch [][]msgWithSenders

This comment has been minimized.

Copy link
@wpaulino

wpaulino May 24, 2019

Collaborator

I think ann is a reasonable abbreviation for announcement but fine with me either way.

return batchSize
}

subBatchSize := (int(batchSize)*int(subBatchDelay) + int(totalDelay) - 1) /

This comment has been minimized.

Copy link
@wpaulino

wpaulino May 24, 2019

Collaborator

Nit: unnecessary cast on batchSize.

@cfromknecht cfromknecht merged commit f8287b0 into lightningnetwork:master May 29, 2019

2 checks passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
coverage/coveralls Coverage increased (+0.01%) to 60.312%
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.