Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Broadcast gossip announcements in sub batches #2985

Merged
merged 3 commits into from May 29, 2019

Conversation

@johng
Copy link
Contributor

@johng johng commented Apr 21, 2019

In this PR we split a list of announcements in the gossiper into sub batches. These are broadcasted at regular intervals instead of all at once.

See #2979

Pull Request Checklist

  • If this is your first time contributing, we recommend you read the Code
    Contribution Guidelines
  • All changes are Go version 1.12 compliant
  • The code being submitted is commented according to Code Documentation and Commenting
  • For new code: Code is accompanied by tests which exercise both
    the positive and negative (error paths) conditions (if applicable)
  • For bug fixes: Code is accompanied by new tests which trigger
    the bug being fixed to prevent regressions
  • Any new logging statements use an appropriate subsystem and
    logging level
  • Code has been formatted with go fmt
  • For code and documentation: lines are wrapped at 80 characters
    (the tab character should be counted as 8 characters, not 4, as some IDEs do
    per default)
  • Running make check does not fail any tests
  • Running go vet does not report any issues
  • Running make lint does not report any new issues that did not
    already exist
  • All commits build properly and pass tests. Only in exceptional
    cases it can be justifiable to violate this condition. In that case, the
    reason should be stated in the commit message.
  • Commits have a logical structure according to Ideal Git Commit Structure
@johng johng changed the title Broadcast gossip announcements into subbatches Broadcast gossip announcements into sub batches Apr 21, 2019
@johng johng changed the title Broadcast gossip announcements into sub batches Broadcast gossip announcements in sub batches Apr 21, 2019
Copy link
Collaborator

@cfromknecht cfromknecht left a comment

@johng thanks for the PR, great work!! i've completed an initial first pass, and the functionality seems to be correct. apart from one or two reliability comments, the rest are mostly around simplifications and adhering to the style guidelines :)

Loading

discovery/gossiper.go Outdated Show resolved Hide resolved
Loading
discovery/gossiper.go Show resolved Hide resolved
Loading
discovery/gossiper.go Outdated Show resolved Hide resolved
Loading
discovery/gossiper.go Outdated Show resolved Hide resolved
Loading
discovery/gossiper.go Outdated Show resolved Hide resolved
Loading
discovery/gossiper.go Outdated Show resolved Hide resolved
Loading
discovery/gossiper.go Outdated Show resolved Hide resolved
Loading
discovery/gossiper.go Show resolved Hide resolved
Loading
discovery/gossiper.go Show resolved Hide resolved
Loading
discovery/gossiper_test.go Outdated Show resolved Hide resolved
Loading
Copy link
Collaborator

@cfromknecht cfromknecht left a comment

@johng thanks for the quick turn around. completed another pass, mostly nits and bringing the PR in line with the style guidelines. appreciate the thorough testing of the batch splits 👍

Loading

go.sum Outdated Show resolved Hide resolved
Loading
discovery/gossiper.go Outdated Show resolved Hide resolved
Loading
discovery/gossiper.go Outdated Show resolved Hide resolved
Loading
discovery/gossiper.go Show resolved Hide resolved
Loading
discovery/gossiper.go Outdated Show resolved Hide resolved
Loading
discovery/gossiper.go Outdated Show resolved Hide resolved
Loading
discovery/gossiper_test.go Outdated Show resolved Hide resolved
Loading
discovery/gossiper.go Show resolved Hide resolved
Loading
discovery/gossiper.go Outdated Show resolved Hide resolved
Loading
discovery/gossiper_test.go Show resolved Hide resolved
Loading
@cfromknecht
Copy link
Collaborator

@cfromknecht cfromknecht commented Apr 26, 2019

Thanks @johng! This looks pretty good to me, would be good to get a second review from @wpaulino or @halseth

Loading

@cfromknecht cfromknecht requested review from halseth and wpaulino Apr 26, 2019
@cfromknecht
Copy link
Collaborator

@cfromknecht cfromknecht commented Apr 26, 2019

Gonna start running this patch on my node and see how it fairs

Loading

@wpaulino
Copy link
Collaborator

@wpaulino wpaulino commented Apr 27, 2019

@johng could you squash the fixup commits and rebase?

Loading

discovery/gossiper.go Outdated Show resolved Hide resolved
Loading
@johng johng force-pushed the sub-batch branch 2 times, most recently from 6ec82ec to a9d3db7 Apr 27, 2019
Copy link
Collaborator

@wpaulino wpaulino left a comment

Mostly just nits from me on commit structure.

Loading

@@ -204,6 +205,14 @@ type Config struct {
// activeSyncer due to the current one not completing its state machine
// within the timeout.
ActiveSyncerTimeoutTicker ticker.Ticker

// MinimumBatchSize is the minimum size of each sub batches of gossip
Copy link
Collaborator

@wpaulino wpaulino Apr 29, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: s/batches/batch/

Loading

// decomposes it into sub batches controlled by the `subBatchSize`.
func splitAnnouncementBatches(subBatchSize int,
announcementBatch []msgWithSenders) [][]msgWithSenders {
var splitAnnouncementBatch [][]msgWithSenders
Copy link
Collaborator

@wpaulino wpaulino Apr 29, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: renaming these to annBatch and splitAnnBatch can help gain some more line space.

Loading

Copy link
Contributor Author

@johng johng Apr 30, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I thought the more read able variable name is more important here

Loading

Copy link
Collaborator

@wpaulino wpaulino May 24, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think ann is a reasonable abbreviation for announcement but fine with me either way.

Loading

server.go Show resolved Hide resolved
Loading
discovery/gossiper.go Outdated Show resolved Hide resolved
Loading
@@ -3571,15 +3571,76 @@ func TestSplitAnnouncementsCorrectSubBatches(t *testing.T) {
announcementBatch := make([]msgWithSenders, batchSize)

splitAnnouncementBatch := splitAnnouncementBatches(
subBatchSize,
announcementBatch)
subBatchSize, announcementBatch,
Copy link
Collaborator

@wpaulino wpaulino Apr 29, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: looks like the TestSplitAnnouncementCorrectSubBatches changes are a fixup of f4a9edc?

Loading

AnnSigner: &mockSigner{nodeKeyPriv1},
SubBatchDelay: time.Second * 5,
MinimumBatchSize: 10,
Router: router,
Copy link
Collaborator

@wpaulino wpaulino Apr 29, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: this fixup should be applied to f4a9edc since that's where the changes were originally introduced.

Loading

@johng
Copy link
Contributor Author

@johng johng commented Apr 30, 2019

@wpaulino following your comments, and since this was a small change for the gossiper I've combined the two testing commits into one so that all commits compile.

Loading

@wpaulino wpaulino added this to the 0.7 milestone May 8, 2019
Copy link
Collaborator

@cfromknecht cfromknecht left a comment

@johng been testing this for a bit on my node, everything appears to be working smoothly! one final comment from me, see inline

Loading


ratio := float64(subBatchDelay) / float64(totalDelay)

subBatchSize := int(math.Ceil(float64(batchSize) * ratio))
Copy link
Collaborator

@cfromknecht cfromknecht May 13, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i think it'd be preferable to do this computation entirely using integer arithmetic, instead of using floating points and math lib functions. current calculation does ceil(batchSize * subBatchDelay / totalDelay), which is equivalent to using ceiling integer division:

subBatchSize := (batchSize*subBatchDelay + totalDelay - 1) / totalDelay

typically we try to avoid using floats when possible, as behavior the is less consistent across architectures

Loading

@cfromknecht
Copy link
Collaborator

@cfromknecht cfromknecht commented May 22, 2019

thanks @johng latest changes lgtm! ready to be squashed

Loading

Copy link
Collaborator

@cfromknecht cfromknecht left a comment

Awesome work @johng! Really happy with how this turned out. LGTM 🍻

Will need one more rebase (there were some issues with our build that have now been fixed), if you pull in master it should get a clean run on travis and be merge-ready. Thanks!

Loading

Copy link
Collaborator

@wpaulino wpaulino left a comment

LGTM 🎱

A few non-blocking nits left inline -- would prefer if the commits were squashed into one as we generally favor each commit being able to build on its own.

Loading

// decomposes it into sub batches controlled by the `subBatchSize`.
func splitAnnouncementBatches(subBatchSize int,
announcementBatch []msgWithSenders) [][]msgWithSenders {
var splitAnnouncementBatch [][]msgWithSenders
Copy link
Collaborator

@wpaulino wpaulino May 24, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think ann is a reasonable abbreviation for announcement but fine with me either way.

Loading

return batchSize
}

subBatchSize := (int(batchSize)*int(subBatchDelay) + int(totalDelay) - 1) /
Copy link
Collaborator

@wpaulino wpaulino May 24, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: unnecessary cast on batchSize.

Loading

@cfromknecht cfromknecht merged commit f8287b0 into lightningnetwork:master May 29, 2019
2 checks passed
Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

3 participants