Broadcast gossip announcements in sub batches #2985
Conversation
johng
changed the title
johng
changed the title
There was a problem hiding this comment.
@johng thanks for the PR, great work!! i've completed an initial first pass, and the functionality seems to be correct. apart from one or two reliability comments, the rest are mostly around simplifications and adhering to the style guidelines :)
There was a problem hiding this comment.
@johng thanks for the quick turn around. completed another pass, mostly nits and bringing the PR in line with the style guidelines. appreciate the thorough testing of the batch splits 👍
|
Gonna start running this patch on my node and see how it fairs |
|
@johng could you squash the fixup commits and rebase? |
johng
force-pushed
the
sub-batch
branch
2 times, most recently
from
6ec82ec
to
a9d3db7
Compare
discovery/gossiper.go
Outdated
| @@ -204,6 +205,14 @@ type Config struct { | |||
| // activeSyncer due to the current one not completing its state machine | |||
| // within the timeout. | |||
| ActiveSyncerTimeoutTicker ticker.Ticker | |||
|
|
|||
| // MinimumBatchSize is the minimum size of each sub batches of gossip | |||
| // decomposes it into sub batches controlled by the `subBatchSize`. | ||
| func splitAnnouncementBatches(subBatchSize int, | ||
| announcementBatch []msgWithSenders) [][]msgWithSenders { | ||
| var splitAnnouncementBatch [][]msgWithSenders |
There was a problem hiding this comment.
Nit: renaming these to annBatch and splitAnnBatch can help gain some more line space.
There was a problem hiding this comment.
I think I thought the more read able variable name is more important here
There was a problem hiding this comment.
I think ann is a reasonable abbreviation for announcement but fine with me either way.
| @@ -3571,15 +3571,76 @@ func TestSplitAnnouncementsCorrectSubBatches(t *testing.T) { | |||
| announcementBatch := make([]msgWithSenders, batchSize) | |||
|
|
|||
| splitAnnouncementBatch := splitAnnouncementBatches( | |||
| subBatchSize, | |||
| announcementBatch) | |||
| subBatchSize, announcementBatch, | |||
There was a problem hiding this comment.
Nit: looks like the TestSplitAnnouncementCorrectSubBatches changes are a fixup of f4a9edc?
discovery/gossiper_test.go
Outdated
| AnnSigner: &mockSigner{nodeKeyPriv1}, | ||
| SubBatchDelay: time.Second * 5, | ||
| MinimumBatchSize: 10, | ||
| Router: router, |
There was a problem hiding this comment.
Nit: this fixup should be applied to f4a9edca7e107e9af31ad8b44c8ed9c5f78c2792 since that's where the changes were originally introduced.
|
@wpaulino following your comments, and since this was a small change for the gossiper I've combined the two testing commits into one so that all commits compile. |
There was a problem hiding this comment.
@johng been testing this for a bit on my node, everything appears to be working smoothly! one final comment from me, see inline
discovery/gossiper.go
Outdated
|
|
||
| ratio := float64(subBatchDelay) / float64(totalDelay) | ||
|
|
||
| subBatchSize := int(math.Ceil(float64(batchSize) * ratio)) |
There was a problem hiding this comment.
i think it'd be preferable to do this computation entirely using integer arithmetic, instead of using floating points and math lib functions. current calculation does ceil(batchSize * subBatchDelay / totalDelay), which is equivalent to using ceiling integer division:
subBatchSize := (batchSize*subBatchDelay + totalDelay - 1) / totalDelaytypically we try to avoid using floats when possible, as behavior the is less consistent across architectures
|
thanks @johng latest changes lgtm! ready to be squashed |
There was a problem hiding this comment.
Awesome work @johng! Really happy with how this turned out. LGTM 🍻
Will need one more rebase (there were some issues with our build that have now been fixed), if you pull in master it should get a clean run on travis and be merge-ready. Thanks!
| // decomposes it into sub batches controlled by the `subBatchSize`. | ||
| func splitAnnouncementBatches(subBatchSize int, | ||
| announcementBatch []msgWithSenders) [][]msgWithSenders { | ||
| var splitAnnouncementBatch [][]msgWithSenders |
There was a problem hiding this comment.
I think ann is a reasonable abbreviation for announcement but fine with me either way.
| return batchSize | ||
| } | ||
|
|
||
| subBatchSize := (int(batchSize)*int(subBatchDelay) + int(totalDelay) - 1) / |
There was a problem hiding this comment.
Nit: unnecessary cast on batchSize.
In this PR we split a list of announcements in the gossiper into sub batches. These are broadcasted at regular intervals instead of all at once.
See #2979
Pull Request Checklist
Contribution Guidelines
the positive and negative (error paths) conditions (if applicable)
the bug being fixed to prevent regressions
logging level
go fmt(the tab character should be counted as 8 characters, not 4, as some IDEs do
per default)
make checkdoes not fail any testsgo vetdoes not report any issuesmake lintdoes not report any new issues that did notalready exist
cases it can be justifiable to violate this condition. In that case, the
reason should be stated in the commit message.