Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Parallelize processing of incoming gossip messages #2463

Closed
wants to merge 26 commits into from
Closed

Parallelize processing of incoming gossip messages #2463

wants to merge 26 commits into from

Conversation

lrettig
Copy link
Member

@lrettig lrettig commented Jun 18, 2021 •
edited

Motivation

Fixes an issue spotted in tn132: processing an inbound block may take a very long time (> 1m), during which time all other inbound blocks (messages on the same gossip channel) are blocking

Changes

  • Adds concurrency to processing of inbound gossip messages (per protocol/channel), with a limit on the degree of concurrency
  • Add timeout to syncer layer fetch

Test Plan

Includes regression tests

TODO

  • Explain motivation or link existing issue(s)
  • Test changes and document test plan
  • Update documentation as needed

DevOps Notes

  • This PR does not require configuration changes (e.g., environment variables, GitHub secrets, VM resources)
  • This PR does not affect public APIs
  • This PR does not rely on a new version of external services (PoET, elasticsearch, etc.)
  • This PR does not make changes to log messages (which monitoring infrastructure may rely on)

@lrettig
Parallelize processing of incoming gossip messages
578e787 
Add param to limit degree of concurrency
Add timeout to syncer
countvonzero
countvonzero suggested changes Jun 24, 2021
View reviewed changes
Copy link
Contributor

@countvonzero countvonzero left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice cleanup

p2p/service/gossip_listener_test.go Outdated Show resolved Hide resolved
p2p/service/gossip_listener_test.go Show resolved Hide resolved
p2p/service/gossip_listener.go Show resolved Hide resolved
@lrettig lrettig added this to the July 5 Sprint milestone Jul 5, 2021
@lrettig lrettig self-assigned this Jul 5, 2021
@lrettig lrettig modified the milestones: July 5 Sprint, July 19 Sprint Jul 17, 2021
@lrettig lrettig modified the milestones: July 19 Sprint, Aug 2 Sprint Aug 2, 2021
@lrettig lrettig mentioned this pull request Aug 10, 2021
Closed
@moshababo moshababo modified the milestones: Aug 2 Sprint, Aug 16 Sprint Aug 16, 2021
@lrettig lrettig modified the milestones: Aug 16 Sprint, Aug 30 Sprint Aug 16, 2021
@lrettig
Copy link
Member Author

lrettig commented Sep 8, 2021

bors merge

bors bot pushed a commit that referenced this pull request Sep 8, 2021
@lrettig @dshulyak
Parallelize processing of incoming gossip messages (#2463)
9fb7cd8 
## Motivation
Fixes an issue spotted in tn132: processing an inbound block may take a very long time (> 1m), during which time all other inbound blocks (messages on the same gossip channel) are blocking

## Changes
- Adds concurrency to processing of inbound gossip messages (per protocol/channel), with a limit on the degree of concurrency
- Add timeout to syncer layer fetch

## Test Plan
Includes regression tests

## TODO
<!-- This section should be removed when all items are complete -->
- [x] Explain motivation or link existing issue(s)
- [x] Test changes and document test plan
- [ ] Update documentation as needed

## DevOps Notes
<!-- Please uncheck these items as applicable to make DevOps aware of changes that may affect releases -->
- [x] This PR does not require configuration changes (e.g., environment variables, GitHub secrets, VM resources)
- [x] This PR does not affect public APIs
- [x] This PR does not rely on a new version of external services (PoET, elasticsearch, etc.)
- [x] This PR does not make changes to log messages (which monitoring infrastructure may rely on)


Co-authored-by: Dmitry Shulyak <yashulyak@gmail.com>
@bors
Copy link

bors bot commented Sep 8, 2021

Build failed:

@lrettig
Copy link
Member Author

lrettig commented Sep 8, 2021

bors merge

bors bot pushed a commit that referenced this pull request Sep 8, 2021
@lrettig @dshulyak
Parallelize processing of incoming gossip messages (#2463)
11ad4ca 
## Motivation
Fixes an issue spotted in tn132: processing an inbound block may take a very long time (> 1m), during which time all other inbound blocks (messages on the same gossip channel) are blocking

## Changes
- Adds concurrency to processing of inbound gossip messages (per protocol/channel), with a limit on the degree of concurrency
- Add timeout to syncer layer fetch

## Test Plan
Includes regression tests

## TODO
<!-- This section should be removed when all items are complete -->
- [x] Explain motivation or link existing issue(s)
- [x] Test changes and document test plan
- [ ] Update documentation as needed

## DevOps Notes
<!-- Please uncheck these items as applicable to make DevOps aware of changes that may affect releases -->
- [x] This PR does not require configuration changes (e.g., environment variables, GitHub secrets, VM resources)
- [x] This PR does not affect public APIs
- [x] This PR does not rely on a new version of external services (PoET, elasticsearch, etc.)
- [x] This PR does not make changes to log messages (which monitoring infrastructure may rely on)


Co-authored-by: Dmitry Shulyak <yashulyak@gmail.com>
@bors
Copy link

bors bot commented Sep 9, 2021

Build failed:

dshulyak
dshulyak requested changes Sep 9, 2021
View reviewed changes
Copy link
Contributor

@dshulyak dshulyak left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

latenodes doesn't fail so often in develop. please no need to retry until it succeeds

@moshababo moshababo removed this from the Aug 16 Sprint milestone Sep 13, 2021
@moshababo moshababo assigned nkryuchkov and unassigned antonlerner Oct 14, 2021
@nkryuchkov
Merge branch 'develop' into inbound-gossip-patch
724c716 
# Conflicts:
#	cmd/node/node.go
#	docker-compose.yml
#	layerfetcher/layers.go
#	p2p/service/gossip_listener.go
#	syncer/syncer.go
#	syncer/syncer_test.go
@nkryuchkov
Fix golangci-lint
b4fa583
@nkryuchkov
Copy link
Contributor

bors merge

@bors
Copy link

bors bot commented Oct 16, 2021

👎 Rejected by code reviews

@nkryuchkov
Copy link
Contributor

bors merge

@bors
Copy link

bors bot commented Oct 16, 2021

👎 Rejected by code reviews

@nkryuchkov
Copy link
Contributor

bors try

bors bot added a commit that referenced this pull request Oct 16, 2021
@bors
Try #2463:
9e9dca6
@bors
Copy link

bors bot commented Oct 16, 2021

try

Build failed:

@nkryuchkov
Copy link
Contributor

try

Build failed:

#2874

@nkryuchkov
Copy link
Contributor

bors try

bors bot added a commit that referenced this pull request Oct 16, 2021
@bors
Try #2463:
15cbf62
@bors
Copy link

bors bot commented Oct 16, 2021

try

Build failed:

@nkryuchkov
Copy link
Contributor

try

Build failed:

late-nodes still fails

@dshulyak
Copy link
Contributor

@nkryuchkov what is the point of merging this? this code is replaced in my change

@nkryuchkov
Copy link
Contributor

Will be overridden by #2801

@nkryuchkov nkryuchkov closed this Oct 18, 2021
@fasmat fasmat deleted the inbound-gossip-patch branch November 7, 2022 08:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dshulyak dshulyak dshulyak requested changes

@nkryuchkov nkryuchkov nkryuchkov approved these changes

@countvonzero countvonzero countvonzero approved these changes

@antonlerner antonlerner Awaiting requested review from antonlerner

Assignees

@nkryuchkov nkryuchkov

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
6 participants
@lrettig @dshulyak @countvonzero @nkryuchkov @antonlerner @moshababo