[Merged by Bors] - Drop peers when outbound message queue is full by lrettig · Pull Request #2450 · spacemeshos/go-spacemesh

lrettig · 2021-06-10T17:02:29Z

Motivation

Closely related to #2435
Partially addresses #1868, #2386 (and maybe #2405)

Changes

When the outbound message queue for a given connection is full, rather than blocking on Send, begin dropping messages and printing errors (rather than allowing the blocking to propagate up the gossip stack)

It would be good to go one step further and drop peers that are hanging, but this is a quick, simple fix for the acute issue we're seeing on the testnets. See #2385.

Test Plan

Includes a new unit test for both conn and msgconn types

TODO

Explain motivation or link existing issue(s)
Test changes and document test plan
Update documentation as needed

DevOps Notes

This PR does not require configuration changes (e.g., environment variables, GitHub secrets, VM resources)
This PR does not affect public APIs
This PR does not rely on a new version of external services (PoET, elasticsearch, etc.)
This PR does not make changes to log messages (which monitoring infrastructure may rely on)

Prevents blocking further up the gossip stack

dshulyak · 2021-08-11T04:16:10Z

This queue is not only for broadcast, but also for direct requests (msg server and fetcher), is it the case? If so we can't simply drop messages, we must disconnect peer and return with error.

lrettig · 2021-08-23T17:22:51Z

This queue is not only for broadcast, but also for direct requests (msg server and fetcher), is it the case? If so we can't simply drop messages, we must disconnect peer and return with error.

This is correct. I updated this to drop peers if the outbound queue is full. Could you please take a look and see if this is what you had in mind?

dshulyak · 2021-08-24T05:28:00Z

bors try

lrettig · 2021-08-29T20:37:21Z

bors merge

## Motivation Closely related to #2435 Partially addresses #1868, #2386 (and maybe #2405) ## Changes When the outbound message queue for a given connection is full, rather than blocking on `Send`, begin dropping messages and printing errors (rather than allowing the blocking to propagate up the gossip stack) It would be good to go one step further and drop peers that are hanging, but this is a quick, simple fix for the acute issue we're seeing on the testnets. See #2385. ## Test Plan Includes a new unit test for both `conn` and `msgconn` types ## TODO  - [x] Explain motivation or link existing issue(s) - [x] Test changes and document test plan - [ ] Update documentation as needed ## DevOps Notes  - [x] This PR does not require configuration changes (e.g., environment variables, GitHub secrets, VM resources) - [x] This PR does not affect public APIs - [x] This PR does not rely on a new version of external services (PoET, elasticsearch, etc.) - [x] This PR does not make changes to log messages (which monitoring infrastructure may rely on)

bors · 2021-08-29T21:35:26Z

Build failed (retrying...):

ci-stage2

## Motivation Closely related to #2435 Partially addresses #1868, #2386 (and maybe #2405) ## Changes When the outbound message queue for a given connection is full, rather than blocking on `Send`, begin dropping messages and printing errors (rather than allowing the blocking to propagate up the gossip stack) It would be good to go one step further and drop peers that are hanging, but this is a quick, simple fix for the acute issue we're seeing on the testnets. See #2385. ## Test Plan Includes a new unit test for both `conn` and `msgconn` types ## TODO  - [x] Explain motivation or link existing issue(s) - [x] Test changes and document test plan - [ ] Update documentation as needed ## DevOps Notes  - [x] This PR does not require configuration changes (e.g., environment variables, GitHub secrets, VM resources) - [x] This PR does not affect public APIs - [x] This PR does not rely on a new version of external services (PoET, elasticsearch, etc.) - [x] This PR does not make changes to log messages (which monitoring infrastructure may rely on)

bors · 2021-08-29T22:25:37Z

Build failed:

ci-stage2

lrettig · 2021-08-30T01:49:41Z

bors merge

## Motivation Closely related to #2435 Partially addresses #1868, #2386 (and maybe #2405) ## Changes When the outbound message queue for a given connection is full, rather than blocking on `Send`, begin dropping messages and printing errors (rather than allowing the blocking to propagate up the gossip stack) It would be good to go one step further and drop peers that are hanging, but this is a quick, simple fix for the acute issue we're seeing on the testnets. See #2385. ## Test Plan Includes a new unit test for both `conn` and `msgconn` types ## TODO  - [x] Explain motivation or link existing issue(s) - [x] Test changes and document test plan - [ ] Update documentation as needed ## DevOps Notes  - [x] This PR does not require configuration changes (e.g., environment variables, GitHub secrets, VM resources) - [x] This PR does not affect public APIs - [x] This PR does not rely on a new version of external services (PoET, elasticsearch, etc.) - [x] This PR does not make changes to log messages (which monitoring infrastructure may rely on)

bors · 2021-08-30T02:05:56Z

Build failed:

ci-stage2

lrettig · 2021-08-30T02:09:26Z

bors merge

## Motivation Closely related to #2435 Partially addresses #1868, #2386 (and maybe #2405) ## Changes When the outbound message queue for a given connection is full, rather than blocking on `Send`, begin dropping messages and printing errors (rather than allowing the blocking to propagate up the gossip stack) It would be good to go one step further and drop peers that are hanging, but this is a quick, simple fix for the acute issue we're seeing on the testnets. See #2385. ## Test Plan Includes a new unit test for both `conn` and `msgconn` types ## TODO  - [x] Explain motivation or link existing issue(s) - [x] Test changes and document test plan - [ ] Update documentation as needed ## DevOps Notes  - [x] This PR does not require configuration changes (e.g., environment variables, GitHub secrets, VM resources) - [x] This PR does not affect public APIs - [x] This PR does not rely on a new version of external services (PoET, elasticsearch, etc.) - [x] This PR does not make changes to log messages (which monitoring infrastructure may rely on)

bors · 2021-08-30T03:07:52Z

Build failed (retrying...):

ci-stage2

## Motivation Closely related to #2435 Partially addresses #1868, #2386 (and maybe #2405) ## Changes When the outbound message queue for a given connection is full, rather than blocking on `Send`, begin dropping messages and printing errors (rather than allowing the blocking to propagate up the gossip stack) It would be good to go one step further and drop peers that are hanging, but this is a quick, simple fix for the acute issue we're seeing on the testnets. See #2385. ## Test Plan Includes a new unit test for both `conn` and `msgconn` types ## TODO  - [x] Explain motivation or link existing issue(s) - [x] Test changes and document test plan - [ ] Update documentation as needed ## DevOps Notes  - [x] This PR does not require configuration changes (e.g., environment variables, GitHub secrets, VM resources) - [x] This PR does not affect public APIs - [x] This PR does not rely on a new version of external services (PoET, elasticsearch, etc.) - [x] This PR does not make changes to log messages (which monitoring infrastructure may rely on)

bors · 2021-08-30T04:08:09Z

Build failed:

ci-stage2

lrettig · 2021-08-30T17:43:18Z

This is done and ready to be merged, but stuck on flaky tests. Assigning to @antonlerner to make sure it gets merged. Let me know if I've missed anything or if there are any questions. Thanks!

dshulyak · 2021-09-02T06:49:33Z

bors try

bors · 2021-09-02T06:51:35Z

try

Build failed:

ci-stage2

Add coinbase addr back to StartSmeshing

lrettig · 2021-09-02T19:25:12Z

bors merge

## Motivation Closely related to #2435 Partially addresses #1868, #2386 (and maybe #2405) ## Changes When the outbound message queue for a given connection is full, rather than blocking on `Send`, begin dropping messages and printing errors (rather than allowing the blocking to propagate up the gossip stack) It would be good to go one step further and drop peers that are hanging, but this is a quick, simple fix for the acute issue we're seeing on the testnets. See #2385. ## Test Plan Includes a new unit test for both `conn` and `msgconn` types ## TODO  - [x] Explain motivation or link existing issue(s) - [x] Test changes and document test plan - [ ] Update documentation as needed ## DevOps Notes  - [x] This PR does not require configuration changes (e.g., environment variables, GitHub secrets, VM resources) - [x] This PR does not affect public APIs - [x] This PR does not rely on a new version of external services (PoET, elasticsearch, etc.) - [x] This PR does not make changes to log messages (which monitoring infrastructure may rely on) Co-authored-by: Dmitry Shulyak <yashulyak@gmail.com>

bors · 2021-09-02T21:27:59Z

Pull request successfully merged into develop.

Build succeeded:

ci-stage2

Drop outbound messages when msgcon queue full

62fc540

Prevents blocking further up the gossip stack

lrettig added P2P labels Jun 10, 2021

lrettig requested a review from y0sher June 10, 2021 17:02

lrettig mentioned this pull request Jun 10, 2021

[Merged by Bors] - Don't hold mutex while sending on connection #2435

Closed

7 tasks

lrettig requested a review from moshababo June 10, 2021 17:41

This was referenced Jun 23, 2021

Late hare messages #2386

Closed

Single failed message postpones other messages #1868

Closed

moshababo assigned lrettig Jun 24, 2021

lrettig added this to the July 5 Sprint milestone Jul 5, 2021

lrettig modified the milestones: July 5 Sprint, July 19 Sprint Jul 17, 2021

lrettig modified the milestones: July 19 Sprint, Aug 2 Sprint Aug 2, 2021

lrettig mentioned this pull request Aug 10, 2021

One slow (disconnected) peer will slow down everyone else #2647

Closed

moshababo modified the milestones: Aug 2 Sprint, Aug 16 Sprint Aug 16, 2021

lrettig modified the milestones: Aug 16 Sprint, Aug 30 Sprint Aug 16, 2021

lrettig mentioned this pull request Aug 16, 2021

Priority queue length > 50 #2681

Closed

lrettig added 2 commits August 23, 2021 13:06

Merge branch 'develop' into drop-p2p-messages

617352e

Drop peers when outbound queue is full

1699c75

lrettig requested a review from dshulyak August 23, 2021 17:22

lrettig changed the title ~~Drop outbound messages when msgcon queue full~~ Drop peers when outbound message queue is full Aug 23, 2021

Don't overwhelm outbound msg queue in test

55fd19b

lrettig assigned antonlerner and unassigned lrettig Aug 30, 2021

lrettig modified the milestones: Aug 16 Sprint, Aug 30 Sprint Aug 30, 2021

Merge branch 'develop' into drop-p2p-messages

4b83ca9

bors bot added a commit that referenced this pull request Sep 2, 2021

Try #2450:

2a5a946

lrettig added 2 commits September 2, 2021 13:16

Merge branch 'develop' into drop-p2p-messages

12573c1

Resolve merge conflict

620712e

Add coinbase addr back to StartSmeshing

bors bot changed the title ~~Drop peers when outbound message queue is full~~ [Merged by Bors] - Drop peers when outbound message queue is full Sep 2, 2021

bors bot closed this Sep 2, 2021

bors bot deleted the drop-p2p-messages branch September 2, 2021 21:28

Conversation

lrettig commented Jun 10, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Changes

Test Plan

TODO

DevOps Notes

Uh oh!

dshulyak commented Aug 11, 2021

Uh oh!

lrettig commented Aug 23, 2021

Uh oh!

dshulyak commented Aug 24, 2021

Uh oh!

lrettig commented Aug 29, 2021

Uh oh!

bors bot commented Aug 29, 2021

Uh oh!

bors bot commented Aug 29, 2021

Uh oh!

lrettig commented Aug 30, 2021

Uh oh!

bors bot commented Aug 30, 2021

Uh oh!

lrettig commented Aug 30, 2021

Uh oh!

bors bot commented Aug 30, 2021

Uh oh!

bors bot commented Aug 30, 2021

Uh oh!

lrettig commented Aug 30, 2021

Uh oh!

dshulyak commented Sep 2, 2021

Uh oh!

bors bot commented Sep 2, 2021

try

Uh oh!

lrettig commented Sep 2, 2021

Uh oh!

bors bot commented Sep 2, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

lrettig commented Jun 10, 2021 •

edited

Loading