[Merged by Bors] - continue processing a layer even when only one hash has blocks by countvonzero · Pull Request #2464 · spacemeshos/go-spacemesh

countvonzero · 2021-06-18T08:16:06Z

Motivation

the code, as is, doesn't work when we receive layer blocks in the following order
t1: for layer hash1, we receive valid block data (err == nil)
t2: for layer hash2, we receive error
sync.getLayerFromNeighbors() will determine there is error for this layer and stop processing.

Changes

make sync process a layer as long as there is a layer hash that returns non-empty blocks
when there is an empty layer hash, and all other layer hashes return errors, determine this layer as empty

Test Plan

added several unit tests. in particular, Test_notifyLayerPromiseResult_OneHasBlockData() fails before this change.

DevOps Notes

This PR does not require configuration changes (e.g., environment variables, GitHub secrets, VM resources)
This PR does not affect public APIs
This PR does not rely on a new version of external services (PoET, elasticsearch, etc.)
This PR does not make changes to log messages (which monitoring infrastructure may rely on)

lrettig · 2021-06-21T14:15:13Z

Please see these related changes from the testnet branch: https://github.com/spacemeshos/go-spacemesh/pull/2449/files#diff-14565cc7a7d654724cd76622a3373579a10b2ef86d8d2e52787d7f0c46089b87L1142-R1170

lrettig

Could you help me understand why a single zero layer result trumps even many other non-zero layer results? I'm not totally convinced this is the correct logic

layerfetcher/layers.go

layerfetcher/layers_test.go

countvonzero · 2021-06-21T14:45:14Z

Could you help me understand why a single zero layer result trumps even many other non-zero layer results? I'm not totally convinced this is the correct logic

@lrettig that's not the logic. the logic is, if there is any valid non-zero layer results, it returns immediately. see line 355.
[line 355]. after you exit the loop, you are sure all that's left are errors. and if there is any ErrZeroLayer, it trumps all other types of errors.

countvonzero · 2021-06-21T14:55:21Z

Could you help me understand why a single zero layer result trumps even many other non-zero layer results? I'm not totally convinced this is the correct logic

@lrettig this is exactly i am fixing. the test i added Test_notifyLayerPromiseResult_OneHasBlockData(). this test fails before this change and passes afterwards.

lrettig

thanks for the explanation!

lrettig · 2021-06-25T02:46:19Z

layerfetcher/layers.go

 	//if we failed getting layer data - notify
 	if extErr != nil {
-		l.log.Error("received error for layer id %v", extErr)
+		l.log.Error("received error %v for layer id %v", extErr, layer)


Suggested change

l.log.Error("received error %v for layer id %v", extErr, layer)

l.log.With().Error("received error", log.Err(extErr), layer)

countvonzero · 2021-06-25T08:29:36Z

bors merge

## Motivation the code, as is, doesn't work when we receive layer blocks in the following order t1: for layer hash1, we receive valid block data (err == nil) t2: for layer hash2, we receive error sync.getLayerFromNeighbors() will determine there is error for this layer and stop processing. ## Changes - make sync process a layer as long as there is a layer hash that returns non-empty blocks - when there is an empty layer hash, and all other layer hashes return errors, determine this layer as empty ## Test Plan added several unit tests. in particular, `Test_notifyLayerPromiseResult_OneHasBlockData()` fails before this change. ## DevOps Notes  - [x] This PR does not require configuration changes (e.g., environment variables, GitHub secrets, VM resources) - [x] This PR does not affect public APIs - [x] This PR does not rely on a new version of external services (PoET, elasticsearch, etc.) - [x] This PR does not make changes to log messages (which monitoring infrastructure may rely on)

bors · 2021-06-25T09:17:30Z

Build failed:

ci-stage2

countvonzero · 2021-06-25T12:57:07Z

bors merge

## Motivation the code, as is, doesn't work when we receive layer blocks in the following order t1: for layer hash1, we receive valid block data (err == nil) t2: for layer hash2, we receive error sync.getLayerFromNeighbors() will determine there is error for this layer and stop processing. ## Changes - make sync process a layer as long as there is a layer hash that returns non-empty blocks - when there is an empty layer hash, and all other layer hashes return errors, determine this layer as empty ## Test Plan added several unit tests. in particular, `Test_notifyLayerPromiseResult_OneHasBlockData()` fails before this change. ## DevOps Notes  - [x] This PR does not require configuration changes (e.g., environment variables, GitHub secrets, VM resources) - [x] This PR does not affect public APIs - [x] This PR does not rely on a new version of external services (PoET, elasticsearch, etc.) - [x] This PR does not make changes to log messages (which monitoring infrastructure may rely on)

bors · 2021-06-25T13:46:19Z

Build failed:

ci-stage2

countvonzero · 2021-06-25T14:19:08Z

bors merge

## Motivation the code, as is, doesn't work when we receive layer blocks in the following order t1: for layer hash1, we receive valid block data (err == nil) t2: for layer hash2, we receive error sync.getLayerFromNeighbors() will determine there is error for this layer and stop processing. ## Changes - make sync process a layer as long as there is a layer hash that returns non-empty blocks - when there is an empty layer hash, and all other layer hashes return errors, determine this layer as empty ## Test Plan added several unit tests. in particular, `Test_notifyLayerPromiseResult_OneHasBlockData()` fails before this change. ## DevOps Notes  - [x] This PR does not require configuration changes (e.g., environment variables, GitHub secrets, VM resources) - [x] This PR does not affect public APIs - [x] This PR does not rely on a new version of external services (PoET, elasticsearch, etc.) - [x] This PR does not make changes to log messages (which monitoring infrastructure may rely on)

bors · 2021-06-25T15:06:09Z

Pull request successfully merged into develop.

Build succeeded:

ci-stage2

countvonzero requested review from antonlerner and lrettig June 18, 2021 08:16

countvonzero changed the title ~~mark layer non-empty even when only one hash has blocks~~ continue processing a layer even when only one hash has blocks Jun 18, 2021

antonlerner approved these changes Jun 21, 2021

View reviewed changes

countvonzero force-pushed the sync-zero-block-layer branch from c765f37 to af71898 Compare June 21, 2021 14:15

lrettig suggested changes Jun 21, 2021

View reviewed changes

layerfetcher/layers.go Show resolved Hide resolved

layerfetcher/layers_test.go Outdated Show resolved Hide resolved

layerfetcher/layers_test.go Outdated Show resolved Hide resolved

lrettig approved these changes Jun 25, 2021

View reviewed changes

kimmy lin added 3 commits June 25, 2021 16:02

mark layer non-empty even there is only one hash with blocks

dd17014

address code review feedback

4a9829c

code review feedback #2

dc386b9

countvonzero force-pushed the sync-zero-block-layer branch from c192907 to dc386b9 Compare June 25, 2021 08:04

bors bot changed the title ~~continue processing a layer even when only one hash has blocks~~ [Merged by Bors] - continue processing a layer even when only one hash has blocks Jun 25, 2021

bors bot closed this Jun 25, 2021

bors bot deleted the sync-zero-block-layer branch June 25, 2021 15:06

	l.log.Error("received error %v for layer id %v", extErr, layer)
	l.log.With().Error("received error", log.Err(extErr), layer)

Conversation

countvonzero commented Jun 18, 2021

Motivation

Changes

Test Plan

DevOps Notes

Uh oh!

lrettig commented Jun 21, 2021

Uh oh!

lrettig left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

countvonzero commented Jun 21, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

countvonzero commented Jun 21, 2021

Uh oh!

lrettig left a comment

Choose a reason for hiding this comment

Uh oh!

lrettig Jun 25, 2021

Choose a reason for hiding this comment

Uh oh!

countvonzero commented Jun 25, 2021

Uh oh!

bors bot commented Jun 25, 2021

Uh oh!

countvonzero commented Jun 25, 2021

Uh oh!

bors bot commented Jun 25, 2021

Uh oh!

countvonzero commented Jun 25, 2021

Uh oh!

bors bot commented Jun 25, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

countvonzero commented Jun 21, 2021 •

edited

Loading