Bound Initial Sync Cache Size #4844

nisdas · 2020-02-12T17:49:25Z

Resolves #4813

Only Cache Boundary States instead of all unfinalized states.
Enforce Cache Limit so that states are flushed from memory to disk, in the
event caches are full.
Filter out boundary candidate states at the start of each epoch.
Use memory pool for Randao mixes field.
Add flag to toggle with garbage collector.

codecov · 2020-02-12T18:00:48Z

Codecov Report

❗ No coverage uploaded for pull request base (master@2da4350). Click here to learn what that means.
The diff coverage is 0%.

@@            Coverage Diff            @@
##             master    #4844   +/-   ##
=========================================
  Coverage          ?   43.48%           
=========================================
  Files             ?      227           
  Lines             ?    17016           
  Branches          ?        0           
=========================================
  Hits              ?     7399           
  Misses            ?     8366           
  Partials          ?     1251

This reverts commit 090d962.

terencechain · 2020-02-12T20:46:24Z

beacon-chain/blockchain/process_block.go

+	if featureconfig.Get().InitSyncCacheState {
+		numOfStates := len(s.initSyncState)
+		if numOfStates > initialSyncCacheSize {
+			stateSlice := make([][32]byte, 0, numOfStates)


Can you please refactor this into its own function?

terencechain · 2020-02-12T20:46:32Z

beacon-chain/blockchain/process_block.go

+		numOfStates := len(s.initSyncState)
+		if numOfStates > initialSyncCacheSize {
+			stateSlice := make([][32]byte, 0, numOfStates)
+			for rt := range s.initSyncState {


… into OOMFix

Co-Authored-By: Raul Jordan <raul@prysmaticlabs.com>

… into OOMFix

terencechain · 2020-02-18T00:33:41Z

beacon-chain/stategen/service.go

@@ -16,3 +22,21 @@ func New(db db.NoHeadAccessDatabase) *State {
 		beaconDB: db,
 	}
 }
+
+// GenerateState generates a state from the provided pre-state till the provided block.
+func (s *State) GenerateState(ctx context.Context, preState *stateTrie.BeaconState, endBlock *ethpb.SignedBeaconBlock) (*stateTrie.BeaconState, error) {


can you put this back in block chain service.go, you can combine it with: https://github.com/prysmaticlabs/prysm/pull/4844/files#diff-c2ffc98dd863a51dc11c4843367b82b6R160

this API pattern with preState and endBlock seem to be only serving block chain service specific use case, because of that, we shouldn't put it here.

I'll export loadBlocks and replayBlocks via an interface for you

Wouldn't something like this also be useful for other services, ex: Sync,etc ? You can generate a new state from a provided pre-state

I agree, but we don't know what the API is for that yet. Example, it may want to use endSlot instead of endBlock. I think until sync actually needs it / uses it. It's better to constraint this within the block chains service.
Or else it may introduce disparity where chain service uses one API to generate state and sync service uses another API to generate state and both APIs reside in stategen service

terencechain · 2020-02-18T00:34:37Z

beacon-chain/blockchain/init_sync_process_block.go

+	if endBlock == nil {
+		return nil, errors.New("provided block root does not have block saved in the db")
+	}
+	stGen := stategen.New(s.beaconDB)


can you initialize this properly in node.go and pass it down to the service struct so stGen is part of *Service?

… into OOMFix

rauljordan

This is really nice, thanks @nisdas

beacon-chain/blockchain/init_sync_process_block.go

terencechain

Lgtm. Thank you for addressing the feedbacks

* bound initial sync * fix lint * Revert "Better block attestation inclusion (prysmaticlabs#4838)" This reverts commit 090d962. * add memory pool * more fixes * revert changes * add hack * revert hack * push halving * bring back hack * increase cache size * more fixes * more changes * new fixes * add test * add reverse test * more tests and clean up * add helper * more cleanup and tests * fix test * remove code * set gc percent flag * lint * lint * Fix comment formatting * Fix some formatting * inverse if statement * remove debug log * Apply suggestions from code review Co-Authored-By: Ivan Martinez <ivanthegreatdev@gmail.com> * Update beacon-chain/state/getters.go Co-Authored-By: Ivan Martinez <ivanthegreatdev@gmail.com> * Update beacon-chain/db/kv/state.go * integrate state generator * gaz * fixes * terence's review * reduce bound further * fix test * separate into new files * gaz * mod build file * add test * revert changes * fix test * Update beacon-chain/core/helpers/slot_epoch.go Co-Authored-By: terence tsao <terence@prysmaticlabs.com> * handle edge case * add back test * fix test again * handle edge case * Update beacon-chain/blockchain/init_sync_process_block.go * Update beacon-chain/blockchain/init_sync_process_block.go * Update beacon-chain/stategen/service_test.go Co-Authored-By: Raul Jordan <raul@prysmaticlabs.com> * Update beacon-chain/blockchain/init_sync_process_block.go Co-Authored-By: Raul Jordan <raul@prysmaticlabs.com> * Update beacon-chain/stategen/service.go Co-Authored-By: Raul Jordan <raul@prysmaticlabs.com> * Update beacon-chain/stategen/service.go Co-Authored-By: Raul Jordan <raul@prysmaticlabs.com> * raul's review * raul's review * fix refs * terence's review * one more fix * Update beacon-chain/blockchain/init_sync_process_block.go Co-authored-by: Raul Jordan <raul@prysmaticlabs.com> Co-authored-by: Ivan Martinez <ivanthegreatdev@gmail.com> Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com> Co-authored-by: terence tsao <terence@prysmaticlabs.com>

nisdas added 2 commits February 13, 2020 01:44

bound initial sync

6a60325

Merge branch 'master' into OOMFix

ba918e3

rauljordan added 2 commits February 12, 2020 14:06

Merge branch 'OOMFix' of github.com:prysmaticlabs/prysm into OOMFix

54a51c3

fix lint

ef8b9a4

rauljordan marked this pull request as ready for review February 12, 2020 20:08

Revert "Better block attestation inclusion (#4838)"

dc489e9

This reverts commit 090d962.

rauljordan previously approved these changes Feb 12, 2020

View reviewed changes

rauljordan added the OK to merge label Feb 12, 2020

terencechain requested changes Feb 12, 2020

View reviewed changes

rauljordan removed the OK to merge label Feb 12, 2020

add memory pool

dc5176d

nisdas dismissed rauljordan’s stale review via dc5176d February 13, 2020 10:58

nisdas added 17 commits February 13, 2020 20:49

more fixes

a8afaa3

Merge branch 'master' of https://github.com/prysmaticlabs/geth-sharding…

dfbe81a

… into OOMFix

revert changes

386668a

add hack

bb103b0

revert hack

66aefaa

push halving

026415c

bring back hack

3b7258a

Merge branch 'master' of https://github.com/prysmaticlabs/geth-sharding…

17d1d11

… into OOMFix

increase cache size

771eca3

more fixes

afa5074

more changes

9775983

new fixes

3ff5133

Merge branch 'master' of https://github.com/prysmaticlabs/geth-sharding…

ee7bc62

… into OOMFix

add test

d034213

add reverse test

56f9d1c

more tests and clean up

478c495

add helper

6556570

prylabs-bulldozer bot added 2 commits February 17, 2020 22:13

Merge refs/heads/master into OOMFix

19b13d5

Merge refs/heads/master into OOMFix

f2407d4

This was referenced Feb 17, 2020

Use Memory Pool for Randao Mixes #4896

Merged

Utilise a Flag to Toggle With the GC #4897

Merged

nisdas and others added 8 commits February 18, 2020 08:01

Update beacon-chain/stategen/service_test.go

564f3b1

Co-Authored-By: Raul Jordan <raul@prysmaticlabs.com>

Update beacon-chain/blockchain/init_sync_process_block.go

17ded5e

Co-Authored-By: Raul Jordan <raul@prysmaticlabs.com>

Update beacon-chain/stategen/service.go

330138f

Co-Authored-By: Raul Jordan <raul@prysmaticlabs.com>

Update beacon-chain/stategen/service.go

21f56b1

Co-Authored-By: Raul Jordan <raul@prysmaticlabs.com>

raul's review

2f2aa2a

Merge branch 'OOMFix' of https://github.com/prysmaticlabs/geth-sharding…

499f672

… into OOMFix

raul's review

dd096b5

Merge branch 'master' of https://github.com/prysmaticlabs/geth-sharding…

b0ebc75

… into OOMFix

terencechain reviewed Feb 18, 2020

View reviewed changes

prylabs-bulldozer bot and others added 7 commits February 18, 2020 00:37

Merge refs/heads/master into OOMFix

454113d

fix refs

18c5f27

Merge refs/heads/master into OOMFix

e0d1a26

Merge refs/heads/master into OOMFix

f1f571a

Merge branch 'master' of https://github.com/prysmaticlabs/geth-sharding…

1ea5d24

… into OOMFix

terence's review

e5ab1fc

one more fix

0276ea4

rauljordan previously approved these changes Feb 18, 2020

View reviewed changes

terencechain reviewed Feb 18, 2020

View reviewed changes

beacon-chain/blockchain/init_sync_process_block.go Outdated Show resolved Hide resolved

Update beacon-chain/blockchain/init_sync_process_block.go

2da4350

terencechain dismissed rauljordan’s stale review via 2da4350 February 18, 2020 16:24

terencechain approved these changes Feb 18, 2020

View reviewed changes

Merge refs/heads/master into OOMFix

8c67eed

rauljordan merged commit 655f57e into master Feb 18, 2020

delete-merged-branch bot deleted the OOMFix branch February 18, 2020 21:11

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bound Initial Sync Cache Size #4844

Bound Initial Sync Cache Size #4844

nisdas commented Feb 12, 2020 •

edited

codecov bot commented Feb 12, 2020 •

edited

terencechain Feb 12, 2020

terencechain Feb 12, 2020

terencechain Feb 18, 2020

nisdas Feb 18, 2020

terencechain Feb 18, 2020 •

edited

terencechain Feb 18, 2020

rauljordan left a comment

terencechain left a comment

Bound Initial Sync Cache Size #4844

Bound Initial Sync Cache Size #4844

Conversation

nisdas commented Feb 12, 2020 • edited

codecov bot commented Feb 12, 2020 • edited

Codecov Report

terencechain Feb 12, 2020

Choose a reason for hiding this comment

terencechain Feb 12, 2020

Choose a reason for hiding this comment

terencechain Feb 18, 2020

Choose a reason for hiding this comment

nisdas Feb 18, 2020

Choose a reason for hiding this comment

terencechain Feb 18, 2020 • edited

Choose a reason for hiding this comment

terencechain Feb 18, 2020

Choose a reason for hiding this comment

rauljordan left a comment

Choose a reason for hiding this comment

terencechain left a comment

Choose a reason for hiding this comment

nisdas commented Feb 12, 2020 •

edited

codecov bot commented Feb 12, 2020 •

edited

terencechain Feb 18, 2020 •

edited