Runtime Optimizations #4648

nisdas · 2020-01-25T02:45:28Z

We parallelize shuffling, by splitting the indices into chunks and then letting those chunks be shuffled by separate goroutines using Scatter.
Add validator callback methods to allow reading from and writing to each validator in the scope
of the provided callback.
Use callbacks to minimize registry clones, and instead directly access validators.
Use 32 byte arrays instead of slices in HTR, so as to allow for easier load on the GC when cleaning the object up.
Reduce copying, instead re-using allocated memory when appropriate.
Fix a cache bug with our exit epoch root in our validator object in HTR.

protolambda · 2020-01-25T12:03:35Z

Why shuffle every committee validator one by one? This has been optimized for almost a year now. Check https://github.com/protolambda/eth2-shuffle
It shares hashes between bits, to effectively get a 256 times speed up. So it only takes a second, synchronously, to get the committees of an epoch, with 4,000,000 validators.
And if you want to make it even faster, you can compute the permutations for the 90 rounds in parallel, and then fold them into the final shuffling

nisdas · 2020-01-25T12:15:49Z

@protolambda Thanks for bringing this up , just realised that for some reason we changed to the un-optimized algorithm one instead of the optimized one here. https://github.com/prysmaticlabs/prysm/blob/master/beacon-chain/core/helpers/shuffle.go#L154

Fixing this right now.

terencechain

Minor comments, lint.. etc

terencechain · 2020-01-25T14:30:27Z

beacon-chain/state/setters.go

@@ -180,6 +180,19 @@ func (b *BeaconState) SetValidators(val []*ethpb.Validator) error {
 	return nil
 }

+func (b *BeaconState) ApplyToEveryValidator(f func(idx int, val *ethpb.Validator) error) error {


terencechain · 2020-01-25T14:30:58Z

beacon-chain/state/getters.go

@@ -268,6 +268,18 @@ func (b *BeaconState) NumofValidators() int {
 	return len(b.state.Validators)
 }

+// ReadFromEveryValidator reads values from every validator and applies it to the provided function.
+// Warning: This method is potentially unsafe, as it exposes the actual validator registry.


This unsafe method worries me, is there no other way?

I am not sure if there is a better way. The alternative is to basically perform a clone of the whole registry. I could remove this, but we will take a performance hit in epoch transitions

Is the performance hit noticeable?

It is unfortunately, it would be multiple registry clones

Not a fan of this read from every func because of side effects. Can we just read state.Validators() in the state transition and pass that down to funcs that need to read it?

That has the potential to lead to an invalid state transition like we saw before. It would be risky passing down the same registry slice, to multiple functions that write to it. I have another way, that would be more verbose but safer than this one. I will try to test it out and see if it works well

terencechain · 2020-01-25T14:31:21Z

beacon-chain/core/epoch/epoch_processing.go

-	for index, validator := range vals {
-		correctEpoch := (currentEpoch + exitLength/2) == validator.WithdrawableEpoch
-		if validator.Slashed && correctEpoch {
+	validatorFunc := func(idx int, val *ethpb.Validator) error {


Can you add a comment on why call back is used here?

can we instead just do

if err := state.ApplyToEveryValidator(func(idx int, val *ethpb.Validator) error { ... })

terencechain

Please get 1 more approval before merging.

cc: @rauljordan and @prestonvanloon

rauljordan · 2020-01-28T04:45:11Z

beacon-chain/core/helpers/committee.go

-		if IsActiveValidator(v, epoch) {
-			indices = append(indices, uint64(i))
+	indices := make([]uint64, 0, state.NumofValidators())
+	// ignore error as no error is returned in above callback.


Callback above? Is the comment necessary?

that is outdated, removing it

rauljordan · 2020-01-28T04:47:31Z

beacon-chain/state/getters.go

@@ -224,6 +260,20 @@ func (b *BeaconState) Validators() []*ethpb.Validator {
 	return res
 }

+// ValidatorsNoClone returns validators participating in consensus on the beacon chain. This
+// method doesn't clone the respective validators.
+func (b *BeaconState) ValidatorsNoClone() []*Validator {


hmm this is so confusing with the return type and *ethpb.Validator. Not clear to the reader what the difference is. Also, the naming can be improved, it seems unsafe from the outside but is not actually unsafe? Maybe we can rename this type to ValidatorWrapper? im not sure, maybe @prestonvanloon has some thoughts

beacon-chain/state/types.go

rauljordan · 2020-01-28T04:48:25Z

beacon-chain/state/types.go

@@ -24,6 +26,10 @@ type BeaconState struct {
 	merkleLayers [][][]byte
 }

+type Validator struct {
+	validator *ethpb.Validator


Not a fan of this, 3 things that say validator in one same data structure

Co-Authored-By: Raul Jordan <raul@prysmaticlabs.com>

…bs/geth-sharding into parallelizeShuffling

* begin state service * begin on the state trie idea * created beacon state structure * add in the full clone getter * return by value instead * add all setters * new state setters are being completed * arrays roots exposed * close to finishing all these headerssss * functionality complete * added in proto benchmark test * test for compatibility * add test for compat * comments fixed * add clone * add clone * remove underlying copies * make it immutable * integrate it into chainservice * revert * wrap up comments for package * address all comments and godocs * address all comments * clone the pending attestation properly * properly clone remaining items * tests pass fixed bug * begin using it instead of head state * prevent nil pointer exceptions * begin using new struct in db * integrated new type into db package * add proper nil checks * using new state in archiver * refactored much of core * editing all the precompute functions * done with most core refactor * fixed up some bugs in the clone comparisons * append current epoch atts * add missing setters * add new setters * fix other core methods * fix up transition * main service and forkchoice * fix rpc * integrated to powchain * some more changes * fix build * improve processing of deposits * fix error * prevent panic * comment * fix process att * gaz * fix up att process * resolve existing review comments * resolve another batch of gh comments * resolve broken cpt state * revise testutil to use the new state * begin updating the state transition func to pass in more compartmentalized args * finish editing transition function to return errors * block operations pretty much done with refactor * state transition fully refactored * got epoch processing completed * fix build in fork choice * fixing more of the build * fix up broken sync package * it builds nowww it buildssss * revert registry changes * Recompute on Read (#4627) * compute on read * fix up eth1 data votes * looking into slashings bug introduced in core/ * able to advance more slots * add logging * can now sync with testnet yay * remove the leaves algorithm and other merkle imports * expose initialize unsafe funcs * Update beacon-chain/db/kv/state.go * lint Co-authored-by: Raul Jordan <raul@prysmaticlabs.com> * More Optimizations for New State (#4641) * map optimization * more optimizations * use a custom hasher * comment * block operations optimizations * Update beacon-chain/state/types.go Co-Authored-By: Raul Jordan <raul@prysmaticlabs.com> * fixed up various operations to use the validator index map access Co-authored-by: Raul Jordan <raul@prysmaticlabs.com> * archiver tests pass * fixing cache tests * cache tests passing * edited validator tests * powchain tests passing * halfway thru sync tests * more sync test fixes * add in tests for state/ * working through rpc tests * assignments tests passed * almost done with rpc/beacon tests * resolved painful validator test * fixed up even more tests * resolve tests * fix build * reduce a randao mixes copy * fixes under //beacon-chain/blockchain/... * build //beacon-chain/core/... * fixes * Runtime Optimizations (#4648) * parallelize shuffling * clean up * lint * fix build * use callback to read from registry * fix array roots and size map * new improvements * reduce hash allocs * improved shuffling * terence's review * use different method * raul's comment * new array roots * remove clone in pre-compute * Update beacon-chain/state/types.go Co-Authored-By: Raul Jordan <raul@prysmaticlabs.com> * raul's review * lint * fix build issues * fix visibility Co-authored-by: Raul Jordan <raul@prysmaticlabs.com> * fix visibility * build works for all * fix blockchain test * fix a few tests * fix more tests * update validator in slashing * archiver passing * fixed rpc/validator * progress on core tests * resolve broken rpc tests * blockchain tests passed * fix up some tests in core * fix message diff * remove unnecessary save * Save validator after slashing * Update validators one by one * another update * fix everything * fix more precompute tests * fix blocks tests * more elegant fix * more helper fixes * change back ? * fix test * fix skip slot * fix test * reset caches * fix testutil * raceoff fixed * passing * Retrieve cached state in the beginning * lint * Fixed tests part 1 * Fixed rest of the tests * Minor changes to avoid copying, small refactor to reduce deplicated code * Handle att req for slot 0 * New beacon state: Only populate merkle layers as needed, copy merkle layers on copy/clone. (#4689) * Only populate merkle layers as needed, copy merkle layers on copy/clone. * use custom copy * Make maps of correct size * slightly fast, doesn't wait for lock Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com> * Target root can't be 0x00 * Don't use cache for current slot (may not be the right fix) * fixed up tests * Remove some copy for init sync. Not sure if it is safe enough for runtime though... testing... * Align with prev logic for process slots cachedState.Slot() < slot * Fix Initial Sync Flag (#4692) * fixes * fix up some test failures due to lack of nil checks * fix up some test failures due to lack of nil checks * fix up imports * revert some changes * imports Co-authored-by: Raul Jordan <raul@prysmaticlabs.com> * resolving further conflicts * Better skip slot cache (#4694) * Return copy of skip slot cache state, disable skip slot cache on sync * fix * Fix pruning * fix up issues with broken tests Co-authored-by: Nishant Das <nish1993@hotmail.com> Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> Co-authored-by: shayzluf <thezluf@gmail.com> Co-authored-by: terence tsao <terence@prysmaticlabs.com> Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com>

* begin state service * begin on the state trie idea * created beacon state structure * add in the full clone getter * return by value instead * add all setters * new state setters are being completed * arrays roots exposed * close to finishing all these headerssss * functionality complete * added in proto benchmark test * test for compatibility * add test for compat * comments fixed * add clone * add clone * remove underlying copies * make it immutable * integrate it into chainservice * revert * wrap up comments for package * address all comments and godocs * address all comments * clone the pending attestation properly * properly clone remaining items * tests pass fixed bug * begin using it instead of head state * prevent nil pointer exceptions * begin using new struct in db * integrated new type into db package * add proper nil checks * using new state in archiver * refactored much of core * editing all the precompute functions * done with most core refactor * fixed up some bugs in the clone comparisons * append current epoch atts * add missing setters * add new setters * fix other core methods * fix up transition * main service and forkchoice * fix rpc * integrated to powchain * some more changes * fix build * improve processing of deposits * fix error * prevent panic * comment * fix process att * gaz * fix up att process * resolve existing review comments * resolve another batch of gh comments * resolve broken cpt state * revise testutil to use the new state * begin updating the state transition func to pass in more compartmentalized args * finish editing transition function to return errors * block operations pretty much done with refactor * state transition fully refactored * got epoch processing completed * fix build in fork choice * fixing more of the build * fix up broken sync package * it builds nowww it buildssss * revert registry changes * Recompute on Read (#4627) * compute on read * fix up eth1 data votes * looking into slashings bug introduced in core/ * able to advance more slots * add logging * can now sync with testnet yay * remove the leaves algorithm and other merkle imports * expose initialize unsafe funcs * Update beacon-chain/db/kv/state.go * lint Co-authored-by: Raul Jordan <raul@prysmaticlabs.com> * More Optimizations for New State (#4641) * map optimization * more optimizations * use a custom hasher * comment * block operations optimizations * Update beacon-chain/state/types.go Co-Authored-By: Raul Jordan <raul@prysmaticlabs.com> * fixed up various operations to use the validator index map access Co-authored-by: Raul Jordan <raul@prysmaticlabs.com> * archiver tests pass * fixing cache tests * cache tests passing * edited validator tests * powchain tests passing * halfway thru sync tests * more sync test fixes * add in tests for state/ * working through rpc tests * assignments tests passed * almost done with rpc/beacon tests * resolved painful validator test * fixed up even more tests * resolve tests * fix build * reduce a randao mixes copy * fixes under //beacon-chain/blockchain/... * build //beacon-chain/core/... * fixes * Runtime Optimizations (#4648) * parallelize shuffling * clean up * lint * fix build * use callback to read from registry * fix array roots and size map * new improvements * reduce hash allocs * improved shuffling * terence's review * use different method * raul's comment * new array roots * remove clone in pre-compute * Update beacon-chain/state/types.go Co-Authored-By: Raul Jordan <raul@prysmaticlabs.com> * raul's review * lint * fix build issues * fix visibility Co-authored-by: Raul Jordan <raul@prysmaticlabs.com> * fix visibility * build works for all * fix blockchain test * fix a few tests * fix more tests * update validator in slashing * archiver passing * fixed rpc/validator * progress on core tests * resolve broken rpc tests * blockchain tests passed * fix up some tests in core * fix message diff * remove unnecessary save * Save validator after slashing * Update validators one by one * another update * fix everything * fix more precompute tests * fix blocks tests * more elegant fix * more helper fixes * change back ? * fix test * fix skip slot * fix test * reset caches * fix testutil * raceoff fixed * passing * Retrieve cached state in the beginning * lint * Fixed tests part 1 * Fixed rest of the tests * Minor changes to avoid copying, small refactor to reduce deplicated code * Handle att req for slot 0 * New beacon state: Only populate merkle layers as needed, copy merkle layers on copy/clone. (#4689) * Only populate merkle layers as needed, copy merkle layers on copy/clone. * use custom copy * Make maps of correct size * slightly fast, doesn't wait for lock Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com> * Target root can't be 0x00 * Don't use cache for current slot (may not be the right fix) * fixed up tests * Remove some copy for init sync. Not sure if it is safe enough for runtime though... testing... * Align with prev logic for process slots cachedState.Slot() < slot * Fix Initial Sync Flag (#4692) * fixes * fix up some test failures due to lack of nil checks * fix up some test failures due to lack of nil checks * fix up imports * revert some changes * imports Co-authored-by: Raul Jordan <raul@prysmaticlabs.com> * resolving further conflicts * Better skip slot cache (#4694) * Return copy of skip slot cache state, disable skip slot cache on sync * fix * Fix pruning * copy on write method * gaz * fix tests * fix up issues with broken tests * remove extra update * remove debugging lines * gofmt Co-authored-by: Raul Jordan <raul@prysmaticlabs.com> Co-authored-by: Nishant Das <nish1993@hotmail.com> Co-authored-by: shayzluf <thezluf@gmail.com> Co-authored-by: terence tsao <terence@prysmaticlabs.com> Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com>

* begin state service * begin on the state trie idea * created beacon state structure * add in the full clone getter * return by value instead * add all setters * new state setters are being completed * arrays roots exposed * close to finishing all these headerssss * functionality complete * added in proto benchmark test * test for compatibility * add test for compat * comments fixed * Merge branch 'master' into state-service * add clone * add clone * remove underlying copies * make it immutable * integrate it into chainservice * revert * wrap up comments for package * address all comments and godocs * address all comments * Merge branch 'master' into state-service * clone the pending attestation properly * Merge branch 'state-service' of github.com:prysmaticlabs/prysm into state-service * properly clone remaining items * tests pass fixed bug * begin using it instead of head state * prevent nil pointer exceptions * Merge branch 'state-service' into use-state-in-runtime * begin using new struct in db * integrated new type into db package * add proper nil checks * using new state in archiver * refactored much of core * editing all the precompute functions * done with most core refactor * fixed up some bugs in the clone comparisons * Merge branch 'state-service' into use-state-in-runtime * append current epoch atts * merged master * add missing setters * add new setters * fix other core methods * fix up transition * main service and forkchoice * fix rpc * integrated to powchain * some more changes * fix build * improve processing of deposits * fix error * prevent panic * comment * fix process att * gaz * fix up att process * resolve existing review comments * Merge branch 'master' into use-state-in-runtime * resolve another batch of gh comments * resolve broken cpt state * revise testutil to use the new state * begin updating the state transition func to pass in more compartmentalized args * finish editing transition function to return errors * block operations pretty much done with refactor * state transition fully refactored * got epoch processing completed * fix build in fork choice * fixing more of the build * fix up broken sync package * it builds nowww it buildssss * revert registry changes * Recompute on Read (#4627) * compute on read * fix up eth1 data votes * looking into slashings bug introduced in core/ * able to advance more slots * add logging * can now sync with testnet yay * remove the leaves algorithm and other merkle imports * expose initialize unsafe funcs * Update beacon-chain/db/kv/state.go * lint Co-authored-by: Raul Jordan <raul@prysmaticlabs.com> * include master * More Optimizations for New State (#4641) * map optimization * more optimizations * use a custom hasher * comment * block operations optimizations * Update beacon-chain/state/types.go Co-Authored-By: Raul Jordan <raul@prysmaticlabs.com> * fixed up various operations to use the validator index map access Co-authored-by: Raul Jordan <raul@prysmaticlabs.com> * archiver tests pass * fixing cache tests * cache tests passing * edited validator tests * powchain tests passing * halfway thru sync tests * more sync test fixes * add in tests for state/ * working through rpc tests * assignments tests passed * almost done with rpc/beacon tests * resolved painful validator test * fixed up even more tests * resolve tests * fix build * reduce a randao mixes copy * fixes under //beacon-chain/blockchain/... * build //beacon-chain/core/... * fixes * Runtime Optimizations (#4648) * parallelize shuffling * clean up * lint * fix build * use callback to read from registry * fix array roots and size map * new improvements * reduce hash allocs * improved shuffling * terence's review * use different method * raul's comment * new array roots * remove clone in pre-compute * Update beacon-chain/state/types.go Co-Authored-By: Raul Jordan <raul@prysmaticlabs.com> * raul's review * lint * fix build issues * fix visibility Co-authored-by: Raul Jordan <raul@prysmaticlabs.com> * Merge branch 'use-state-in-runtime' of https://github.com/prysmaticlabs/geth-sharding into resolve-tests * fix visibility * build works for all * fix blockchain test * fix a few tests * fix more tests * resolve conf * sync * update validator in slashing * Merge branch 'resolve-tests' of github.com:prysmaticlabs/prysm into resolve-tests * archiver passing * Merge branch 'resolve-tests' of github.com:prysmaticlabs/prysm into resolve-tests * fixed rpc/validator * progress on core tests * Merge branch 'resolve-tests' of github.com:prysmaticlabs/prysm into resolve-tests * resolve broken rpc tests * Merge branch 'resolve-tests' of github.com:prysmaticlabs/prysm into resolve-tests * blockchain tests passed * fix up some tests in core * Merge branch 'master' of github.com:prysmaticlabs/prysm into resolve-tests * fix message diff * remove unnecessary save * Merge branch 'master' of github.com:prysmaticlabs/prysm into resolve-tests * Save validator after slashing * Update validators one by one * another update * fix everything * Merge branch 'resolve-tests' of https://github.com/prysmaticlabs/geth-sharding into resolve-tests * fix more precompute tests * fix blocks tests * Merge branch 'resolve-tests' of github.com:prysmaticlabs/prysm into resolve-tests * more elegant fix * more helper fixes * Merge branch 'resolve-tests' of https://github.com/prysmaticlabs/geth-sharding into resolve-tests * change back ? * fix test * fix skip slot * fix test * reset caches * fix testutil * raceoff fixed * passing * Retrieve cached state in the beginning * lint * Merge branch 'master' of github.com:prysmaticlabs/prysm into resolve-tests * Fixed tests part 1 * Fixed rest of the tests * Merge branch 'master' into optimize-process-att * Minor changes to avoid copying, small refactor to reduce deplicated code * Merge branch 'master' into resolve-tests * Handle att req for slot 0 * New beacon state: Only populate merkle layers as needed, copy merkle layers on copy/clone. (#4689) * Only populate merkle layers as needed, copy merkle layers on copy/clone. * use custom copy * Make maps of correct size * slightly fast, doesn't wait for lock Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com> * Merge branch 'resolve-tests' of github.com:prysmaticlabs/prysm into resolve-tests * Merge branch 'master' into resolve-tests * Target root can't be 0x00 * Merge refs/heads/master into resolve-tests * Merge branch 'resolve-tests' of github.com:prysmaticlabs/prysm into resolve-tests * Don't use cache for current slot (may not be the right fix) * Merge branch 'resolve-tests' of git+ssh://github.com/prysmaticlabs/prysm into resolve-tests * Merge branch 'resolve-tests' of github.com:prysmaticlabs/prysm into resolve-tests * fixed up tests * Remove some copy for init sync. Not sure if it is safe enough for runtime though... testing... * Align with prev logic for process slots cachedState.Slot() < slot * Fix Initial Sync Flag (#4692) * fixes * fix up some test failures due to lack of nil checks * fix up some test failures due to lack of nil checks * fix up imports * revert some changes * imports Co-authored-by: Raul Jordan <raul@prysmaticlabs.com> * Conflict * Conflict * resolve confs * resolving further conflicts * Better skip slot cache (#4694) * Return copy of skip slot cache state, disable skip slot cache on sync * fix * Fix pruning * Merge refs/heads/master into resolve-tests * Merge refs/heads/master into resolve-tests * copy on write method * gaz * fix confs * Merge refs/heads/resolve-tests into copy-on-write * fix tests * Merge branch 'copy-on-write' of github.com:prysmaticlabs/prysm into copy-on-write * fix up issues with broken tests * Merge refs/heads/resolve-tests into copy-on-write * Merge branch 'master' of github.com:prysmaticlabs/prysm into copy-on-write * remove extra update * remove debugging lines * gofmt * Merge refs/heads/master into copy-on-write * Merge refs/heads/master into copy-on-write * Merge refs/heads/master into copy-on-write * Merge branch 'master' of github.com:prysmaticlabs/prysm * Passing 1 to SetEndSlot * Better way to get genesis block * Merge branch 'master' of github.com:prysmaticlabs/prysm into bug/ApiGenesisBlock * Checking for nil genBlk * Testing conditional order * Reverting conditions, and no idea why build fails * Merge branch 'master' into bug/ApiGenesisBlock * Saving to genesis block root in tests * Merge branch 'bug/ApiGenesisBlock' of github.com:AgentJ-WR/prysm into bug/ApiGenesisBlock * Saving root * Merge branch 'master' of github.com:prysmaticlabs/prysm into bug/ApiGenesisBlock * Adding regression test * Updating regression test * Merge branch 'master' into bug/ApiGenesisBlock * Merge branch 'master' into bug/ApiGenesisBlock * Merge branch 'master' into bug/ApiGenesisBlock * Merge branch 'master' into bug/ApiGenesisBlock

* begin state service * begin on the state trie idea * created beacon state structure * add in the full clone getter * return by value instead * add all setters * new state setters are being completed * arrays roots exposed * close to finishing all these headerssss * functionality complete * added in proto benchmark test * test for compatibility * add test for compat * comments fixed * add clone * add clone * remove underlying copies * make it immutable * integrate it into chainservice * revert * wrap up comments for package * address all comments and godocs * address all comments * clone the pending attestation properly * properly clone remaining items * tests pass fixed bug * begin using it instead of head state * prevent nil pointer exceptions * begin using new struct in db * integrated new type into db package * add proper nil checks * using new state in archiver * refactored much of core * editing all the precompute functions * done with most core refactor * fixed up some bugs in the clone comparisons * append current epoch atts * add missing setters * add new setters * fix other core methods * fix up transition * main service and forkchoice * fix rpc * integrated to powchain * some more changes * fix build * improve processing of deposits * fix error * prevent panic * comment * fix process att * gaz * fix up att process * resolve existing review comments * resolve another batch of gh comments * resolve broken cpt state * revise testutil to use the new state * begin updating the state transition func to pass in more compartmentalized args * finish editing transition function to return errors * block operations pretty much done with refactor * state transition fully refactored * got epoch processing completed * fix build in fork choice * fixing more of the build * fix up broken sync package * it builds nowww it buildssss * revert registry changes * Recompute on Read (prysmaticlabs#4627) * compute on read * fix up eth1 data votes * looking into slashings bug introduced in core/ * able to advance more slots * add logging * can now sync with testnet yay * remove the leaves algorithm and other merkle imports * expose initialize unsafe funcs * Update beacon-chain/db/kv/state.go * lint Co-authored-by: Raul Jordan <raul@prysmaticlabs.com> * More Optimizations for New State (prysmaticlabs#4641) * map optimization * more optimizations * use a custom hasher * comment * block operations optimizations * Update beacon-chain/state/types.go Co-Authored-By: Raul Jordan <raul@prysmaticlabs.com> * fixed up various operations to use the validator index map access Co-authored-by: Raul Jordan <raul@prysmaticlabs.com> * archiver tests pass * fixing cache tests * cache tests passing * edited validator tests * powchain tests passing * halfway thru sync tests * more sync test fixes * add in tests for state/ * working through rpc tests * assignments tests passed * almost done with rpc/beacon tests * resolved painful validator test * fixed up even more tests * resolve tests * fix build * reduce a randao mixes copy * fixes under //beacon-chain/blockchain/... * build //beacon-chain/core/... * fixes * Runtime Optimizations (prysmaticlabs#4648) * parallelize shuffling * clean up * lint * fix build * use callback to read from registry * fix array roots and size map * new improvements * reduce hash allocs * improved shuffling * terence's review * use different method * raul's comment * new array roots * remove clone in pre-compute * Update beacon-chain/state/types.go Co-Authored-By: Raul Jordan <raul@prysmaticlabs.com> * raul's review * lint * fix build issues * fix visibility Co-authored-by: Raul Jordan <raul@prysmaticlabs.com> * fix visibility * build works for all * fix blockchain test * fix a few tests * fix more tests * update validator in slashing * archiver passing * fixed rpc/validator * progress on core tests * resolve broken rpc tests * blockchain tests passed * fix up some tests in core * fix message diff * remove unnecessary save * Save validator after slashing * Update validators one by one * another update * fix everything * fix more precompute tests * fix blocks tests * more elegant fix * more helper fixes * change back ? * fix test * fix skip slot * fix test * reset caches * fix testutil * raceoff fixed * passing * Retrieve cached state in the beginning * lint * Fixed tests part 1 * Fixed rest of the tests * Minor changes to avoid copying, small refactor to reduce deplicated code * Handle att req for slot 0 * New beacon state: Only populate merkle layers as needed, copy merkle layers on copy/clone. (prysmaticlabs#4689) * Only populate merkle layers as needed, copy merkle layers on copy/clone. * use custom copy * Make maps of correct size * slightly fast, doesn't wait for lock Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com> * Target root can't be 0x00 * Don't use cache for current slot (may not be the right fix) * fixed up tests * Remove some copy for init sync. Not sure if it is safe enough for runtime though... testing... * Align with prev logic for process slots cachedState.Slot() < slot * Fix Initial Sync Flag (prysmaticlabs#4692) * fixes * fix up some test failures due to lack of nil checks * fix up some test failures due to lack of nil checks * fix up imports * revert some changes * imports Co-authored-by: Raul Jordan <raul@prysmaticlabs.com> * resolving further conflicts * Better skip slot cache (prysmaticlabs#4694) * Return copy of skip slot cache state, disable skip slot cache on sync * fix * Fix pruning * fix up issues with broken tests Co-authored-by: Nishant Das <nish1993@hotmail.com> Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> Co-authored-by: shayzluf <thezluf@gmail.com> Co-authored-by: terence tsao <terence@prysmaticlabs.com> Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com>

) * begin state service * begin on the state trie idea * created beacon state structure * add in the full clone getter * return by value instead * add all setters * new state setters are being completed * arrays roots exposed * close to finishing all these headerssss * functionality complete * added in proto benchmark test * test for compatibility * add test for compat * comments fixed * add clone * add clone * remove underlying copies * make it immutable * integrate it into chainservice * revert * wrap up comments for package * address all comments and godocs * address all comments * clone the pending attestation properly * properly clone remaining items * tests pass fixed bug * begin using it instead of head state * prevent nil pointer exceptions * begin using new struct in db * integrated new type into db package * add proper nil checks * using new state in archiver * refactored much of core * editing all the precompute functions * done with most core refactor * fixed up some bugs in the clone comparisons * append current epoch atts * add missing setters * add new setters * fix other core methods * fix up transition * main service and forkchoice * fix rpc * integrated to powchain * some more changes * fix build * improve processing of deposits * fix error * prevent panic * comment * fix process att * gaz * fix up att process * resolve existing review comments * resolve another batch of gh comments * resolve broken cpt state * revise testutil to use the new state * begin updating the state transition func to pass in more compartmentalized args * finish editing transition function to return errors * block operations pretty much done with refactor * state transition fully refactored * got epoch processing completed * fix build in fork choice * fixing more of the build * fix up broken sync package * it builds nowww it buildssss * revert registry changes * Recompute on Read (prysmaticlabs#4627) * compute on read * fix up eth1 data votes * looking into slashings bug introduced in core/ * able to advance more slots * add logging * can now sync with testnet yay * remove the leaves algorithm and other merkle imports * expose initialize unsafe funcs * Update beacon-chain/db/kv/state.go * lint Co-authored-by: Raul Jordan <raul@prysmaticlabs.com> * More Optimizations for New State (prysmaticlabs#4641) * map optimization * more optimizations * use a custom hasher * comment * block operations optimizations * Update beacon-chain/state/types.go Co-Authored-By: Raul Jordan <raul@prysmaticlabs.com> * fixed up various operations to use the validator index map access Co-authored-by: Raul Jordan <raul@prysmaticlabs.com> * archiver tests pass * fixing cache tests * cache tests passing * edited validator tests * powchain tests passing * halfway thru sync tests * more sync test fixes * add in tests for state/ * working through rpc tests * assignments tests passed * almost done with rpc/beacon tests * resolved painful validator test * fixed up even more tests * resolve tests * fix build * reduce a randao mixes copy * fixes under //beacon-chain/blockchain/... * build //beacon-chain/core/... * fixes * Runtime Optimizations (prysmaticlabs#4648) * parallelize shuffling * clean up * lint * fix build * use callback to read from registry * fix array roots and size map * new improvements * reduce hash allocs * improved shuffling * terence's review * use different method * raul's comment * new array roots * remove clone in pre-compute * Update beacon-chain/state/types.go Co-Authored-By: Raul Jordan <raul@prysmaticlabs.com> * raul's review * lint * fix build issues * fix visibility Co-authored-by: Raul Jordan <raul@prysmaticlabs.com> * fix visibility * build works for all * fix blockchain test * fix a few tests * fix more tests * update validator in slashing * archiver passing * fixed rpc/validator * progress on core tests * resolve broken rpc tests * blockchain tests passed * fix up some tests in core * fix message diff * remove unnecessary save * Save validator after slashing * Update validators one by one * another update * fix everything * fix more precompute tests * fix blocks tests * more elegant fix * more helper fixes * change back ? * fix test * fix skip slot * fix test * reset caches * fix testutil * raceoff fixed * passing * Retrieve cached state in the beginning * lint * Fixed tests part 1 * Fixed rest of the tests * Minor changes to avoid copying, small refactor to reduce deplicated code * Handle att req for slot 0 * New beacon state: Only populate merkle layers as needed, copy merkle layers on copy/clone. (prysmaticlabs#4689) * Only populate merkle layers as needed, copy merkle layers on copy/clone. * use custom copy * Make maps of correct size * slightly fast, doesn't wait for lock Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com> * Target root can't be 0x00 * Don't use cache for current slot (may not be the right fix) * fixed up tests * Remove some copy for init sync. Not sure if it is safe enough for runtime though... testing... * Align with prev logic for process slots cachedState.Slot() < slot * Fix Initial Sync Flag (prysmaticlabs#4692) * fixes * fix up some test failures due to lack of nil checks * fix up some test failures due to lack of nil checks * fix up imports * revert some changes * imports Co-authored-by: Raul Jordan <raul@prysmaticlabs.com> * resolving further conflicts * Better skip slot cache (prysmaticlabs#4694) * Return copy of skip slot cache state, disable skip slot cache on sync * fix * Fix pruning * copy on write method * gaz * fix tests * fix up issues with broken tests * remove extra update * remove debugging lines * gofmt Co-authored-by: Raul Jordan <raul@prysmaticlabs.com> Co-authored-by: Nishant Das <nish1993@hotmail.com> Co-authored-by: shayzluf <thezluf@gmail.com> Co-authored-by: terence tsao <terence@prysmaticlabs.com> Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com>

) * begin state service * begin on the state trie idea * created beacon state structure * add in the full clone getter * return by value instead * add all setters * new state setters are being completed * arrays roots exposed * close to finishing all these headerssss * functionality complete * added in proto benchmark test * test for compatibility * add test for compat * comments fixed * Merge branch 'master' into state-service * add clone * add clone * remove underlying copies * make it immutable * integrate it into chainservice * revert * wrap up comments for package * address all comments and godocs * address all comments * Merge branch 'master' into state-service * clone the pending attestation properly * Merge branch 'state-service' of github.com:prysmaticlabs/prysm into state-service * properly clone remaining items * tests pass fixed bug * begin using it instead of head state * prevent nil pointer exceptions * Merge branch 'state-service' into use-state-in-runtime * begin using new struct in db * integrated new type into db package * add proper nil checks * using new state in archiver * refactored much of core * editing all the precompute functions * done with most core refactor * fixed up some bugs in the clone comparisons * Merge branch 'state-service' into use-state-in-runtime * append current epoch atts * merged master * add missing setters * add new setters * fix other core methods * fix up transition * main service and forkchoice * fix rpc * integrated to powchain * some more changes * fix build * improve processing of deposits * fix error * prevent panic * comment * fix process att * gaz * fix up att process * resolve existing review comments * Merge branch 'master' into use-state-in-runtime * resolve another batch of gh comments * resolve broken cpt state * revise testutil to use the new state * begin updating the state transition func to pass in more compartmentalized args * finish editing transition function to return errors * block operations pretty much done with refactor * state transition fully refactored * got epoch processing completed * fix build in fork choice * fixing more of the build * fix up broken sync package * it builds nowww it buildssss * revert registry changes * Recompute on Read (prysmaticlabs#4627) * compute on read * fix up eth1 data votes * looking into slashings bug introduced in core/ * able to advance more slots * add logging * can now sync with testnet yay * remove the leaves algorithm and other merkle imports * expose initialize unsafe funcs * Update beacon-chain/db/kv/state.go * lint Co-authored-by: Raul Jordan <raul@prysmaticlabs.com> * include master * More Optimizations for New State (prysmaticlabs#4641) * map optimization * more optimizations * use a custom hasher * comment * block operations optimizations * Update beacon-chain/state/types.go Co-Authored-By: Raul Jordan <raul@prysmaticlabs.com> * fixed up various operations to use the validator index map access Co-authored-by: Raul Jordan <raul@prysmaticlabs.com> * archiver tests pass * fixing cache tests * cache tests passing * edited validator tests * powchain tests passing * halfway thru sync tests * more sync test fixes * add in tests for state/ * working through rpc tests * assignments tests passed * almost done with rpc/beacon tests * resolved painful validator test * fixed up even more tests * resolve tests * fix build * reduce a randao mixes copy * fixes under //beacon-chain/blockchain/... * build //beacon-chain/core/... * fixes * Runtime Optimizations (prysmaticlabs#4648) * parallelize shuffling * clean up * lint * fix build * use callback to read from registry * fix array roots and size map * new improvements * reduce hash allocs * improved shuffling * terence's review * use different method * raul's comment * new array roots * remove clone in pre-compute * Update beacon-chain/state/types.go Co-Authored-By: Raul Jordan <raul@prysmaticlabs.com> * raul's review * lint * fix build issues * fix visibility Co-authored-by: Raul Jordan <raul@prysmaticlabs.com> * Merge branch 'use-state-in-runtime' of https://github.com/prysmaticlabs/geth-sharding into resolve-tests * fix visibility * build works for all * fix blockchain test * fix a few tests * fix more tests * resolve conf * sync * update validator in slashing * Merge branch 'resolve-tests' of github.com:prysmaticlabs/prysm into resolve-tests * archiver passing * Merge branch 'resolve-tests' of github.com:prysmaticlabs/prysm into resolve-tests * fixed rpc/validator * progress on core tests * Merge branch 'resolve-tests' of github.com:prysmaticlabs/prysm into resolve-tests * resolve broken rpc tests * Merge branch 'resolve-tests' of github.com:prysmaticlabs/prysm into resolve-tests * blockchain tests passed * fix up some tests in core * Merge branch 'master' of github.com:prysmaticlabs/prysm into resolve-tests * fix message diff * remove unnecessary save * Merge branch 'master' of github.com:prysmaticlabs/prysm into resolve-tests * Save validator after slashing * Update validators one by one * another update * fix everything * Merge branch 'resolve-tests' of https://github.com/prysmaticlabs/geth-sharding into resolve-tests * fix more precompute tests * fix blocks tests * Merge branch 'resolve-tests' of github.com:prysmaticlabs/prysm into resolve-tests * more elegant fix * more helper fixes * Merge branch 'resolve-tests' of https://github.com/prysmaticlabs/geth-sharding into resolve-tests * change back ? * fix test * fix skip slot * fix test * reset caches * fix testutil * raceoff fixed * passing * Retrieve cached state in the beginning * lint * Merge branch 'master' of github.com:prysmaticlabs/prysm into resolve-tests * Fixed tests part 1 * Fixed rest of the tests * Merge branch 'master' into optimize-process-att * Minor changes to avoid copying, small refactor to reduce deplicated code * Merge branch 'master' into resolve-tests * Handle att req for slot 0 * New beacon state: Only populate merkle layers as needed, copy merkle layers on copy/clone. (prysmaticlabs#4689) * Only populate merkle layers as needed, copy merkle layers on copy/clone. * use custom copy * Make maps of correct size * slightly fast, doesn't wait for lock Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com> * Merge branch 'resolve-tests' of github.com:prysmaticlabs/prysm into resolve-tests * Merge branch 'master' into resolve-tests * Target root can't be 0x00 * Merge refs/heads/master into resolve-tests * Merge branch 'resolve-tests' of github.com:prysmaticlabs/prysm into resolve-tests * Don't use cache for current slot (may not be the right fix) * Merge branch 'resolve-tests' of git+ssh://github.com/prysmaticlabs/prysm into resolve-tests * Merge branch 'resolve-tests' of github.com:prysmaticlabs/prysm into resolve-tests * fixed up tests * Remove some copy for init sync. Not sure if it is safe enough for runtime though... testing... * Align with prev logic for process slots cachedState.Slot() < slot * Fix Initial Sync Flag (prysmaticlabs#4692) * fixes * fix up some test failures due to lack of nil checks * fix up some test failures due to lack of nil checks * fix up imports * revert some changes * imports Co-authored-by: Raul Jordan <raul@prysmaticlabs.com> * Conflict * Conflict * resolve confs * resolving further conflicts * Better skip slot cache (prysmaticlabs#4694) * Return copy of skip slot cache state, disable skip slot cache on sync * fix * Fix pruning * Merge refs/heads/master into resolve-tests * Merge refs/heads/master into resolve-tests * copy on write method * gaz * fix confs * Merge refs/heads/resolve-tests into copy-on-write * fix tests * Merge branch 'copy-on-write' of github.com:prysmaticlabs/prysm into copy-on-write * fix up issues with broken tests * Merge refs/heads/resolve-tests into copy-on-write * Merge branch 'master' of github.com:prysmaticlabs/prysm into copy-on-write * remove extra update * remove debugging lines * gofmt * Merge refs/heads/master into copy-on-write * Merge refs/heads/master into copy-on-write * Merge refs/heads/master into copy-on-write * Merge branch 'master' of github.com:prysmaticlabs/prysm * Passing 1 to SetEndSlot * Better way to get genesis block * Merge branch 'master' of github.com:prysmaticlabs/prysm into bug/ApiGenesisBlock * Checking for nil genBlk * Testing conditional order * Reverting conditions, and no idea why build fails * Merge branch 'master' into bug/ApiGenesisBlock * Saving to genesis block root in tests * Merge branch 'bug/ApiGenesisBlock' of github.com:AgentJ-WR/prysm into bug/ApiGenesisBlock * Saving root * Merge branch 'master' of github.com:prysmaticlabs/prysm into bug/ApiGenesisBlock * Adding regression test * Updating regression test * Merge branch 'master' into bug/ApiGenesisBlock * Merge branch 'master' into bug/ApiGenesisBlock * Merge branch 'master' into bug/ApiGenesisBlock * Merge branch 'master' into bug/ApiGenesisBlock

* begin state service * begin on the state trie idea * created beacon state structure * add in the full clone getter * return by value instead * add all setters * new state setters are being completed * arrays roots exposed * close to finishing all these headerssss * functionality complete * added in proto benchmark test * test for compatibility * add test for compat * comments fixed * add clone * add clone * remove underlying copies * make it immutable * integrate it into chainservice * revert * wrap up comments for package * address all comments and godocs * address all comments * clone the pending attestation properly * properly clone remaining items * tests pass fixed bug * begin using it instead of head state * prevent nil pointer exceptions * begin using new struct in db * integrated new type into db package * add proper nil checks * using new state in archiver * refactored much of core * editing all the precompute functions * done with most core refactor * fixed up some bugs in the clone comparisons * append current epoch atts * add missing setters * add new setters * fix other core methods * fix up transition * main service and forkchoice * fix rpc * integrated to powchain * some more changes * fix build * improve processing of deposits * fix error * prevent panic * comment * fix process att * gaz * fix up att process * resolve existing review comments * resolve another batch of gh comments * resolve broken cpt state * revise testutil to use the new state * begin updating the state transition func to pass in more compartmentalized args * finish editing transition function to return errors * block operations pretty much done with refactor * state transition fully refactored * got epoch processing completed * fix build in fork choice * fixing more of the build * fix up broken sync package * it builds nowww it buildssss * revert registry changes * Recompute on Read (prysmaticlabs#4627) * compute on read * fix up eth1 data votes * looking into slashings bug introduced in core/ * able to advance more slots * add logging * can now sync with testnet yay * remove the leaves algorithm and other merkle imports * expose initialize unsafe funcs * Update beacon-chain/db/kv/state.go * lint Co-authored-by: Raul Jordan <raul@prysmaticlabs.com> * More Optimizations for New State (prysmaticlabs#4641) * map optimization * more optimizations * use a custom hasher * comment * block operations optimizations * Update beacon-chain/state/types.go Co-Authored-By: Raul Jordan <raul@prysmaticlabs.com> * fixed up various operations to use the validator index map access Co-authored-by: Raul Jordan <raul@prysmaticlabs.com> * archiver tests pass * fixing cache tests * cache tests passing * edited validator tests * powchain tests passing * halfway thru sync tests * more sync test fixes * add in tests for state/ * working through rpc tests * assignments tests passed * almost done with rpc/beacon tests * resolved painful validator test * fixed up even more tests * resolve tests * fix build * reduce a randao mixes copy * fixes under //beacon-chain/blockchain/... * build //beacon-chain/core/... * fixes * Runtime Optimizations (prysmaticlabs#4648) * parallelize shuffling * clean up * lint * fix build * use callback to read from registry * fix array roots and size map * new improvements * reduce hash allocs * improved shuffling * terence's review * use different method * raul's comment * new array roots * remove clone in pre-compute * Update beacon-chain/state/types.go Co-Authored-By: Raul Jordan <raul@prysmaticlabs.com> * raul's review * lint * fix build issues * fix visibility Co-authored-by: Raul Jordan <raul@prysmaticlabs.com> * fix visibility * build works for all * fix blockchain test * fix a few tests * fix more tests * update validator in slashing * archiver passing * fixed rpc/validator * progress on core tests * resolve broken rpc tests * blockchain tests passed * fix up some tests in core * fix message diff * remove unnecessary save * Save validator after slashing * Update validators one by one * another update * fix everything * fix more precompute tests * fix blocks tests * more elegant fix * more helper fixes * change back ? * fix test * fix skip slot * fix test * reset caches * fix testutil * raceoff fixed * passing * Retrieve cached state in the beginning * lint * Fixed tests part 1 * Fixed rest of the tests * Minor changes to avoid copying, small refactor to reduce deplicated code * Handle att req for slot 0 * New beacon state: Only populate merkle layers as needed, copy merkle layers on copy/clone. (prysmaticlabs#4689) * Only populate merkle layers as needed, copy merkle layers on copy/clone. * use custom copy * Make maps of correct size * slightly fast, doesn't wait for lock Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com> * Target root can't be 0x00 * Don't use cache for current slot (may not be the right fix) * fixed up tests * Remove some copy for init sync. Not sure if it is safe enough for runtime though... testing... * Align with prev logic for process slots cachedState.Slot() < slot * Fix Initial Sync Flag (prysmaticlabs#4692) * fixes * fix up some test failures due to lack of nil checks * fix up some test failures due to lack of nil checks * fix up imports * revert some changes * imports Co-authored-by: Raul Jordan <raul@prysmaticlabs.com> * resolving further conflicts * Better skip slot cache (prysmaticlabs#4694) * Return copy of skip slot cache state, disable skip slot cache on sync * fix * Fix pruning * fix up issues with broken tests Co-authored-by: Nishant Das <nish1993@hotmail.com> Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> Co-authored-by: shayzluf <thezluf@gmail.com> Co-authored-by: terence tsao <terence@prysmaticlabs.com> Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com>

) * begin state service * begin on the state trie idea * created beacon state structure * add in the full clone getter * return by value instead * add all setters * new state setters are being completed * arrays roots exposed * close to finishing all these headerssss * functionality complete * added in proto benchmark test * test for compatibility * add test for compat * comments fixed * add clone * add clone * remove underlying copies * make it immutable * integrate it into chainservice * revert * wrap up comments for package * address all comments and godocs * address all comments * clone the pending attestation properly * properly clone remaining items * tests pass fixed bug * begin using it instead of head state * prevent nil pointer exceptions * begin using new struct in db * integrated new type into db package * add proper nil checks * using new state in archiver * refactored much of core * editing all the precompute functions * done with most core refactor * fixed up some bugs in the clone comparisons * append current epoch atts * add missing setters * add new setters * fix other core methods * fix up transition * main service and forkchoice * fix rpc * integrated to powchain * some more changes * fix build * improve processing of deposits * fix error * prevent panic * comment * fix process att * gaz * fix up att process * resolve existing review comments * resolve another batch of gh comments * resolve broken cpt state * revise testutil to use the new state * begin updating the state transition func to pass in more compartmentalized args * finish editing transition function to return errors * block operations pretty much done with refactor * state transition fully refactored * got epoch processing completed * fix build in fork choice * fixing more of the build * fix up broken sync package * it builds nowww it buildssss * revert registry changes * Recompute on Read (prysmaticlabs#4627) * compute on read * fix up eth1 data votes * looking into slashings bug introduced in core/ * able to advance more slots * add logging * can now sync with testnet yay * remove the leaves algorithm and other merkle imports * expose initialize unsafe funcs * Update beacon-chain/db/kv/state.go * lint Co-authored-by: Raul Jordan <raul@prysmaticlabs.com> * More Optimizations for New State (prysmaticlabs#4641) * map optimization * more optimizations * use a custom hasher * comment * block operations optimizations * Update beacon-chain/state/types.go Co-Authored-By: Raul Jordan <raul@prysmaticlabs.com> * fixed up various operations to use the validator index map access Co-authored-by: Raul Jordan <raul@prysmaticlabs.com> * archiver tests pass * fixing cache tests * cache tests passing * edited validator tests * powchain tests passing * halfway thru sync tests * more sync test fixes * add in tests for state/ * working through rpc tests * assignments tests passed * almost done with rpc/beacon tests * resolved painful validator test * fixed up even more tests * resolve tests * fix build * reduce a randao mixes copy * fixes under //beacon-chain/blockchain/... * build //beacon-chain/core/... * fixes * Runtime Optimizations (prysmaticlabs#4648) * parallelize shuffling * clean up * lint * fix build * use callback to read from registry * fix array roots and size map * new improvements * reduce hash allocs * improved shuffling * terence's review * use different method * raul's comment * new array roots * remove clone in pre-compute * Update beacon-chain/state/types.go Co-Authored-By: Raul Jordan <raul@prysmaticlabs.com> * raul's review * lint * fix build issues * fix visibility Co-authored-by: Raul Jordan <raul@prysmaticlabs.com> * fix visibility * build works for all * fix blockchain test * fix a few tests * fix more tests * update validator in slashing * archiver passing * fixed rpc/validator * progress on core tests * resolve broken rpc tests * blockchain tests passed * fix up some tests in core * fix message diff * remove unnecessary save * Save validator after slashing * Update validators one by one * another update * fix everything * fix more precompute tests * fix blocks tests * more elegant fix * more helper fixes * change back ? * fix test * fix skip slot * fix test * reset caches * fix testutil * raceoff fixed * passing * Retrieve cached state in the beginning * lint * Fixed tests part 1 * Fixed rest of the tests * Minor changes to avoid copying, small refactor to reduce deplicated code * Handle att req for slot 0 * New beacon state: Only populate merkle layers as needed, copy merkle layers on copy/clone. (prysmaticlabs#4689) * Only populate merkle layers as needed, copy merkle layers on copy/clone. * use custom copy * Make maps of correct size * slightly fast, doesn't wait for lock Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com> * Target root can't be 0x00 * Don't use cache for current slot (may not be the right fix) * fixed up tests * Remove some copy for init sync. Not sure if it is safe enough for runtime though... testing... * Align with prev logic for process slots cachedState.Slot() < slot * Fix Initial Sync Flag (prysmaticlabs#4692) * fixes * fix up some test failures due to lack of nil checks * fix up some test failures due to lack of nil checks * fix up imports * revert some changes * imports Co-authored-by: Raul Jordan <raul@prysmaticlabs.com> * resolving further conflicts * Better skip slot cache (prysmaticlabs#4694) * Return copy of skip slot cache state, disable skip slot cache on sync * fix * Fix pruning * copy on write method * gaz * fix tests * fix up issues with broken tests * remove extra update * remove debugging lines * gofmt Co-authored-by: Raul Jordan <raul@prysmaticlabs.com> Co-authored-by: Nishant Das <nish1993@hotmail.com> Co-authored-by: shayzluf <thezluf@gmail.com> Co-authored-by: terence tsao <terence@prysmaticlabs.com> Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com>

) * begin state service * begin on the state trie idea * created beacon state structure * add in the full clone getter * return by value instead * add all setters * new state setters are being completed * arrays roots exposed * close to finishing all these headerssss * functionality complete * added in proto benchmark test * test for compatibility * add test for compat * comments fixed * Merge branch 'master' into state-service * add clone * add clone * remove underlying copies * make it immutable * integrate it into chainservice * revert * wrap up comments for package * address all comments and godocs * address all comments * Merge branch 'master' into state-service * clone the pending attestation properly * Merge branch 'state-service' of github.com:prysmaticlabs/prysm into state-service * properly clone remaining items * tests pass fixed bug * begin using it instead of head state * prevent nil pointer exceptions * Merge branch 'state-service' into use-state-in-runtime * begin using new struct in db * integrated new type into db package * add proper nil checks * using new state in archiver * refactored much of core * editing all the precompute functions * done with most core refactor * fixed up some bugs in the clone comparisons * Merge branch 'state-service' into use-state-in-runtime * append current epoch atts * merged master * add missing setters * add new setters * fix other core methods * fix up transition * main service and forkchoice * fix rpc * integrated to powchain * some more changes * fix build * improve processing of deposits * fix error * prevent panic * comment * fix process att * gaz * fix up att process * resolve existing review comments * Merge branch 'master' into use-state-in-runtime * resolve another batch of gh comments * resolve broken cpt state * revise testutil to use the new state * begin updating the state transition func to pass in more compartmentalized args * finish editing transition function to return errors * block operations pretty much done with refactor * state transition fully refactored * got epoch processing completed * fix build in fork choice * fixing more of the build * fix up broken sync package * it builds nowww it buildssss * revert registry changes * Recompute on Read (prysmaticlabs#4627) * compute on read * fix up eth1 data votes * looking into slashings bug introduced in core/ * able to advance more slots * add logging * can now sync with testnet yay * remove the leaves algorithm and other merkle imports * expose initialize unsafe funcs * Update beacon-chain/db/kv/state.go * lint Co-authored-by: Raul Jordan <raul@prysmaticlabs.com> * include master * More Optimizations for New State (prysmaticlabs#4641) * map optimization * more optimizations * use a custom hasher * comment * block operations optimizations * Update beacon-chain/state/types.go Co-Authored-By: Raul Jordan <raul@prysmaticlabs.com> * fixed up various operations to use the validator index map access Co-authored-by: Raul Jordan <raul@prysmaticlabs.com> * archiver tests pass * fixing cache tests * cache tests passing * edited validator tests * powchain tests passing * halfway thru sync tests * more sync test fixes * add in tests for state/ * working through rpc tests * assignments tests passed * almost done with rpc/beacon tests * resolved painful validator test * fixed up even more tests * resolve tests * fix build * reduce a randao mixes copy * fixes under //beacon-chain/blockchain/... * build //beacon-chain/core/... * fixes * Runtime Optimizations (prysmaticlabs#4648) * parallelize shuffling * clean up * lint * fix build * use callback to read from registry * fix array roots and size map * new improvements * reduce hash allocs * improved shuffling * terence's review * use different method * raul's comment * new array roots * remove clone in pre-compute * Update beacon-chain/state/types.go Co-Authored-By: Raul Jordan <raul@prysmaticlabs.com> * raul's review * lint * fix build issues * fix visibility Co-authored-by: Raul Jordan <raul@prysmaticlabs.com> * Merge branch 'use-state-in-runtime' of https://github.com/prysmaticlabs/geth-sharding into resolve-tests * fix visibility * build works for all * fix blockchain test * fix a few tests * fix more tests * resolve conf * sync * update validator in slashing * Merge branch 'resolve-tests' of github.com:prysmaticlabs/prysm into resolve-tests * archiver passing * Merge branch 'resolve-tests' of github.com:prysmaticlabs/prysm into resolve-tests * fixed rpc/validator * progress on core tests * Merge branch 'resolve-tests' of github.com:prysmaticlabs/prysm into resolve-tests * resolve broken rpc tests * Merge branch 'resolve-tests' of github.com:prysmaticlabs/prysm into resolve-tests * blockchain tests passed * fix up some tests in core * Merge branch 'master' of github.com:prysmaticlabs/prysm into resolve-tests * fix message diff * remove unnecessary save * Merge branch 'master' of github.com:prysmaticlabs/prysm into resolve-tests * Save validator after slashing * Update validators one by one * another update * fix everything * Merge branch 'resolve-tests' of https://github.com/prysmaticlabs/geth-sharding into resolve-tests * fix more precompute tests * fix blocks tests * Merge branch 'resolve-tests' of github.com:prysmaticlabs/prysm into resolve-tests * more elegant fix * more helper fixes * Merge branch 'resolve-tests' of https://github.com/prysmaticlabs/geth-sharding into resolve-tests * change back ? * fix test * fix skip slot * fix test * reset caches * fix testutil * raceoff fixed * passing * Retrieve cached state in the beginning * lint * Merge branch 'master' of github.com:prysmaticlabs/prysm into resolve-tests * Fixed tests part 1 * Fixed rest of the tests * Merge branch 'master' into optimize-process-att * Minor changes to avoid copying, small refactor to reduce deplicated code * Merge branch 'master' into resolve-tests * Handle att req for slot 0 * New beacon state: Only populate merkle layers as needed, copy merkle layers on copy/clone. (prysmaticlabs#4689) * Only populate merkle layers as needed, copy merkle layers on copy/clone. * use custom copy * Make maps of correct size * slightly fast, doesn't wait for lock Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com> * Merge branch 'resolve-tests' of github.com:prysmaticlabs/prysm into resolve-tests * Merge branch 'master' into resolve-tests * Target root can't be 0x00 * Merge refs/heads/master into resolve-tests * Merge branch 'resolve-tests' of github.com:prysmaticlabs/prysm into resolve-tests * Don't use cache for current slot (may not be the right fix) * Merge branch 'resolve-tests' of git+ssh://github.com/prysmaticlabs/prysm into resolve-tests * Merge branch 'resolve-tests' of github.com:prysmaticlabs/prysm into resolve-tests * fixed up tests * Remove some copy for init sync. Not sure if it is safe enough for runtime though... testing... * Align with prev logic for process slots cachedState.Slot() < slot * Fix Initial Sync Flag (prysmaticlabs#4692) * fixes * fix up some test failures due to lack of nil checks * fix up some test failures due to lack of nil checks * fix up imports * revert some changes * imports Co-authored-by: Raul Jordan <raul@prysmaticlabs.com> * Conflict * Conflict * resolve confs * resolving further conflicts * Better skip slot cache (prysmaticlabs#4694) * Return copy of skip slot cache state, disable skip slot cache on sync * fix * Fix pruning * Merge refs/heads/master into resolve-tests * Merge refs/heads/master into resolve-tests * copy on write method * gaz * fix confs * Merge refs/heads/resolve-tests into copy-on-write * fix tests * Merge branch 'copy-on-write' of github.com:prysmaticlabs/prysm into copy-on-write * fix up issues with broken tests * Merge refs/heads/resolve-tests into copy-on-write * Merge branch 'master' of github.com:prysmaticlabs/prysm into copy-on-write * remove extra update * remove debugging lines * gofmt * Merge refs/heads/master into copy-on-write * Merge refs/heads/master into copy-on-write * Merge refs/heads/master into copy-on-write * Merge branch 'master' of github.com:prysmaticlabs/prysm * Passing 1 to SetEndSlot * Better way to get genesis block * Merge branch 'master' of github.com:prysmaticlabs/prysm into bug/ApiGenesisBlock * Checking for nil genBlk * Testing conditional order * Reverting conditions, and no idea why build fails * Merge branch 'master' into bug/ApiGenesisBlock * Saving to genesis block root in tests * Merge branch 'bug/ApiGenesisBlock' of github.com:AgentJ-WR/prysm into bug/ApiGenesisBlock * Saving root * Merge branch 'master' of github.com:prysmaticlabs/prysm into bug/ApiGenesisBlock * Adding regression test * Updating regression test * Merge branch 'master' into bug/ApiGenesisBlock * Merge branch 'master' into bug/ApiGenesisBlock * Merge branch 'master' into bug/ApiGenesisBlock * Merge branch 'master' into bug/ApiGenesisBlock

nisdas added 2 commits January 25, 2020 10:42

parallelize shuffling

237352a

clean up

92514c2

nisdas added the Ready For Review A pull request ready for code review label Jan 25, 2020

nisdas added 5 commits January 25, 2020 10:48

lint

677b96e

fix build

e866239

use callback to read from registry

10fffe6

fix array roots and size map

3811d81

new improvements

c924667

nisdas changed the title ~~Parallelize Shuffling~~ Runtime Optimizations Jan 25, 2020

nisdas requested a review from rauljordan January 25, 2020 10:33

reduce hash allocs

8417b88

terencechain requested changes Jan 25, 2020

View reviewed changes

nisdas added 6 commits January 25, 2020 23:11

improved shuffling

3c3a13b

terence's review

7eae84b

use different method

9c4bddf

raul's comment

fdbb58e

new array roots

823b01c

remove clone in pre-compute

7c079a0

terencechain approved these changes Jan 28, 2020

View reviewed changes

rauljordan reviewed Jan 28, 2020

View reviewed changes

nisdas and others added 3 commits January 28, 2020 12:55

Update beacon-chain/state/types.go

ed86b50

Co-Authored-By: Raul Jordan <raul@prysmaticlabs.com>

raul's review

dcf0da8

Merge branch 'parallelizeShuffling' of https://github.com/prysmaticla…

3500a7d

…bs/geth-sharding into parallelizeShuffling

rauljordan approved these changes Jan 28, 2020

View reviewed changes

nisdas added 2 commits January 28, 2020 13:15

lint

d3c1ab9

fix build issues

a46a700

rauljordan approved these changes Jan 28, 2020

View reviewed changes

fix visibility

a824b89

nisdas merged commit c65bfcc into use-state-in-runtime Jan 28, 2020

delete-merged-branch bot deleted the parallelizeShuffling branch January 28, 2020 05:51

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Runtime Optimizations #4648

Runtime Optimizations #4648

nisdas commented Jan 25, 2020 •

edited

protolambda commented Jan 25, 2020 •

edited

nisdas commented Jan 25, 2020

terencechain left a comment

terencechain Jan 25, 2020

terencechain Jan 25, 2020

nisdas Jan 25, 2020

terencechain Jan 25, 2020

nisdas Jan 25, 2020

rauljordan Jan 25, 2020

nisdas Jan 26, 2020

terencechain Jan 25, 2020

rauljordan Jan 25, 2020

terencechain left a comment

rauljordan Jan 28, 2020

nisdas Jan 28, 2020

rauljordan Jan 28, 2020

rauljordan Jan 28, 2020

Runtime Optimizations #4648

Runtime Optimizations #4648

Conversation

nisdas commented Jan 25, 2020 • edited

protolambda commented Jan 25, 2020 • edited

nisdas commented Jan 25, 2020

terencechain left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

terencechain left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nisdas commented Jan 25, 2020 •

edited

protolambda commented Jan 25, 2020 •

edited