Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tortoise increasing memory #1675

Merged
merged 27 commits into from Jan 16, 2020
Merged

Tortoise increasing memory #1675

merged 27 commits into from Jan 16, 2020

Conversation

@almogdepaz
Copy link
Member

almogdepaz commented Jan 9, 2020

Motivation

fix tortoise increasing memory issue

@almogdepaz almogdepaz requested a review from barakshani Jan 9, 2020
almogdepaz added 5 commits Jan 9, 2020
@@ -156,6 +158,7 @@ func defaultBaseConfig() BaseConfig {
PoETServer: "127.0.0.1",
Hdist: 5,
GenesisActiveSet: 5,
BlockCacheSize: 20,

This comment has been minimized.

Copy link
@barakshani

barakshani Jan 9, 2020

I think it is too small; used to be 100 (maybe also not justified)

This comment has been minimized.

Copy link
@almogdepaz

almogdepaz Jan 9, 2020

Author Member

i was the one to set the 100, i dont think it needs to be bigger then the window in regards to the tortoise, beacon is not important, how far back does the atx's forBlockInView go ?

This comment has been minimized.

Copy link
@barakshani

barakshani Jan 9, 2020

full epoch, but if we don't need cache for that it doesn't matter.

return err
}

if !valid {
ni.With().Warning("block is contextually invalid", log.BlockId(bid.String()))
ni.With().Warning("block is contextually invalid", log.BlockId(b.String()))

This comment has been minimized.

Copy link
@barakshani

barakshani Jan 9, 2020

what does b.String() outputs? if we only need the block id, would b.Id().String works?

This comment has been minimized.

Copy link
@almogdepaz

almogdepaz Jan 9, 2020

Author Member

basically "its ok" but ill do it more explicit

@@ -150,6 +163,7 @@ func (ni *NinjaTortoise) PersistTortoise() error {
return err
}
return ni.Persist(mesh.TORTOISE, ni.ninjaTortoise)
return nil

This comment has been minimized.

Copy link
@barakshani

This comment has been minimized.

Copy link
@almogdepaz

almogdepaz Jan 9, 2020

Author Member

oops, thats what happens when you trust the merge tool

tortoise/ninja_tortoise.go Show resolved Hide resolved
tortoise/ninja_tortoise.go Show resolved Hide resolved
tortoise/ninja_tortoise.go Outdated Show resolved Hide resolved
func (ni *NinjaTortoise) initTallyToBase(base votingPattern, p votingPattern, windowStart types.LayerID) {
mp := make(map[BlockIDLayerTuple]vec, len(ni.TTally[base]))
for b, v := range ni.TTally[base] {
if b.Layer() < windowStart {

This comment has been minimized.

Copy link
@barakshani

barakshani Jan 11, 2020

Something doesn't add up here. This condition seems to always be satisfied (except maybe for the very first layers), because b.Layer() < p_base <= windowStart.
More generally, understanding the leak cause, it seems that this entire function is unnecessary: we never need to actually copy p_base's opinion, as it is irreversible, and in addition we should have done SaveOpinion to p_base opinion by here.

This comment has been minimized.

Copy link
@almogdepaz

almogdepaz Jan 12, 2020

Author Member

well your kinda right, i guess that now that we treat everything beneath pbase as irreversible and we already evicted all the "irreversible" blocks from the maps, we only need to consider blocks above pbase, need to remember that the case where windowstart > pbase is not handled.
but what about pbase - hdist we have explicit votes there, and i think we have to consider them

This comment has been minimized.

Copy link
@almogdepaz

almogdepaz Jan 12, 2020

Author Member

maybe we should switch to
windowStart = Max(ni.PBase.Layer()-ni.Hdist, newlyr.Index()-Window+1)

almogdepaz added 3 commits Jan 12, 2020
return err
}

if !valid {
ni.With().Warning("block is contextually invalid", log.BlockId(bid.String()))
ni.With().Warning("block is contextually invalid", log.BlockId(b.BlockID.String()))

This comment has been minimized.

Copy link
@y0sher

y0sher Jan 12, 2020

Contributor

here you use b.BlockID and in other places you use b.Id() why not unify ?

This comment has been minimized.

Copy link
@almogdepaz

almogdepaz Jan 12, 2020

Author Member

barak asked for this, im not about to refactor the whole thing, anyway barak is going through this i dont see why you should too

TEffectiveToBlocks map[votingPattern][]types.BlockID //inverse blocks effective pattern
TSupport map[votingPattern]int //for pattern p the number of blocks that support p
TComplete map[votingPattern]struct{} //complete voting Patterns
TEffectiveToBlocks map[votingPattern][]BlockIDLayerTuple //inverse blocks effective pattern

This comment has been minimized.

Copy link
@y0sher

y0sher Jan 12, 2020

Contributor

not suggesting we should do it now be technically it could also be
map[votingPattern]map[types.LayerID][]types.BlockID right ?

This comment has been minimized.

Copy link
@almogdepaz

almogdepaz Jan 12, 2020

Author Member

could but its simpler this way

@@ -55,6 +55,10 @@ func AddCommands(cmd *cobra.Command) {

cmd.PersistentFlags().IntVar(&config.GenesisActiveSet, "genesis-active-size",
config.GenesisActiveSet, "The active set size for the genesis flow")

cmd.PersistentFlags().IntVar(&config.BlockCacheSize, "block-cache-size",

This comment has been minimized.

Copy link
@y0sher

y0sher Jan 12, 2020

Contributor

please add this in config.toml as well just for reference.

almogdepaz added 4 commits Jan 12, 2020
@almogdepaz

This comment has been minimized.

Copy link
Member Author

almogdepaz commented Jan 15, 2020

bors merge

bors bot pushed a commit that referenced this pull request Jan 15, 2020
## Motivation
fix tortoise increasing memory issue
@bors

This comment has been minimized.

Copy link

bors bot commented Jan 15, 2020

@almogdepaz

This comment has been minimized.

Copy link
Member Author

almogdepaz commented Jan 15, 2020

bors merge

bors bot pushed a commit that referenced this pull request Jan 15, 2020
## Motivation
fix tortoise increasing memory issue
@bors

This comment has been minimized.

Copy link

bors bot commented Jan 15, 2020

@almogdepaz

This comment has been minimized.

Copy link
Member Author

almogdepaz commented Jan 15, 2020

bors merge

bors bot pushed a commit that referenced this pull request Jan 15, 2020
## Motivation
fix tortoise increasing memory issue
almogdepaz added 2 commits Jan 16, 2020
# Conflicts:
#	hare/eligibility/oracle_test.go
@almogdepaz

This comment has been minimized.

Copy link
Member Author

almogdepaz commented Jan 16, 2020

bors merge

bors bot pushed a commit that referenced this pull request Jan 16, 2020
## Motivation
fix tortoise increasing memory issue
@bors

This comment has been minimized.

Copy link

bors bot commented Jan 16, 2020

Canceled

@almogdepaz

This comment has been minimized.

Copy link
Member Author

almogdepaz commented Jan 16, 2020

bors merge

1 similar comment
@almogdepaz

This comment has been minimized.

Copy link
Member Author

almogdepaz commented Jan 16, 2020

bors merge

@almogdepaz almogdepaz merged commit 66e9a7e into develop Jan 16, 2020
1 of 2 checks passed
1 of 2 checks passed
bors Waiting in queue
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
@bors bors bot deleted the tortoise_increasing_memory branch Jan 16, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.