Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Make leveldb options configurable #13

Open
JustinAzoff opened this issue Apr 10, 2018 · 3 comments
Open

Make leveldb options configurable #13

JustinAzoff opened this issue Apr 10, 2018 · 3 comments

Comments

@JustinAzoff
Copy link
Owner

flow-indexer/store/leveldb.go

Lines 29 to 36 in 52b1dce

opts.BlockSize = 32768
opts.WriteBuffer = 67108864
opts.BlockCacheCapacity = 524288000
opts.OpenFilesCacheCapacity = 1024
opts.CompactionTableSize = 32 * 1024 * 1024
opts.WriteL0SlowdownTrigger = 16
opts.WriteL0PauseTrigger = 64
opts.Filter = filter.NewBloomFilter(10)

@JustinAzoff
Copy link
Owner Author

ah, one trickyness with this is there is support for more than one store, but I could add a somewhat generic store option type? some sort of cache size would be common to many of them.

@JustinAzoff
Copy link
Owner Author

also https://dave.cheney.net/2014/10/17/functional-options-for-friendly-apis

@JustinAzoff
Copy link
Owner Author

From testing, it seems the problem for memory usage is mostly the WriteBuffer, so I may just release a new version that drops that back to the default.

JustinAzoff added a commit that referenced this issue Apr 11, 2018
@JustinAzoff
Use default WriteBuffer size
82e7915 
Using a write buffer of 64M was causing flow-indexer to use 7G of ram
while indexing.  Changing this back to the default of 4M cuts the peak
usage in half.  I'm still not sure how a 64M buffer ends up turning into
7G of usage, but I think it has something to do with GC and leveldb
allocating a 64M buffer for each transaction.

This is a partial fix for #13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@JustinAzoff