Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use a memory pool to reduce block deserialization heap usage #2025

Merged

Conversation

@wezrule
Copy link
Collaborator

commented May 24, 2019

When profiling bootstrapping for only 30 seconds, there is 50MB of heap memory allocated in nano::block::deserialize_block this has now been eliminated. Instead of many smaller allocations used frequently, a memory pool per block type is reused. Requesting heap memory is also slow as it requires an OS call which can take an indeterminate amount of time, so there should be a performance gain as well, although I have not quantitatively measured this. There will be longer term benefits as well because there will be less fragmentation of memory.

Note from the documentation:

The boost::singleton_pool is thread safe if there is only 1 thread running before main and 1 thread running after main. The underlying pool constructed by the singleton is never freed. This means that memory allocated by a singleton_pool can be still used after main() has completed, but may mean that some memory checking programs will complain about leaks from singleton_pool.

I did notice that the work_watcher thread is still running in the background, I now stop this in wallets::stop.

Also removed an unused nano::block_uniquer * parameter from one of the deserialize_block overloads.

ASAN run was clean, valgrind did find some new "still-reachable" memory blocks. @cryptocode suggested to call purge_memory which freed this memory. Am doing this in a guarded class in main for all binaries which use blocks.

@wezrule wezrule added this to the V19.0 milestone May 24, 2019

@wezrule wezrule requested review from SergiySW and cryptocode May 24, 2019

@wezrule wezrule self-assigned this May 24, 2019

@cryptocode
Copy link
Collaborator

left a comment

LGTM

@wezrule wezrule added this to RC 4 (TBD) in V19 May 25, 2019

@wezrule wezrule merged commit f9021b3 into nanocurrency:master May 25, 2019

2 checks passed

continuous-integration/appveyor/pr AppVeyor build succeeded
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details

@wezrule wezrule deleted the wezrule:improve_block_deserialization_heap_usage branch May 25, 2019

@wezrule wezrule added the performance label May 27, 2019

argakiig added a commit that referenced this pull request Jun 11, 2019

Use a memory pool to reduce block deserialization heap usage (#2025)
* Use a memory pool when deserializing blocks to reduce heap usage

* Stop work watcher

* Purge memory at end of main

* Formatting

* Forgot to add it to add purging in rpc_test

* Formatting
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.