New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix high memory consumption during snapshot import #4689

Merged
merged 2 commits into from Dec 8, 2017

Conversation

Projects
None yet
4 participants
@gumb0
Member

gumb0 commented Dec 7, 2017

The accounts with huge storage are splitted between several chunks of snapshot. Previous approach was to first collect the full storage of such account in std::map in memory and then construct a trie out of it and save to database. This consumed a lot of RAM.

The new approach is to commit to the database the parts of the storage of each chunk (so there's one commit per each chunk).

Instead of accumulating in memory the full storage of the accounts sp…
…litted between several snapshot chunks commit to database the parts from each chunk separately.

@gumb0 gumb0 added the in progress label Dec 7, 2017

@codecov-io

This comment has been minimized.

Show comment
Hide comment
@codecov-io

codecov-io Dec 7, 2017

Codecov Report

Merging #4689 into develop will increase coverage by 0.69%.
The diff coverage is 64.28%.

Impacted file tree graph

@@             Coverage Diff             @@
##           develop    #4689      +/-   ##
===========================================
+ Coverage    60.93%   61.63%   +0.69%     
===========================================
  Files          343      343              
  Lines        27112    27964     +852     
  Branches      3101     3042      -59     
===========================================
+ Hits         16521    17235     +714     
- Misses        9612     9706      +94     
- Partials       979     1023      +44
Impacted Files Coverage Δ
libethereum/StateImporter.cpp 0% <0%> (ø) ⬆️
libethereum/SnapshotImporter.h 28.57% <0%> (-4.77%) ⬇️
libethereum/StateImporter.h 66.66% <0%> (ø) ⬆️
...est/unittests/libethereum/SnapshotImporterTest.cpp 100% <100%> (ø) ⬆️
libethereum/SnapshotImporter.cpp 75.96% <75%> (-1.73%) ⬇️
libevm/VM.h 58.82% <0%> (-17.85%) ⬇️
libethcore/Common.h 76.47% <0%> (-5.89%) ⬇️
utils/json_spirit/json_spirit_value.h 57.52% <0%> (-4.88%) ⬇️
libp2p/UDP.h 71.91% <0%> (-3.38%) ⬇️
libdevcore/CommonData.h 81.55% <0%> (-1.78%) ⬇️
... and 33 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 40c7ceb...437da1c. Read the comment docs.

codecov-io commented Dec 7, 2017

Codecov Report

Merging #4689 into develop will increase coverage by 0.69%.
The diff coverage is 64.28%.

Impacted file tree graph

@@             Coverage Diff             @@
##           develop    #4689      +/-   ##
===========================================
+ Coverage    60.93%   61.63%   +0.69%     
===========================================
  Files          343      343              
  Lines        27112    27964     +852     
  Branches      3101     3042      -59     
===========================================
+ Hits         16521    17235     +714     
- Misses        9612     9706      +94     
- Partials       979     1023      +44
Impacted Files Coverage Δ
libethereum/StateImporter.cpp 0% <0%> (ø) ⬆️
libethereum/SnapshotImporter.h 28.57% <0%> (-4.77%) ⬇️
libethereum/StateImporter.h 66.66% <0%> (ø) ⬆️
...est/unittests/libethereum/SnapshotImporterTest.cpp 100% <100%> (ø) ⬆️
libethereum/SnapshotImporter.cpp 75.96% <75%> (-1.73%) ⬇️
libevm/VM.h 58.82% <0%> (-17.85%) ⬇️
libethcore/Common.h 76.47% <0%> (-5.89%) ⬇️
utils/json_spirit/json_spirit_value.h 57.52% <0%> (-4.88%) ⬇️
libp2p/UDP.h 71.91% <0%> (-3.38%) ⬇️
libdevcore/CommonData.h 81.55% <0%> (-1.78%) ⬇️
... and 33 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 40c7ceb...437da1c. Read the comment docs.

@gumb0 gumb0 removed the in progress label Dec 8, 2017

@gumb0 gumb0 requested review from pirapira and chfast Dec 8, 2017

@chfast

chfast approved these changes Dec 8, 2017

Show outdated Hide outdated libethereum/StateImporter.cpp
@pirapira

Looks good to me.

@gumb0 gumb0 merged commit d3573f5 into develop Dec 8, 2017

4 of 5 checks passed

continuous-integration/appveyor/pr Waiting for AppVeyor build to complete
Details
ci/circleci: Linux-Clang5 Your tests passed on CircleCI!
Details
ci/circleci: Linux-GCC6-Debug Your tests passed on CircleCI!
Details
ci/circleci: macOS-XCode9 Your tests passed on CircleCI!
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details

@gumb0 gumb0 deleted the snapshot-import-memory-fix branch Dec 8, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment