-
Notifications
You must be signed in to change notification settings - Fork 36.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Compensate for memory peak at flush time #10126
Conversation
Please tag as 0.14.1. |
@sipa are any of your experiments relatively simple to reproduce? I'd like to step through the high usage spots to get a better feel for what's happening. |
@theuni Using the top commit at https://github.com/sipa/bitcoin/commits/repmemusage I've done tests with reindexes and syncs from scratch, killing and restarting bitcoind at certain memory usage points. Then looked at (MAX_RSS - 280MB) / (last_reported_cache=_line). |
utACK 7228ce8 |
7228ce8 Compensate for memory peak at flush time (Pieter Wuille) Tree-SHA512: 97e9848410fab061402c85d8440c54a50dd8a0203b2ea194013ea116700a6dc1b4b26b8c5f9c9c68c1f5c6b935c5d6c737437c1911b003d9ff5445c570cd449d
@sipa I still object to this PR without at least some increase in the default dbcache. I believe that in the event the mempool is full and there is no loan available then performance will be unnecessarily poor. I think 300 MB (with this new calculation) is approximately the bare minimum we can achieve without likely exceeding it by accident when connecting a block. It is not appropriate for a default. |
7228ce8 Compensate for memory peak at flush time (Pieter Wuille) Tree-SHA512: 97e9848410fab061402c85d8440c54a50dd8a0203b2ea194013ea116700a6dc1b4b26b8c5f9c9c68c1f5c6b935c5d6c737437c1911b003d9ff5445c570cd449d
7228ce8 Compensate for memory peak at flush time (Pieter Wuille) Tree-SHA512: 97e9848410fab061402c85d8440c54a50dd8a0203b2ea194013ea116700a6dc1b4b26b8c5f9c9c68c1f5c6b935c5d6c737437c1911b003d9ff5445c570cd449d
It seems that our conversion from CCoinsCacheDB into a CDBBatch, and the following processing of the batch by LevelDB causes a spike in memory usage. In various experiments (all on x86_64, though), I've determined that this spike is between x1.5 and x1.9.
Take this into account ahead of time before deciding whether to flush or not.