Compensate for memory peak at flush time

[sipa]

It seems that our conversion from CCoinsCacheDB into a CDBBatch, and the following processing of the batch by LevelDB causes a spike in memory usage. In various experiments (all on x86_64, though), I've determined that this spike is between x1.5 and x1.9.

Take this into account ahead of time before deciding whether to flush or not.

[gmaxwell]

Please tag as 0.14.1.

[theuni]

@sipa are any of your experiments relatively simple to reproduce? I'd like to step through the high usage spots to get a better feel for what's happening.

[sipa]

@theuni Using the top commit at https://github.com/sipa/bitcoin/commits/repmemusage I've done tests with reindexes and syncs from scratch, killing and restarting bitcoind at certain memory usage points. Then looked at (MAX_RSS - 280MB) / (last_reported_cache=_line).

[laanwj]

utACK 7228ce8

[morcos]

@sipa I still object to this PR without at least some increase in the default dbcache. I believe that in the event the mempool is full and there is no loan available then performance will be unnecessarily poor. I think 300 MB (with this new calculation) is approximately the bare minimum we can achieve without likely exceeding it by accident when connecting a block. It is not appropriate for a default.