rpc: move UniValue in blockToJSON #30094
Conversation
Without explicitly declaring the move, these UniValues get copied, causing increased memory usage. Fix this by explicitly moving the UniValue objects. Used by `rest_block` and `getblock` RPC.
|
The following sections might be updated with supplementary metadata relevant to reviewers and maintainers. Code CoverageFor detailed information about the code coverage, see the test coverage report. ReviewsSee the guideline for information on the review process.
If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update. |
|
Before and after of RSS:
|
|
review ACK b77bad3 |
|
Also "fixes" #30052 |
|
Unrelated to the blockToJSON changes here, as a follow-up, it may be possible to "fix" #25229 in the same way? |
|
Actually, looks like this could use a |
Without explicitly declaring the move, these UniValues get copied, causing increased memory usage. Fix this by explicitly moving the UniValue objects. Used by `rest_block` and `getblock` RPC. Github-Pull: bitcoin#30094 Rebased-From: b77bad3
I can push a similar commit here if it's worth merging the two. Or open a separate PR? Perhaps with 3 ACKs a separate PR is preferable? I also am only halfway through making a "big wallet" (using achow101's make-big-wallet script, so I have no suitable wallet to test such a commit against currently.
I don't see
I can check the remainder of the codebase for other occurrences too. |
Whoops, of course you're right. It probably should though :) |
I was already working on this and... wow. You've opened a big can of worms here @willcl-ark :). But that's a good thing because it should reduce memusage and increase performance I'd assume. See here for my first pass. It's a beast. https://github.com/theuni/bitcoin/commits/less-univalue-copies/ The first commit mostly just does the naive thing of adding The second commit adds These are changes required for bitcoind only. I haven't done tests/fuzz/bench/etc. To flesh them out, I changed the function signatures of the corresponding UniValue functions to accept rvalues only. Edit: added 2 more commits to avoid copying strings. @willcl-ark let me know if you've already been working on this some and how you'd like to collaborate on it. There's quite a bit of work to do here :) |
Without explicitly declaring the move, these UniValues get copied, causing increased memory usage. Fix this by explicitly moving the UniValue objects. Used by `rest_block` and `getblock` RPC. Github-Pull: bitcoin#30094 Rebased-From: b77bad3
|
Backported to 27.x in #30092. |
Without explicitly declaring the move, these UniValues get copied, causing increased memory usage. Fix this by explicitly moving the UniValue objects. Used by `rest_block` and `getblock` RPC. Github-Pull: bitcoin#30094 Rebased-From: b77bad3
Wow! This looks like a great start! Will begin to review...
I have not yet started to look at this so will avoid duplicating the effort! All I had started was a branch to add explicit move semantics (which it turns out is not needed): master...willcl-ark:bitcoin:univalue-move Happy to collaborate though if you have any specific things in mind you'd like help with :) I do have a somewhat-reproducible benchmark set up for this specifc REST call, it's just querying every 5000th block. I was also using @achow101 's make-big-wallet to make a big wallet last night (wow-- it took 12 hours to generate!) in order to try and benchmark the I don't think it's strictly needed, as moving a large UniValue is almost certainly an improvement over a large copy, but I still like to see for myself. |
|
@theuni FWIW I do see some increased RSS measurements on your current branch vs master (with my one, single test using listtransactions rpc on a large wallet):
This is also what I saw on my linked "univalue-move" branch above. I didn't investigate why exactly this happens, yet. It seems like |
|
backport for 26.x in #29899 |
|
@willcl-ark Thanks! Huh, I can't think of any reason why that would be the case. Saving on copies should be nothing but a win. Odd. I think I'll start by opening a PR for the simplest/most obvious moves. Maybe you could help me bench those? |
|
@willcl-ark I added a quick/dirty bench to my branch that shows the difference between copies and moves. It seems correct to me. Do you have any interest in doing the same for memory usage, just as a sanity check that nothing funky is happening low-level? Beyond that, I'm not really sure where to start with your graph... it really doesn't make any sense to me :) |
|
You benchmark looks good and reports the result I'd expect too! I'll investigate my own measurements some more in the mean time... |
d7707d9 rpc: avoid copying into UniValue (Cory Fields) Pull request description: These are the simple (and hopefully obviously correct) copies that can be moves instead. This is a follow-up from #30094 (comment) As it turns out, there are hundreds of places where we copy UniValues needlessly. It should be the case that moves are always preferred over copies, so there should be no downside to these changes. willcl-ark, however, noticed that memory usage may increase in some cases. Logically this makes no sense to me. The only plausible explanation imo is that because the moves are faster, more ops/second occur in some cases. This list of moves was obtained by changing the function signatures of the UniValue functions to accept only rvalues, then compiling and fixing them up one by one. There still exist many places where copies are being made. These can/should be fixed up, but weren't done here for the sake of doing the easy ones first. I ran these changes through clang-tidy with `performance-move-const-arg` and `bugprone-use-after-move` and no bugs were detected (though that's obviously not to say it can be trusted 100%). As stated above, there are still lots of other less trivial fixups to do after these including: - Using non-const UniValues where possible so that moves can happen - Refactoring code in order to be able to move a UniValue without introducing a use-after-move - Refactoring functions to accept UniValues by value rather than by const reference ACKs for top commit: achow101: ACK d7707d9 ryanofsky: Code review ACK d7707d9. No changes since last review other than rebase. I agree benchmarks showing increased peak memory usage and RSS are surprising, but number of allocations is down as expected, and runtime is also decreased. willcl-ark: ACK d7707d9 Tree-SHA512: 7f511be73984553c278186286a7d161a34b2574c7f5f1a0edc87c2913b4c025a0af5241ef9af2df17547f2e4ef79710aa5bbb762fc9472435781c0488dba3435
Without explicitly declaring the move, these UniValues get copied, causing increased memory usage. Fix this by explicitly moving the UniValue objects. Used by `rest_block` and `getblock` RPC. Github-Pull: bitcoin#30094 Rebased-From: b77bad3
aa7e876 [doc] add draft release notes for 26.2rc1 (glozow) 21d9aaa p2p, bugfix: detect addnode cjdns peers in GetAddedNodeInfo() (Jon Atack) ec5ce2f windeploy: Renew certificate (Ava Chow) 96d0e81 rpc: Reword SighashFromStr error message (MarcoFalke) 6685aff rpc: move UniValue in blockToJSON (willcl-ark) 7f45e00 depends: Fix build of Qt for 32-bit platforms (laanwj) f9b76ba ci: Pull in qtbase5-dev instead of seperate low-level libraries (laanwj) c587753 doc: Suggest installing dev packages for debian/ubuntu qt5 build (laanwj) 7ecdb08 ci: Bump s390x to ubuntu:24.04 (MarcoFalke) d9ef6cf sign: don't assume we are parsing a sane Miniscript (Antoine Poinsot) e4859c8 depends: fix mingw-w64 Qt DEBUG=1 build (fanquake) bb46b90 Fix #29767, set m_synced = true after Commit() (nanlour) bf5b6fc Throw error if invalid parameters passed to getnetworkhashps RPC endpoint (Jameson Lopp) a81a922 [rpc, bugfix] Enforce maximum value for setmocktime (dergoegge) d39ea51 Change Luke Dashjr seed to dashjr-list-of-p2p-nodes.us (Luke Dashjr) c21bbcc [doc] archive 26.1 release notes (glozow) Pull request description: Archives 26.1 release notes and adds draft release notes for 26.2rc1 Also backports: - #29691 - #29869 - #28554 - #29747 - #29853 - #29856 - #29764 - #29776 - #29985 - #30094 - #29870 - #30149 - #30085 ACKs for top commit: stickies-v: re-ACK aa7e876, only changes are fixing commit msg and transifex reference willcl-ark: ACK aa7e876 Tree-SHA512: b81ba6092640de696d782114cdf43e7ed1d63ea0a3231cade30653c2743d87700e0f852a1b1fcc42ae313b2d4f004e6026ddbad87d58c2fde0a660e90026ed98
22701a4 doc: update manual pages for 27.1rc1 (fanquake) 9e91907 build: bump version to 27.1rc1 (fanquake) 9b4640c doc: update release-notes.md for 27.1 (fanquake) 80032d6 qt: 27.1rc1 translations update (Hennadii Stepanov) 423bd6d windeploy: Renew certificate (Ava Chow) 77b2321 depends: Fetch miniupnpc sources from an alternative website (Hennadii Stepanov) 31adcfa test: add GetAddedNodeInfo() CJDNS regression unit test (Jon Atack) 9cdb9ed p2p, bugfix: detect addnode cjdns peers in GetAddedNodeInfo() (Jon Atack) 3c26058 crypto: disable asan for sha256_sse4 with clang and -O0 (Cory Fields) 0ba11cf rpc: move UniValue in blockToJSON (willcl-ark) dedf319 gui: don't permit port in proxy IP option (willcl-ark) d1289a1 gui: fix create unsigned transaction fee bump (furszy) Pull request description: Backports: * bitcoin-core/gui#812 * bitcoin-core/gui#813 * #30085 * #30094 * #30097 * #30149 * #30151 Bump to 27.1rc1. ACKs for top commit: stickies-v: re-ACK 22701a4 willcl-ark: reACK 22701a4 hebasto: re-ACK 22701a4. Tree-SHA512: 6eca44ba7e6664eb4677646597dfdaf56a241c8c3e95e0ab8929ee2fc3671303fc6c2634d359b4523dbd452ac5e54fd1f4c7c2bf7e9c5209395f8cb3b4753fb3
Fixes: #24542
Fixes: #30052
Without explicitly declaring the move, these
UniValuesget copied, causing increased memory usage. Fix this by explicitly moving theUniValueobjects.Used by
rest_blockandgetblockRPC.