Coinstats Index #19521
The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.
Reviewers, this pull request conflicts with the following ones:
If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.
@PierreRochard Thanks a lot for the comments! I am happy to include these values you requested and drafted that up in a new commit (will need another day for clean-up, docs, tests etc. but it should work already). I actually had done some similar work already for my blog post and was thinking about adding some more numbers myself, just opted to keep the changeset small at the end.
The RPC now has the values for every block by passing a verbose flag
I just noticed when I was almost done, that I wasn't sure if you would like these requested values to be cumulative or per-block. They are per-block now but it is trivial to change that. Actually, I could even do both. And another questions: should the unspendable output values be included or excluded from the other output values. Example:
And I saw your request for the data dump on twitter. Will look into that as a follow-up :)
I find it easier to do the cumulative sum in pandas than to back in to the per-block, but the answer also depends on the performance of the underlying index for aggregate queries. If this behaves like a SQL database and we want it to be normalized, it should be per-block, but I'm not religious about it.
@PierreRochard Thanks for testing and further feedback. As I have written the tests today and did further cleanup I have found that I hadn't been really doing what you were looking for in the coinbase_amount. I was simply returning the block subsidy there.
I am pushing my fixes and test as a separate commit for now since you were already testing with the first version:
I think the formula in
I think the new code also fixes the BIP30 issue you discovered but I am still syncing the new version of the index with not-super-fast hardware so I will be able to check tomorrow.
There is already a check for that and it should error in the case you describe: https://github.com/bitcoin/bitcoin/pull/19521/files#diff-decae4be02fb8a47ab4557fe74a9cb853bdfa3ec0fa1b515c0a1e5de91f4ad0bR1168
If you found a scenario where it doesn't work, please let me know. There is at least a basic test for it here but maybe I have overlooked something: https://github.com/bitcoin/bitcoin/pull/19521/files#diff-a6434325d09a6df4b371513c837907dfc1a97cf540709def4bded9b2a17e3f49R115
779e638 coinstats: Add comments for new coinstatsindex values (Fabian Jahr) 5b3d4e7 Index: Improve logging in coinstatsindex (Fabian Jahr) d4356d4 rpc: Block until synced if coinstatsindex is used in gettxoutsetinfo (Fabian Jahr) a5f6791 rpc: Add missing gettxoutsetinfo help docs (Fabian Jahr) 01386bf Index: Return early from failed coinstatsindex init (Fabian Jahr) 1e38423 index: Use batch writing in coinstatsindex WriteBlock (Fabian Jahr) fb65dde scripted-diff: Fix coinstats data member names (Fabian Jahr) 8ea8c92 index: Avoid unnecessary type casts in coinstatsindex (Fabian Jahr) Pull request description: This is a collection of smaller follow-ups to #19521, addressing several post-merge review comments. ACKs for top commit: Sjors: re-utACK 779e638 jonatack: re-ACK 779e638 diff since last review involves doc changes only; rebased to current master and verified clean debug build/no silent conflicts, unit tests, and feature_coinstatsindex functional test laanwj: Code review ACK 779e638 Talkless: re-utACK 779e638 after cosmetic changes. Tree-SHA512: cb0d038d230c582d7fe3041c89b1e04d39971fab3739d540c609cf826754c6c513b12ded08ac92180aec7a9d7a70114ece50357bd1a902de4adaae9f30b8d699
Summary: Division of MuHash objects are very expensive and multiplication relatively cheap. The whole idea of introducing and tracking numerator and denominators seperately as a representation of the internal state was so that divisions would be rare. So using divison in the Remove method did not make any sense and was just a silly mistake which is corrected here. This is a partial backport of [[bitcoin/bitcoin#19521 | core#19521]] bitcoin/bitcoin@2e2648a Test Plan: `ninja all check-all` Reviewers: #bitcoin_abc, Fabien Reviewed By: #bitcoin_abc, Fabien Subscribers: Fabien Differential Revision: https://reviews.bitcoinabc.org/D11168
This is part of the "UTXO Set Statistics" project in Core, which aims to make things like "checking coin supply" faster. This stuff, and related things in coinstats.cpp, have always been broken for Elements in the sense that we consider only explicit outputs and we ignore assets (so everything just gets added together to get a meaningless total). It probably wouldn't be too hard to restrict this to only consider policyAsset, but it's out of scope for a rebase IMO. Also, I think this situation is fine .. I don't understand the motivation for this or why Core is merging this when they refuse to merge an address index .. but I guess we'll see if there are users who care about this data and who care about it being meaningful on Elements. Also, apologies for the big diff -- there were some mechanical changes to deal with CT amounts, but most of the changes related to the difference in how fees are accounted for. While I'm not thrilled with this PR, its functional test is really good! So I think what I eventually came up with is internally consistent.
Summary: Main commit bitcoin/bitcoin@dd58a4d > index: Add Coinstats index > > The index holds the values previously calculated in coinstats.cpp for each block, representing the state of the UTXO set at each height. ----- bitcoin/bitcoin@57a026c > test: Add unit test for Coinstats index ----- bitcoin/bitcoin@8ea8c92 > index: Avoid unnecessary type casts in coinstatsindex ----- Replace two database writes with a single atomic batch write, to avoid a risk of database corruption. See [[ bitcoin/bitcoin@dd58a4d#r634089179 | the corresponding review ]]. bitcoin/bitcoin@1e38423 > index: Use batch writing in coinstatsindex WriteBlock ----- This is a backport of [[bitcoin/bitcoin#19521 | core#19521]] [4 & 8/17] and [[bitcoin/bitcoin#22047 | core#22047]] [1 & 3/3] Depends on D11597 Test Plan: `ninja all check-all` Reviewers: #bitcoin_abc, sdulfari Reviewed By: #bitcoin_abc, sdulfari Subscribers: sdulfari Differential Revision: https://reviews.bitcoinabc.org/D11598
Summary: This is a backport of [[bitcoin/bitcoin#19521 | core#19521]] [5/17] bitcoin/bitcoin@3c914d5 partial bitcoin/bitcoin@6a4c0c0 The functional test only checks that "-coinstatsindex" argument does not break anything. The rest of the functional test from commit 6a4c0c09ab is not yet applicable and will be added in the next commit. This includes also minor documentation fixups from [[ bitcoin/bitcoin#21818 | core#21818]] Depends on D11598 and D11595 Test Plan: `ninja all check-all` Reviewers: #bitcoin_abc, Fabien Reviewed By: #bitcoin_abc, Fabien Subscribers: Fabien Differential Revision: https://reviews.bitcoinabc.org/D11599
Summary: This is a backport of [[bitcoin/bitcoin#19521 | core#19521]] [6 & 7/17] bitcoin/bitcoin@3f166ec bitcoin/bitcoin@6a4c0c0 (removal of 'disk_size' and 'transactions' from the index and test the RPC on specific heights) Notes: - in the source commits there is a call to `::ChainstateActive().ForceFlushStateToDisk();` added, but it is removed in a following commit in the same PR. This is obviously duplicated with the `active_chainstate.ForceFlushStateToDisk()` a few lines later, so I omitted it. - I added a mention about `disk_size` not being available when using the index in the doc of the RPC call. This is added later in the same PR in an unrelated commit in the source material. - the new `hash_or_height` RPC parameter needs to be added to the `CRPCConvertParam`table for the unit tests to pass. This is done in a later commit in the same PR in the source material Depends on D11599 Test Plan: `ninja all check-all` Reviewers: #bitcoin_abc, Fabien Reviewed By: #bitcoin_abc, Fabien Subscribers: Fabien Differential Revision: https://reviews.bitcoinabc.org/D11600
Summary: This is a backport of [[bitcoin/bitcoin#19521 | core#19521]] [9 & 10/17] bitcoin/bitcoin@ca01bb8 bitcoin/bitcoin@655d929 Depends on D11600 Test Plan: `ninja && test/functional/test_runner.py rpc_misc` Reviewers: #bitcoin_abc, Fabien Reviewed By: #bitcoin_abc, Fabien Subscribers: Fabien Differential Revision: https://reviews.bitcoinabc.org/D11604
Summary: This is a backport of [[bitcoin/bitcoin#19521 | core#19521]] [11/17] and [[bitcoin/bitcoin#22047 | core#22047]] [2/3] bitcoin/bitcoin@2501576 bitcoin/bitcoin@fb65dde Depends on D11604 Test Plan: `ninja all check-all` Reviewers: #bitcoin_abc, Fabien Reviewed By: #bitcoin_abc, Fabien Differential Revision: https://reviews.bitcoinabc.org/D11605
Summary: This is a backport of [[bitcoin/bitcoin#19521 | core#19521]] [12/17] bitcoin/bitcoin@e0938c2 Note that there are minor differences in some amounts compared to the source commit due to differences in tx fees. Depends on D11605 Test Plan: `ninja check-functional` Reviewers: #bitcoin_abc, Fabien Reviewed By: #bitcoin_abc, Fabien Differential Revision: https://reviews.bitcoinabc.org/D11606
… test for coinstatsindex behavior in reorgs Summary: This is a backport of [[bitcoin/bitcoin#19521 | core#19521]] [15 & 16/17] bitcoin/bitcoin@90c966b bitcoin/bitcoin@23fe504 Co-authored-by: Sjors Provoost <email@example.com> Depends on D11608 Test Plan: `ninja all check-all` Reviewers: #bitcoin_abc, Fabien Reviewed By: #bitcoin_abc, Fabien Differential Revision: https://reviews.bitcoinabc.org/D11609