rpc: allow dumptxoutset to dump human-readable data #18689

pierreN · 2020-04-17T19:40:38Z

Adds additional optional arguments to dumptxoutset. If any are present, a human-readable file is written to disk instead of the compact binary serialized form currently in use. This does not change the current default behavior of dumptxoutset.

Thanks to the future assumeutxo feature (#15605), we now have a dumptxoutset RPC (#16899) which can write the whole UTXO set to disk. However, the current format, although compact, is not easily readable by standard tools (e.g. for someone who would like to study the UTXO set). Plus this binary format might change in the future AFAIK.

Providing power users an easy way to have a human-readable dump of the UTXOs would be a useful feature. We would this way replace 3rd party hackish tools with possible side effects.

On my machine (slow SSD):

dumping the whole original 4GB binary UTXO set takes around 1mn40
dumping the set in whole ASCII form takes less than 9GB and 3mn30 (ofc file size/time depends on which ASCII data you write to disk; you can select them via the format argument).

Thanks!

hebasto · 2020-04-17T20:03:04Z

What are possible/expected use cases?

maflcko · 2020-04-17T20:09:46Z

Concept ACK

DrahtBot · 2020-04-17T22:57:11Z

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Conflicts

Reviewers, this pull request conflicts with the following ones:

Remove GetDataDir(net_specific) function #21850 (Remove GetDataDir(net_specific) function by kiminuo)
validation: UpdateTip/CheckBlockIndex assumeutxo support #21526 (validation: UpdateTip/CheckBlockIndex assumeutxo support by jamesob)
Add scanblocks RPC call #20664 (Add scanblocks RPC call by jonasschnelli)
rpc: getblockfrompeer #20295 (rpc: getblockfrompeer by Sjors)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

pierreN · 2020-04-18T02:16:58Z

@hebasto the most common use case will be for users to easily study the whole UTXO set in only a few minutes.

With this PR it should be really easy to dump any format you want via the format parameter (type of data, number of occurrences and order are all respected).
Also, just adding one element to the ascii_types vector allows you to dump any new type of ASCII data.

For example, you can now trivially plot a graph of when current UTXOs were created (took 3mn30 on my machine):

$ bitcoin-cli dumptxoutset utxos.dat '["height","value"]' false ' '
$ awk '{if($1 in map) { map[$1] += $2; } else { map[$1] = $2 }} END { for(height in map) { print height, map[height]; }}' utxos.dat | sort -n > plot.dat
$ gnuplot -e "plot 'plot.dat' w l; pause -1;"

Another example would be to sum the amount of coinbase values in the UTXO set (took 2mn30):

$ bitcoin-cli dumptxoutset utxos.dat '["coinbase","value"]' false ' '
$ fgrep "1 " utxos.dat | awk '{sum += $2} END {print sum/100000000, "btc unspent coinbase"}'
1.76077e+06 btc unspent coinbase

Or really, just anything the user wants. With some more work from the user, it could also make it easier to track some indicators (such as bootstrapping/syncing SOPR).

Another example: doing bitcoin file archaeology.
Since all methods using scriptPubSig to etch data on the blockchain spam the UTXO set, you can retrieve a superset of all TXIDs of "files stored in the blockchain" via this RPC call.

pierreN · 2020-05-01T05:09:11Z

rebased cd20cb8

brakmic · 2020-05-01T13:50:36Z

ACK cd20cb8

Built, run and tested on macOS Catalina 10.15.4

./test/functional/rpc_dumptxoutset.py
2020-05-01T13:40:08.556000Z TestFramework (INFO): Initializing test directory /var/folders/7q/4ffytzk562dd2ky4bfg9_w7h0000gn/T/bitcoin_func_test_oc2tzp0y
2020-05-01T13:40:11.072000Z TestFramework (INFO): no_option
2020-05-01T13:40:11.113000Z TestFramework (INFO): all_data
2020-05-01T13:40:11.216000Z TestFramework (INFO): partial_data_1
2020-05-01T13:40:11.304000Z TestFramework (INFO): partial_data_order
2020-05-01T13:40:11.369000Z TestFramework (INFO): partial_data_double
2020-05-01T13:40:11.447000Z TestFramework (INFO): no_header
2020-05-01T13:40:11.537000Z TestFramework (INFO): separator
2020-05-01T13:40:11.617000Z TestFramework (INFO): all_options
2020-05-01T13:40:11.748000Z TestFramework (INFO): Stopping nodes
2020-05-01T13:40:12.313000Z TestFramework (INFO): Cleaning up /var/folders/7q/4ffytzk562dd2ky4bfg9_w7h0000gn/T/bitcoin_func_test_oc2tzp0y on exit
2020-05-01T13:40:12.313000Z TestFramework (INFO): Tests successful

./src/bitcoin-cli -regtest dumptxoutset dump.dat '["txid", "vout"]' false ':'
{
  "coins_written": 407,
  "base_hash": "01ba165996f7a7899e56b37584398adb892a5df7566b95e8de457ab588784740",
  "base_height": 407,
  "path": "/Users/brakmic/Library/Application Support/Bitcoin/regtest/dump.dat"
}

cat "/Users/brakmic/Library/Application Support/Bitcoin/regtest/dump.dat"
208c48f15ed2971709d81da915b72255e50b9251c558dc45981632ed6e4cd300:0
338e1fde4b86e2daaba2bd7cb4f8d77e600f47e7814645aafb480f56f4f41103:0
e73b0564bd56d359bd8df64fa3b9fd8586c3ff0430081aff1f97a9600c834403:0
23ff11ec2801f1c4838fc19863f7fa8d9283ac29e644180a6eaba160fd2a9c03:0
03ba026c466ab490a19b0aa8a39abeeccc6cff24d4a24a34d5f4304ae21e5304:0
98e8ceec62fb6442acaa939461482a46d7c03968082430661854815571eca204:0
4b84d555d8a19ab3cc38152e446fdbd059ec535ab67806a61628f238e495ff04:0
[...snip...]

Github-Pull: bitcoin#18689 Rebased-From: cd20cb8

luke-jr · 2020-06-09T23:51:06Z

src/rpc/blockchain.cpp

+    if (!is_compact) {
+        const auto& arr = request.params[1].get_array();
+        const std::unordered_map<std::string, cb_t> ascii_map(std::begin(ascii_types), std::end(ascii_types));
+        for(auto i = 0; i < arr.size(); ++i) {


auto doesn't really work in this context...

warning: comparison of integer expressions of different signedness: ‘int’ and ‘size_t’ {aka ‘long unsigned int’} [-Wsign-compare]

Ha, funny that a compiler can get the warning but don't properly choose the type of i. I guess the 0 must confuse it.

Thanks for catching this, I've just updated the branch.

Github-Pull: bitcoin#18689 Rebased-From: cd20cb8

jamesob · 2020-08-26T01:48:13Z

Cool, I'll take a look in the next few days.

jamesob · 2020-11-19T15:02:24Z

Concept ACK - will review soon.

maflcko · 2021-04-13T18:42:52Z

Still needs rebase

pierreN · 2021-05-06T21:39:07Z

Sorry for the few months delay.
I have a bit more time now and will try to follow through with this PR.

I'll update the branch in a few days (I was syncing when my old SSD died).

Sjors · 2021-05-21T13:41:07Z

Consider moving this functionality to the new bitcoin-util instead. You could add a command that converts the binary format to human readable.

DrahtBot · 2021-05-24T09:40:32Z

🐙 This pull request conflicts with the target branch and needs rebase.

_{Want to unsubscribe from rebase notifications on this pull request? Just convert this pull request to a "draft".}

luke-jr · 2021-10-11T06:17:50Z

src/rpc/blockchain.cpp

@@ -2564,7 +2608,7 @@ static RPCHelpMan dumptxoutset()
    };
 }

-UniValue CreateUTXOSnapshot(NodeContext& node, CChainState& chainstate, CAutoFile& afile)
+UniValue CreateUTXOSnapshot(const bool is_compact, const bool show_header, const std::string& separator, NodeContext& node, CChainState& chainstate, CAutoFile& afile, const std::vector<std::pair<std::string, coinascii_cb_t>>& requested)


IMO it'd be nicer to avoid the two mutually-exclusive bools. Maybe a good case for a class enum?

luke-jr · 2021-10-11T06:19:01Z

src/test/validation_chainstatemanager_tests.cpp

@@ -183,7 +183,7 @@ CreateAndActivateUTXOSnapshot(NodeContext& node, const fs::path root, F malleati
    FILE* outfile{fsbridge::fopen(snapshot_path, "wb")};
    CAutoFile auto_outfile{outfile, SER_DISK, CLIENT_VERSION};

-    UniValue result = CreateUTXOSnapshot(node, node.chainman->ActiveChainstate(), auto_outfile);
+    UniValue result = CreateUTXOSnapshot(false, false, "", node, node.chainman->ActiveChainstate(), auto_outfile, {});


The first false here should be true, as ActivateSnapshot can only handle the binary/compact format.

josibake · 2021-12-28T10:43:53Z

Concept ACK

@pierreN are you still working on this? I'm happy to try and take it over the finish line if you're not.

jamesob · 2022-01-03T19:10:48Z

re-Concept ACK and at a high-level the code looks pretty good. Nice job on the tests. In need of a rebase though.

Github-Pull: bitcoin#18689 Rebased-From: 65d0697 [partial]

…le output Github-Pull: bitcoin#18689 Rebased-From: 65d0697 [partial]

luke-jr · 2022-02-08T21:04:56Z

If you decide to revive this PR, I've done an extensive rebase at master...luke-jr:rpc_dumptxoutset_hr (leave off the last commit), rebasing it on top of (but not compatible with) #24202, and splitting up the different functionality across multiple commits.

maflcko · 2022-03-22T09:11:00Z

I think this was picked up in ##24202 , so can be closed?

DrahtBot added RPC/REST/ZMQ Tests labels Apr 17, 2020

This was referenced Apr 18, 2020

rpc: remove deprecated CRPCCommand constructor #18531

Merged

rpc: add extensive file checks for dumptxoutset and dumpwallet #17623

Closed

maflcko mentioned this pull request Apr 25, 2020

[Feature] Unspent Transaction Outputs Index #4007

Closed

DrahtBot mentioned this pull request Apr 29, 2020

rpc: Do not advertise dumptxoutset as a way to flush the chainstate #18809

Merged

DrahtBot added the Needs rebase label Apr 30, 2020

pierreN force-pushed the feature-utxo-ascii branch 2 times, most recently from ca0df84 to cd20cb8 Compare May 1, 2020 05:08

DrahtBot removed the Needs rebase label May 1, 2020

DrahtBot mentioned this pull request May 7, 2020

assumeutxo #15606

Closed

18 tasks

luke-jr pushed a commit to bitcoinknots/bitcoin that referenced this pull request Jun 9, 2020

rpc: allow dumptxoutset to dump human-readable data

8cf4bf7

Github-Pull: bitcoin#18689 Rebased-From: cd20cb8

luke-jr reviewed Jun 9, 2020

View reviewed changes

luke-jr pushed a commit to bitcoinknots/bitcoin that referenced this pull request Jun 10, 2020

rpc: allow dumptxoutset to dump human-readable data

6c0bf88

Github-Pull: bitcoin#18689 Rebased-From: cd20cb8

pierreN force-pushed the feature-utxo-ascii branch from cd20cb8 to 82046cf Compare June 14, 2020 08:56

DrahtBot mentioned this pull request Jun 16, 2020

Replace boost::filesystem with std::filesystem #19245

Closed

5 tasks

This was referenced Jul 8, 2020

Use operator/ in fs::absolute to prepare for C++17 #19466

Closed

Log RPC parameters (arguments) if -debug=rpcparams #16365

Closed

DrahtBot mentioned this pull request Aug 26, 2020

validation: UTXO snapshot activation #19806

Merged

DrahtBot mentioned this pull request Aug 31, 2020

Assert that RPCArg names are equal to CRPCCommand ones (blockchain,rawtransaction) #19849

Merged

DrahtBot added the Needs rebase label Sep 22, 2020

maflcko removed the Tests label Nov 20, 2020

maflcko mentioned this pull request Apr 13, 2021

How to convert dumptxoutset rpc result to human readable JSON #21670

Open

maflcko added the Up for grabs label Apr 13, 2021

DrahtBot removed the Needs rebase label May 6, 2021

rpc: allow dumptxoutset to dump human-readable data

65d0697

pierreN force-pushed the feature-utxo-ascii branch from c109bce to 65d0697 Compare May 6, 2021 21:11

This was referenced May 7, 2021

Remove GetDataDir(net_specific) function #21850

Merged

validation: UpdateTip/CheckBlockIndex assumeutxo support #21526

Merged

Add scanblocks RPC call #20664

Closed

rpc: getblockfrompeer #20295

Merged

DrahtBot added the Needs rebase label May 24, 2021

luke-jr suggested changes Oct 11, 2021

View reviewed changes

w0xlt mentioned this pull request Jan 29, 2022

rpc: allow dumptxoutset to dump human-readable data #24202

Closed

luke-jr pushed a commit to luke-jr/bitcoin that referenced this pull request Feb 8, 2022

rpc: Strange (but historical) header formatting for dumptxoutset

17d3ed7

Github-Pull: bitcoin#18689 Rebased-From: 65d0697 [partial]

luke-jr pushed a commit to luke-jr/bitcoin that referenced this pull request Feb 8, 2022

rpc: Allow choosing fields for human-readable dumptxoutset output

6511adb

Github-Pull: bitcoin#18689 Rebased-From: 65d0697 [partial]

luke-jr pushed a commit to luke-jr/bitcoin that referenced this pull request Feb 8, 2022

rpc: Add dumptxoutset param to disable header on human-readable output

648dc4d

Github-Pull: bitcoin#18689 Rebased-From: 65d0697 [partial]

luke-jr pushed a commit to luke-jr/bitcoin that referenced this pull request Feb 8, 2022

rpc: Enable changing the field delimiter in dumptxoutset human-readab…

61f3da3

…le output Github-Pull: bitcoin#18689 Rebased-From: 65d0697 [partial]

prusnak mentioned this pull request Mar 21, 2022

rpc: dumptxoutset as sqlite file #24628

Open

maflcko removed the Up for grabs label Mar 22, 2022

dunxen mentioned this pull request Apr 26, 2022

rpc: Add sqlite format option for dumptxoutset #24952

Closed

fanquake closed this May 12, 2022

theStack mentioned this pull request Apr 6, 2023

contrib: add tool to convert compact-serialized UTXO set to SQLite database #27432

Open

bitcoin locked and limited conversation to collaborators May 12, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

rpc: allow dumptxoutset to dump human-readable data #18689

rpc: allow dumptxoutset to dump human-readable data #18689

pierreN commented Apr 17, 2020

hebasto commented Apr 17, 2020

maflcko commented Apr 17, 2020

DrahtBot commented Apr 17, 2020 •

edited

Loading

pierreN commented Apr 18, 2020 •

edited

Loading

pierreN commented May 1, 2020

brakmic commented May 1, 2020

luke-jr Jun 9, 2020

pierreN Jun 14, 2020 •

edited

Loading

jamesob commented Aug 26, 2020 •

edited

Loading

jamesob commented Nov 19, 2020

maflcko commented Apr 13, 2021

pierreN commented May 6, 2021

Sjors commented May 21, 2021 •

edited

Loading

DrahtBot commented May 24, 2021

luke-jr Oct 11, 2021

luke-jr Oct 11, 2021

josibake commented Dec 28, 2021

jamesob commented Jan 3, 2022

luke-jr commented Feb 8, 2022

maflcko commented Mar 22, 2022

rpc: allow dumptxoutset to dump human-readable data #18689

rpc: allow dumptxoutset to dump human-readable data #18689

Conversation

pierreN commented Apr 17, 2020

hebasto commented Apr 17, 2020

maflcko commented Apr 17, 2020

DrahtBot commented Apr 17, 2020 • edited Loading

Conflicts

pierreN commented Apr 18, 2020 • edited Loading

pierreN commented May 1, 2020

brakmic commented May 1, 2020

luke-jr Jun 9, 2020

Choose a reason for hiding this comment

pierreN Jun 14, 2020 • edited Loading

Choose a reason for hiding this comment

jamesob commented Aug 26, 2020 • edited Loading

jamesob commented Nov 19, 2020

maflcko commented Apr 13, 2021

pierreN commented May 6, 2021

Sjors commented May 21, 2021 • edited Loading

DrahtBot commented May 24, 2021

luke-jr Oct 11, 2021

Choose a reason for hiding this comment

luke-jr Oct 11, 2021

Choose a reason for hiding this comment

josibake commented Dec 28, 2021

jamesob commented Jan 3, 2022

luke-jr commented Feb 8, 2022

maflcko commented Mar 22, 2022

DrahtBot commented Apr 17, 2020 •

edited

Loading

pierreN commented Apr 18, 2020 •

edited

Loading

pierreN Jun 14, 2020 •

edited

Loading

jamesob commented Aug 26, 2020 •

edited

Loading

Sjors commented May 21, 2021 •

edited

Loading