assumeutxo (2)

[jamesob]

• Background and FAQ: https://github.com/jamesob/assumeutxo-docs/tree/2019-04-proposal/proposal
• Prior progress/project: https://github.com/bitcoin/bitcoin/projects/11
• Replaces assumeutxo #15606, which was closed due to Github slowness. Original description and commentary can be found there.

---

This changeset finishes the first phase of the assumeutxo project. It makes UTXO snapshots loadable via RPC (loadtxoutset) and adds assumeutxo parameters to chainparams. It contains all the remaining changes necessary to both use an assumedvalid snapshot chainstate and do a full validation sync in the background.

This may look like a lot to review, but note that

• ~200 lines are a (non-essential) demo shell script
• Many lines are functional test, documentation, and relatively dilute RPC code.

So it shouldn't be as burdensome to review as the linecount might suggest.

• P2P: minor changes are made to init.cpp and net_processing.cpp to make simultaneous IBD across multiple chainstates work.
• Pruning: implement correct pruning behavior when using a background chainstate
• Blockfile separation: to prevent "fragmentation" in blockfile storage, have background chainstates use separate blockfiles from active snapshot chainstates to avoid interleaving heights and impairing pruning.
• Indexing: some CValidationInterface events are given with an additional parameter, ChainstateRole, and all indexers ignore events from ChainstateRole::ASSUMEDVALID so that indexation only happens sequentially.
• Have -reindex properly wipe snapshot chainstates.
• RPC: introduce RPC commands loadtxoutset and (hidden) getchainstates.
• Release docs & first assumeutxo commitment: add notes and a particular assumeutxo hash value for first AU-enabled release.
  • This will complete the project and allow use of UTXO snapshots for faster node bootstrap.

The next phase, if it were to be pursued, would be coming up with a way to distribute the UTXO snapshots over the P2P network.

---

UTXO snapshots

Create your own with ./contrib/devtools/utxo_snapshot.sh, e.g.

./contrib/devtools/utxo_snapshot.sh 788000 utxo.dat ./src/bitcoin-cli -datadir=$(pwd)/testdata`)

or use the pre-generated ones listed below.

• Testnet: 2'500'000 (Sjors):
  • torrent: magnet:?xt=urn:btih:511e09f4bf853aefab00de5c070b1e031f0ecbe9&dn=utxo-testnet-2500000.dat&tr=udp%3A%2F%2Ftracker.bitcoin.sprovoost.nl%3A6969
  • sha256: 79db4b025448cc0ac388d8589a28eab02de53055d181e34eb47391717aa16388
• Signet: 160'000 (Sjors):
  • torrent: magnet:?xt=urn:btih:9da986cb27b3980ea7fd06b21e199b148d486880&dn=utxo-signet-160000.dat&tr=udp%3A%2F%2Ftracker.bitcoin.sprovoost.nl%3A6969
  • sha256: eeeca845385ba91e84ef58c09d38f98f246a24feadaad57fe1e5874f3f92ef8c
• Mainnet: 800'000 (Sjors):
  • Note: this needs the following commit cherry-picked in: Sjors@24deb20
  • torrent: magnet:?xt=urn:btih:50ee955bef37f5ec3e5b0df4cf0288af3d715a2e&dn=utxo-800000.dat&tr=udp%3A%2F%2Ftracker.bitcoin.sprovoost.nl%3A6969

Testing

For fun (~5min)

If you want to do a quick test, you can run ./contrib/devtools/test_utxo_snapshots.sh and follow the instructions. This is mostly obviated by the functional tests, though.

For real (longer)

If you'd like to experience a real usage of assumeutxo, you can do that too.
I've cut a new snapshot at height 788'000 (http://img.jameso.be/utxo-788000.dat - but you can do it yourself with ./contrib/devtools/utxo_snapshot.sh if you want). Download that, and then create a datadir for testing:

$ cd ~/src/bitcoin  # or whatever

# get the snapshot
$ curl http://img.jameso.be/utxo-788000.dat > utxo-788000.dat

# you'll want to do this if you like copy/pasting 
$ export AU_DATADIR=/home/${USER}/au-test # or wherever

$ mkdir ${AU_DATADIR}
$ vim ${AU_DATADIR}/bitcoin.conf

dbcache=8000  # or, you know, something high
blockfilterindex=1
coinstatsindex=1
prune=3000
logthreadnames=1

Obtain this branch, build it, and then start bitcoind:

$ git remote add jamesob https://github.com/jamesob/bitcoin
$ git fetch jamesob assumeutxo
$ git checkout jamesob/assumeutxo

$ ./configure $conf_args && make  # (whatever you like to do here)

# start 'er up and watch the logs
$ ./src/bitcoind -datadir=${AU_DATADIR}

Then, in some other window, load the snapshot

$ ./src/bitcoin-cli -datadir=${AU_DATADIR} loadtxoutset $(pwd)/utxo-788000.dat

You'll see some log messages about headers retrieval and waiting to see the snapshot in the headers chain. Once you get the full headers chain, you'll spend a decent amount of time (~10min) loading the snapshot, checking it, and flushing it to disk. After all that happens, you should be syncing to tip in pretty short order, and you'll see the occasional [background validation] log message go by.

In yet another window, you can check out chainstate status with

$ ./src/bitcoin-cli -datadir=${AU_DATADIR} getchainstates

as well as usual favorites like getblockchaininfo.

[DrahtBot]

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Reviews

See the guideline for information on the review process.

Type	Reviewers
ACK	achow101
Concept ACK	D33r-Gee, Sjors
Stale ACK	pablomartin4btc, fjahr, naumenkogs, ryanofsky

If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

Conflicts

Reviewers, this pull request conflicts with the following ones:

• #28550 (Covenant tools softfork by jamesob)
• #28453 (wallet: Receive silent payment transactions by achow101)
• #28368 (Fee Estimator updates from Validation Interface/CScheduler thread by ismaelsadeeq)
• #28120 (p2p: make block download logic aware of limited peers threshold by furszy)
• #28031 (Package Relay 1/3: Introduce TxDownloadManager and improve orphan-handling by glozow)
• #27837 (net: introduce block tracker to retry to download blocks after failure by furszy)
• #27770 (Introduce 'getblockfileinfo' RPC command by furszy)
• #26966 (index: blockfilter initial sync speedup, parallelize process by furszy)
• #26762 (bugfix: Make CCheckQueue RAII-styled (attempt 2) by hebasto)
• #26045 (rpc: Optimize serialization disk space of dumptxoutset by aureleoules)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

[jamesob]

CI's passing after a silent conflict in the rebase. I've added a link to @Sjors' snapshot torrent in the PR description.

No known outstanding issues here; ready for testing!

[jamesob]

Rebased.

[Sjors]

When running 8f431ad I noticed (and reproduced) that -prune is not fully honored when loading a snapshot and doing background validation. When I set -prune=550 the blocks dir ends up somewhere between 3 and 5 GB. Judging by the blk….dat timestamps it seems that both the snapshot and background IBD hold on to more blocks than they should. Tested on Ubuntu 23.04 and with coinstats- and blockfilterindex enabled. The first time I tested I allowed the node to sync for a few hours before loading the snapshot, the second time I loaded the snapshot almost immediately.

[fjahr]

@jamesob What's your feeling on how important figuring out the fix for the pruning issue is at the moment? People don't consider it blocking #24008, right? Just curious about where to spend review time right now. FWIW, I noted the same issue on getblockfrompeer and wrote a test for it back then, in case you haven't seen it, it might come in handy: #23813

[jamesob]

What's your feeling on how important figuring out the fix for the pruning issue is at the moment? People don't consider it blocking #24008, right?

Right, pruning issues shouldn't block #24008 - those changes need to happen regardless of how we manage pruning with snapshots.

I'll reproduce/investigate the pruning issues over the coming days. But it's worth noting that @Sjors didn't see any regressions in pruning on this branch before loading the snapshot, meaning the degraded pruning behavior only kicks in if the user loads a snapshot. And even though, the code just doesn't prune as aggressively as it seems it should.

[jamesob]

Hm, for what it's worth, pruning is working as expected for me on this branch.

Every 2.0s: cat <(./src/bitcoin-cli -datadir=/home/james/tmp/bitcoin-au-testing getmempoolinfo) <( ./src/bitcoin-cli -datadir=/home/james/tmp/bitcoin-au-testing getchainstates)                                                                                                                                                                         fido: Wed May 10 11:16:49 2023

{
  "loaded": true,
  "size": 108146,
  "bytes": 42192499,
  "usage": 234882016,
  "total_fee": 23.04459003,
  "maxmempool": 300000000,
  "mempoolminfee": 0.00013381,
  "minrelaytxfee": 0.00001000,
  "incrementalrelayfee": 0.00001000,
  "unbroadcastcount": 0,
  "fullrbf": false
}
{
  "active_chain_type": "validated_snapshot",
  "validated_snapshot": {
    "blocks": 789094,
    "bestblockhash": "00000000000000000002dad6b3cd82fc00075978eb0d63dcbf6bce3bdc2c3f32",
    "difficulty": 48005534313578.78,
    "verificationprogress": 0.9999931560421257,
    "snapshot_blockhash": "00000000000000000001f3fa1b4c03c877740778f56b0d5456b18dd88f7f695e",
    "initialblockdownload": false,
    "coins_db_cache_bytes": 8388608,
    "coins_tip_cache_bytes": 11498684416,
    "has_mempool": true
  },
  "headers": 789094
}

Will attempt to repro with a fresh run, I guess?

[jamesob]

Rebased.

[Sjors]

Reviewed the remaining commits. I'll try to pick off some comments from the past few days as followup(s) and/or issues.

[Sjors]

In general a user for who 0.5 GB makes a difference is going to have a problem. The downloaded snapshot itself is multiple GB and the chainstate (even without assumeutxo) is growing at 0.5 GB per month. We could add a hint that it's safe to delete utxo-xxxxxx.dat once the RPC returns (if there's no crash).

[Sjors]

373cf91 being able to build out of order is necessary for a pruned node.

[note: currently running an IBD on mainnet with -prune=2000 and -blockfilterindex=1 to see what happens… I'm guessing it will be missing a range of blocks above the snapshot? If so then we should either refuse to load a snapshot or delay pruning with this combination. If we delay pruning, we should warn the user that enabling the index will temporarily require a lot of extra storage (x GB per month of how old the snapshot is)]

[hernanmarino]

post merge ACK . I've been testing thoroughly on mainnet, and everything worked as expected. I 'll provide more detalied feedback on follow ups.

[Sjors]

Imo most post-merge comments have now been addressed in either followup PRs or open issues: #28608, #28616, #28618, #28620, #28621, #28619, (already merged: #28590, #28589, #28562)

The tracing issue could be updated to point to them.

These seem unaddressed:

• assumeutxo (2) #27596 (comment) (something racy)
• assumeutxo (2) #27596 (comment) (dead code to begin with?)
  assumeutxo (2) #27596 (comment) (extra empty mempool check for clearer error message)

[pablomartin4btc]

post-merge re tACK edbed31

Tested on mainnet using utxo-800000.dat file from @Sjors.

./src/bitcoin-cli -datadir=${AU_DATADIR} getchainstates
{
  "headers": 811416,
  "chainstates": [
    {
      "blocks": 49020,
      "bestblockhash": "00000000283ec5677b8757c0b652e67087b968e337e9779b35d9092bf4003882",
      "difficulty": 6.085476906000794,
      "verificationprogress": 5.482400033922018e-05,
      "coins_db_cache_bytes": 419430,
      "coins_tip_cache_bytes": 784649420,
      "validated": true
    },
    {
      "blocks": 800000,
      "bestblockhash": "00000000000000000002a7c4c1e48d76c5a37902165a270156b7a8d72728a054",
      "difficulty": 53911173001054.59,
      "verificationprogress": 0.9591139668014443,
      "coins_db_cache_bytes": 7969177,
      "coins_tip_cache_bytes": 14908338995,
      "snapshot_blockhash": "00000000000000000002a7c4c1e48d76c5a37902165a270156b7a8d72728a054",
      "validated": false
    }
  ]
}

I think description should be updated removing references to previous snapshot file from @jamesob, as if a reviewer uses that file without updating the code in src/kernel/chainparams.cpp according to @Sjors commit but with correct details of 788000 block, running loadtxoutset will error with:

/src/bitcoin-cli -datadir=${AU_DATADIR} loadtxoutset ${AU_DATADIR}/utxo-788000.dat
error code: -32603
error message:
Unable to load UTXO snapshot .../.test_utxo/utxo-788000.dat

Also when re-try to run again loadtxoutset after its completion user would obtain:

./src/bitcoin-cli -datadir=${AU_DATADIR} loadtxoutset ${AU_DATADIR}/utxo-800000.dat
error code: -32603
error message:
Unable to load UTXO snapshot .../.test_utxo_2/utxo-800000.dat

While in the bitcoind logs a user would see the correct issue:

2023-10-09T20:26:13Z [httpworker.2] [snapshot] waiting to see blockheader 00000000000000000002a7c4c1e48d76c5a37902165a270156b7a8d72728a054 in headers chain before snapshot activation
2023-10-09T20:26:13Z [httpworker.2] [snapshot] can't activate a snapshot-based chainstate more than once

And I think when I previously tested 1b1d711, unless I dreamt it, I saw the correct error description can't activate a snapshot-based chainstate more than once as the output from cli.

I agree loadtxoutset timeout issue should be solved.