blockchain: Implement header proof storage. #2938
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
davecgh
added
the
database upgrade
davecgh
force-pushed
the
blockchain_implement_header_proof_storage
branch
4 times, most recently
from
d040850
to
59a8b63
Compare
JoeGruffins
reviewed
May 10, 2022
There was a problem hiding this comment.
Updated in only 8 seconds on mainnet.
I hit a panic after the upgrade on testnet, not sure if related to these changes atm:
panic
2022-05-10 14:27:03.218 [INF] CHAN: Done storing header commitments. Total entries: 589725 in 14.283s
2022-05-10 14:27:03.218 [INF] CHAN: Done upgrading database in 14.283s.
2022-05-10 14:27:03.218 [INF] CHAN: Loading block index...
2022-05-10 14:27:09.315 [INF] STKE: Stake database version 1 loaded
2022-05-10 14:27:09.321 [INF] CHAN: Deployment version 10 loaded
2022-05-10 14:27:09.321 [INF] CHAN: UTXO cache initializing (max size: 150 MiB)...
2022-05-10 14:27:09.321 [INF] CHAN: UTXO cache initialization completed
2022-05-10 14:27:09.321 [INF] CHAN: Blockchain database version info: chain: 13, compression: 1, block index: 3, spend journal: 3
2022-05-10 14:27:09.321 [INF] CHAN: UTXO database version info: version: 2, compression: 1, utxo set: 3
2022-05-10 14:27:09.321 [INF] CHAN: Best known header: height 912950, hash 00000000012ba2fb0f2c369e61e6da12327a567b338611d7e15ce58ba3667638
2022-05-10 14:27:09.321 [INF] CHAN: Chain state: height 912950, hash 00000000012ba2fb0f2c369e61e6da12327a567b338611d7e15ce58ba3667638, total transactions 13542309, work 17543353012825557485, progress 100.00%
2022-05-10 14:27:09.321 [INF] INDX: Transaction index is enabled
2022-05-10 14:27:09.322 [INF] INDX: Address index is enabled
2022-05-10 14:27:09.322 [INF] INDX: address index: recovering from tip 691713 (000000000005707b04e7099cae5a13e841050b7f945826b40eaf5cdca162839b)
2022-05-10 14:27:09.339 [CRT] CHAN: missing spend journal data for 000000000005707b04e7099cae5a13e841050b7f945826b40eaf5cdca162839b
2022-05-10 14:27:09.339 [INF] DCRD: Gracefully shutting down the UTXO database...
2022-05-10 14:27:09.339 [INF] DCRD: Gracefully shutting down the block database...
2022-05-10 14:27:09.445 [INF] DCRD: Shutdown complete
panic: missing spend journal data for 000000000005707b04e7099cae5a13e841050b7f945826b40eaf5cdca162839b [recovered]
panic: missing spend journal data for 000000000005707b04e7099cae5a13e841050b7f945826b40eaf5cdca162839b
goroutine 1 [running]:
github.com/decred/dcrd/database/v3/ffldb.rollbackOnPanic(0xc0000fab40)
/home/joe/git/dcrd/database/ffldb/db.go:1894 +0x5b
panic({0x9946c0, 0xc010d71720})
/usr/lib/go/src/runtime/panic.go:838 +0x207
github.com/decred/dcrd/blockchain/v5.panicf({0xa73b15?, 0xc0000c9918?}, {0xc000432d88?, 0xc010fa2c68?, 0xc000432d40?})
/home/joe/git/dcrd/blockchain/chain.go:56 +0xa7
github.com/decred/dcrd/blockchain/v5.dbFetchSpendJournalEntry({0xc39d90?, 0xc0000fab40?}, 0xc0094d2a80, 0x1)
/home/joe/git/dcrd/blockchain/chainio.go:774 +0x333
github.com/decred/dcrd/blockchain/v5.(*ChainQueryerAdapter).PrevScripts(0xc00b1e6248, {0xc39d90, 0xc0000fab40}, 0xc0094d2a80)
/home/joe/git/dcrd/blockchain/chain.go:2185 +0x65
github.com/decred/dcrd/blockchain/v5/indexers.recoverIndex.func1({0xc39d90, 0xc0000fab40})
/home/joe/git/dcrd/blockchain/indexers/common.go:536 +0xdb
github.com/decred/dcrd/database/v3/ffldb.(*db).Update(0xc0002c2340?, 0xc006280e60)
/home/joe/git/dcrd/database/ffldb/db.go:1951 +0x8b
github.com/decred/dcrd/blockchain/v5/indexers.recoverIndex({0xc37a50?, 0xc00013a000}, {0xc39f20, 0xc0094d2800})
/home/joe/git/dcrd/blockchain/indexers/common.go:527 +0x571
github.com/decred/dcrd/blockchain/v5/indexers.(*AddrIndex).Init(0xc0005f0050?, {0xc37a50, 0xc00013a000}, 0xc0001c4f00)
/home/joe/git/dcrd/blockchain/indexers/addrindex.go:646 +0xeb
github.com/decred/dcrd/blockchain/v5/indexers.NewAddrIndex(0xc0005f0050, {0xc38b10?, 0xc00009a080}, {0xc39ea0, 0xc00b1e6248})
/home/joe/git/dcrd/blockchain/indexers/addrindex.go:1164 +0x235
main.newServer({0xc37a50, 0xc00013a300}, {0xc000133b70, 0x1, 0x1}, {0xc38b10, 0xc00009a080}, 0xc0004f41c0, 0xc0001c4f00, {0xc000028d80, ...})
/home/joe/git/dcrd/server.go:3538 +0xd25
main.dcrdMain()
/home/joe/git/dcrd/dcrd.go:207 +0xe6a
main.main()
/home/joe/git/dcrd/dcrd.go:264 +0x8c
It also worked fine on a separate testnet db. It looks like the block hash it spits out was an orphan? https://testnet.dcrdata.org/block/000000000005707b04e7099cae5a13e841050b7f945826b40eaf5cdca162839b
Actually it happens with either db if I run with --addrindex
Is not related to these changes. Made an issue #2944
JoeGruffins
approved these changes
May 10, 2022
davecgh
force-pushed
the
blockchain_implement_header_proof_storage
branch
2 times, most recently
from
6063889
to
0d218f9
Compare
davecgh
force-pushed
the
blockchain_implement_header_proof_storage
branch
2 times, most recently
from
f9b1ef4
to
46aaf46
Compare
davecgh
force-pushed
the
blockchain_implement_header_proof_storage
branch
from
46aaf46
to
aa03b84
Compare
matheusd
approved these changes
May 30, 2022
There was a problem hiding this comment.
Tested successfully in testnet and mainnet. Mainnet upgrade took 25s.
Also tested fetching the filters via RPC for a block before, at activation and after activation, before and after the upgrade to double check the API.
Only added a small note on a comment, but otherwise looks good to me.
Comment on lines
+4748
to
+4751
| case wire.MainNet: | ||
| storeFromHeight = 431488 | ||
| case wire.TestNet3: | ||
| storeFromHeight = 323328 |
This adds a field to the header commitment data struct to store the hash of the filter along with the filter so that it can be reused without needing to recalculate the hash. starting
This adds logic to avoid hitting the database when a compact filter that can't possibly be available by first checking the block index to ensure the requested block is both available and its data is known.
This modifies the chain logic to create and store the individual commitment hashes covered by the commitment root field of the header of each block and also adds code to migrate the database to retroactively create and store entries for all applicable historical blocks. The upgrade can be interrupted at any point and future invocations will resume from the point it was interrupted. The following is a high level overview of the changes: - Introduce a new database bucket to house the header commitments - Add serialization code for use when storing and loading the individual header commitment hashes - Add full test coverage for new serialization code - Store the commitment hashes in the db when connecting blocks - Implement database migration code to retroactively store the commitment hashes for all applicable historical blocks - Bump the chain database version to 13 - Support resuming from interrupted upgrades - Add a new func on the internal header commitment data struct that returns the v1 header commitment hashes to consolidate the logic - Update FilterByBlockHash to load the header commitments from the db and generate the inclusion proof accordingly
davecgh
force-pushed
the
blockchain_implement_header_proof_storage
branch
2 times, most recently
from
a262188
to
bba1ed8
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This requires #2937.Testing Notes
As of this PR, the expected behavior is that there is a single migration that takes around 1 to 2 minutes to complete. after which it will no longer be possible to downgrade.
As the warning above notes, if you try to run an older software version after this migration has completed, you will get an error message similar to
Unable to start server on [:9108]: the current blockchain database is no longer compatible with this version of the software (13 > 12).This modifies the chain logic to create and store the individual commitment hashes covered by the commitment root field of the header of each block, adds code to migrate the database to retroactively create and store entries for all applicable historical blocks, and updates the code that generates inclusion proofs to use the stored data instead so that it will continue to work properly when new header commitments are added without further modification.
The upgrade can be interrupted at any point and future invocations will resume from the point it was interrupted.
The following is a high level overview of the changes:
FilterByBlockHashimplementation: