database: Major redesign of database package. #380
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
davecgh
force-pushed
the
database2
branch
4 times, most recently
from
7c1708e
to
0833b32
Compare
|
Rebased and updated for |
| // and lookup map. | ||
| // | ||
| // openBlocksLRU tracks how the open files are refenced by pushing the | ||
| // least recently used files to end of the list. When a file needs to |
There was a problem hiding this comment.
It's right. Most recently used goes to the beginning, least recently used to the end.
There was a problem hiding this comment.
However I'll reword it in terms of the most recently used since I can see how that might be worded in a confusing fashion.
|
Rebased to latest master. |
|
Rebased to latest master. |
|
Rebased to latest master. |
davecgh
force-pushed
the
database2
branch
2 times, most recently
from
0e37dac
to
525e996
Compare
|
OK, so I'm making a push to get this merged within the new couple of days. It has been running for months with no issues. Calling for final reviews. |
|
Reviewed 29 of 42 files at r1, 4 of 7 files at r2, 1 of 2 files at r4, 5 of 8 files at r5, 5 of 5 files at r6. Comments from the review on Reviewable.io |
|
LGTM 👍 I've been running with database2 in production for a few months and it works great. |
davecgh
force-pushed
the
database2
branch
3 times, most recently
from
34fee5d
to
8b2bd5d
Compare
|
OK from me. Been running for months without issue. |
|
ok here too. |
This commit contains a complete redesign and rewrite of the database package that approaches things in a vastly different manner than the previous version. This is the first part of several stages that will be needed to ultimately make use of this new package. Some of the reason for this were discussed in btcsuite#255, however a quick summary is as follows: - The previous database could only contain blocks on the main chain and reorgs required deleting the blocks from the database. This made it impossible to store orphans and could make external RPC calls for information about blocks during the middle of a reorg fail. - The previous database interface forced a high level of bitcoin-specific intelligence such as spend tracking into each backend driver. - The aforementioned point led to making it difficult to implement new backend drivers due to the need to repeat a lot of non-trivial logic which is better handled at a higher layer, such as the blockchain package. - The old database stored all blocks in leveldb. This made it extremely inefficient to do things such as lookup headers and individual transactions since the entire block had to be loaded from leveldb (which entails it doing data copies) to get access. In order to address all of these concerns, and others not mentioned, the database interface has been redesigned as follows: - Two main categories of functionality are provided: block storage and metadata storage - All block storage and metadata storage are done via read-only and read-write MVCC transactions with both manual and managed modes - Support for multiple concurrent readers and a single writer - Readers use a snapshot and therefore are not blocked by the writer - Some key properties of the block storage and retrieval API: - It is generic and does NOT contain additional bitcoin logic such spend tracking and block linking - Provides access to the raw serialized bytes so deserialization is not forced for callers that don't need it - Support for fetching headers via independent functions which allows implementations to provide significant optimizations - Ability to efficiently retrieve arbitrary regions of blocks (transactions, scripts, etc) - A rich metadata storage API is provided: - Key/value with arbitrary data - Support for buckets and nested buckets - Bucket iteration through a couple of different mechanisms - Cursors for efficient and direct key seeking - Supports registration of backend database implementations - Comprehensive test coverage - Provides strong documentation with example usage This commit also contains an implementation of the previously discussed interface named ffldb (flat file plus leveldb metadata backend). Here is a quick overview: - Highly optimized for read performance with consistent write performance regardless of database size - All blocks are stored in flat files on the file system - Bulk block region fetching is optimized to perform linear reads which improves performance on spindle disks - Anti-corruption mechanisms: - Flat files contain full block checksums to quickly an easily detect database corruption without needing to do expensive merkle root calculations - Metadata checksums - Open reconciliation - Extensive test coverage: - Comprehensive blackbox interface testing - Whitebox testing which uses intimate knowledge to exercise uncommon failure paths such as deleting files out from under the database - Corruption tests (replacing random data in the files) In addition, this commit also contains a new tool under the new database directory named dbtool which provides a few basic commands for testing the database. It is designed around commands, so it could be useful to expand on in the future. Finally, this commit addresses the following issues: - Adds support for and therefore closes btcsuite#255 - Fixes btcsuite#199 - Fixes btcsuite#201 - Implements and closes btcsuite#256 - Obsoletes and closes btcsuite#257 - Closes btcsuite#247 once the required chain and btcd modifications are in place to make use of this new code
This commit adds a database cache layer to the ffldb database backend so that callers can commit multiple transactions without having to incur the overhead of a disk sync on every new block.
|
Reviewed 39 of 39 files at r7. Comments from the review on Reviewable.io |
3 tasks
Closed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This pull request contains a complete redesign and rewrite of the database package that approaches things in a vastly different manner than the previous version. This is the first part of several stages that will be needed to ultimately make use of this new package.
Some of the reasons for this were discussed in #255, however a quick summary is as follows:
In order to address all of these concerns, and others not mentioned, the database interface has been redesigned as follows:
This pull request also contains an implementation of the previously discussed interface named ffldb (flat file plus leveldb metadata backend). Here is a quick overview:
In addition, there is a new tool provided under the new database directory named
dbtoolwhich provides a few basic commands for testing the database. It is designed around commands, so it could be useful to expand on in the future.Finally, this commit addresses the following issues:
to make use of this new code
There are several things that need to happen before this PR can be merged and ultimately used: