Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add file block storage #703

Merged
merged 31 commits into from Apr 12, 2019
Merged

Add file block storage #703

merged 31 commits into from Apr 12, 2019

Conversation

@braydonf
Copy link
Contributor

braydonf commented Feb 20, 2019

This adds a new module blockstore for storing blocks. It adds a file block store that has several advantages:

  • Reduction of disk i/o when syncing the chain, this is due to reduction of database compaction (#585).
  • Transaction data can be queried directly from block data, the txindex can point to this data instead of duplicating the tx data (#585).
  • Ability to recovery from database corruption, as indexes are regenerated in the case that there are issues (#682).
  • Bootstap a node from block data from other implementations, such as Bitcoin Core.

There can be multiple implementations of the blockstore interface, to give better flexibility to how data is stored. Currently this includes FileBlockStore and LevelBlockStore, this could later be expanded to other methods.

@codecov-io

This comment was marked as outdated.

Copy link

codecov-io commented Feb 20, 2019

Codecov Report

Merging #703 into master will increase coverage by 0.68%.
The diff coverage is 96.53%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #703      +/-   ##
==========================================
+ Coverage   55.92%   56.61%   +0.68%     
==========================================
  Files         105      112       +7     
  Lines       27754    28140     +386     
  Branches     4751     4800      +49     
==========================================
+ Hits        15521    15931     +410     
+ Misses      12233    12209      -24
Impacted Files Coverage Δ
lib/blockchain/layout.js 100% <ø> (ø) ⬆️
lib/blockstore/records.js 100% <100%> (ø)
lib/node/fullnode.js 57.83% <100%> (+1.04%) ⬆️
lib/blockstore/level.js 100% <100%> (ø)
lib/blockchain/chain.js 60.88% <100%> (+0.18%) ⬆️
lib/blockstore/layout.js 100% <100%> (ø)
lib/blockstore/common.js 100% <100%> (ø)
lib/blockstore/abstract.js 100% <100%> (ø)
lib/node/node.js 65.11% <33.33%> (-0.57%) ⬇️
lib/blockchain/chaindb.js 57.21% <68%> (+0.4%) ⬆️
... and 10 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f57bd51...5438357. Read the comment docs.

@braydonf braydonf force-pushed the braydonf:blockstore branch 2 times, most recently from e101f32 to c459753 Feb 20, 2019
@braydonf braydonf force-pushed the braydonf:blockstore branch 3 times, most recently from 94b8a2b to 1fa7ece Feb 20, 2019
@braydonf

This comment has been minimized.

Copy link
Contributor Author

braydonf commented Feb 25, 2019

Here are some of the initial results with the benchmarks:

FileBlockStore SSD

real	0m6.444s
user	0m4.610s
sys	0m3.884s

LevelBlockStore SSD

real	1m18.667s
user	1m0.571s
sys	0m12.097s

FileBlockStore HDD

real	0m22.239s
user	0m6.652s
sys	0m6.436s

LevelBlockStore HDD

real	3m5.204s
user	1m11.664s
sys	0m11.664s

For the complete breakdown of the performance of each operation see: https://gist.github.com/braydonf/952ace7bcd7bbf529516e16a1d11e67e

@braydonf

This comment has been minimized.

Copy link
Contributor Author

braydonf commented Feb 28, 2019

bcoin-blockstore-bench-write
bcoin-blockstore-bench-read
bcoin-blockstore-bench-randomread
bcoin-blockstore-bench-prune

Charts were generated via this branch: https://github.com/braydonf/bcoin/tree/blockstore-charts and this commit braydonf@5f1eec7

@braydonf braydonf force-pushed the braydonf:blockstore branch from 69bfb56 to 8f4bbac Feb 28, 2019
@tuxcanfly tuxcanfly force-pushed the braydonf:blockstore branch from 8f4bbac to 85bbd88 Mar 1, 2019
@braydonf braydonf force-pushed the braydonf:blockstore branch from 85bbd88 to 8f4bbac Mar 1, 2019
@braydonf

This comment has been minimized.

Copy link
Contributor Author

braydonf commented Mar 5, 2019

This PR does not yet integrate the block store to be used with the chain and node yet, @tuxcanfly has a work-in-progress branch at tuxcanfly@7d53f63 integrating the file block store with chain. And it's showing significant improvements to the initial sync time. From my testing it shaved around 12 hours from the sync for mainnet.

lib/blockstore/level.js Outdated Show resolved Hide resolved
@braydonf braydonf force-pushed the braydonf:blockstore branch 2 times, most recently from 1612c66 to f219e56 Mar 7, 2019
@braydonf

This comment has been minimized.

Copy link
Contributor Author

braydonf commented Mar 7, 2019

Okay, this PR now integrates file blockstore with chain and the fullnode, and also rebased on latest master.

@braydonf

This comment has been minimized.

Copy link
Contributor Author

braydonf commented Mar 9, 2019

Here are the results from logging iotop for disk i/o during the IBD and chain sync for mainnet.


fileblockstore-sync-a
This was from the current PR branch at braydonf@823f483 with both the blocks and undo blocks stored with FileBlockStore.


fileblockstore-sync-b
This was from the previously mentioned branch at tuxcanfly@7d53f63 and did not yet include undo blocks in FileBlockStore.


leveldb-sync
This was from the latest master branch at: ea1c6be and stores blocks and undo blocks in LevelDB.

@braydonf

This comment has been minimized.

Copy link
Contributor Author

braydonf commented Mar 13, 2019

Tested with pruned and SPV nodes, and is working well.

Copy link
Member

tuxcanfly left a comment

As benchmarks above show, there's a marked improvement in performance for writing files which boosts the initial sync time. LGTM with some minor updates 👍

test/blockstore-test.js Outdated Show resolved Hide resolved
test/blockstore-test.js Show resolved Hide resolved
test/blockstore-test.js Show resolved Hide resolved
lib/blockstore/file.js Show resolved Hide resolved
lib/blockstore/file.js Outdated Show resolved Hide resolved
test/blockstore-test.js Show resolved Hide resolved
test/blockstore-test.js Show resolved Hide resolved
lib/blockstore/file.js Outdated Show resolved Hide resolved
test/blockstore-test.js Show resolved Hide resolved
lib/blockstore/file.js Show resolved Hide resolved
@pinheadmz

This comment has been minimized.

Copy link
Member

pinheadmz commented Mar 22, 2019

I wonder if we should have a quick check to make sure all files are present. I renamed a blk file and got an error in the log, but bcoin continued to run:

[error] (node) ENOENT: no such file or directory, open '/Users/matthewzipkin/.bcoin/testnet/blocks/blk00001.dat'

... though it couldn't add any new blocks to the chain (the file I mangled was the tip).

Downstream in the logs some weird messages:
[warning] (net) Received unrequested block: 0000000000adb71f55f8eb49cd99c899ce6ec294ab2a07ae608c1917e422c8f3 (95.216.35.252:18333).

I guess losing a file screwed up network sync also?

Note Bitcoin Core runs this check at startup:

2019-03-22T21:25:43Z Checking all blk files are present...

@pinheadmz

This comment has been minimized.

Copy link
Member

pinheadmz commented Mar 23, 2019

Should we include bcoin.define('Blockstore', './blockstore'); in bcoin.js?
The README has a little guide on creating a blockstore but it's not available from the bcoin object.

@braydonf

This comment has been minimized.

Copy link
Contributor Author

braydonf commented Mar 23, 2019

There is a scan at the startup for all blk* and blu* files and that there are corresponding index entries for those files, if there is not, as in the case that the ./blocks/index has been removed or has become corrupted, it will regenerate the index from the files.

@braydonf

This comment has been minimized.

Copy link
Contributor Author

braydonf commented Mar 25, 2019

For clarity there are several follow-up PRs after this one to:

  • De-duplicate the transaction for the txindex and other indexing fixes for addrindex, and separating the indexes from chain as in #424 (updates on this are currently in progress).
  • Generate the chain data in consideration that blocks may already exist, for the case of importing from another local node, and other cases where regenerating the chain is necessary.
braydonf and others added 13 commits Mar 14, 2019
There is potential for around a 10% to 23% increase to the performance
of block reads by using `allocUnsafe`, however there is already around
a 3 to 6 times increase to the performance. While it's safe to use
`allocUnsafe` as the number of bytes read is checked to be the same
size as the buffer allocation, there is a potential for test cases
to introduce _other_ behavior for `fs.read` that may not have the
same behavior, though this isn't currently the case.
@braydonf braydonf force-pushed the braydonf:blockstore branch from def91a1 to 50fe51c Apr 8, 2019
@braydonf

This comment has been minimized.

Copy link
Contributor Author

braydonf commented Apr 8, 2019

Okay, I think this is ready to land.

bench/blockstore.js Show resolved Hide resolved
- Multiple parallel runs of the same test will not conflict
  as a unique identifier is added to the test directory.
- The base test directory can be configured for various
  environments, and can be changed via the environment
  variable `TEMP`, see the implementation of `os.tmpdir()`
  for specific details.
Copy link
Member

tynes left a comment

Good job with the extensive test cases. I like test/blockstore-test.js.
Overall mostly nits, but this looks good to me. I have synced it a few times without a problem using Node.js v10.

test/blockstore-test.js Show resolved Hide resolved
@braydonf braydonf merged commit 5438357 into bcoin-org:master Apr 12, 2019
3 checks passed
3 checks passed
ci/circleci: install Your tests passed on CircleCI!
Details
ci/circleci: lint Your tests passed on CircleCI!
Details
ci/circleci: test Your tests passed on CircleCI!
Details
braydonf added a commit that referenced this pull request Apr 12, 2019
Add file block storage
@braydonf braydonf deleted the braydonf:blockstore branch Apr 12, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
7 participants
You can’t perform that action at this time.