Improve disk fragmentation of block files #141
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Great! This should help disk based drives a lot. |
Extend AppendBlockFile with a new "size" parameter that is used to check for disk space directly in the routine. It is also verified that the number of written bytes matches the size, but the size is not yet used for anything else. In the future, it will be used to preallocate the block file in chunks.
CheckDiskSpace is used also in init.cpp, so can't make it static.
Keep a number of "reserved" zero bytes at the end of the block files, and if they are not enough in AppendBlockFile, always extend the file size by a larger chunk. This may prevent excessive fragmentation in NTFS / Windows.
Instead of constructing a manual CTxDB object in AppendBlockFile, use a passed-in one. This seems to be less prone to dead-locks.
|
I ran a comparison: Full blockchain download with direct local connection after system defragmentation. Windows, NTFS, i3 with disk based HD. |
|
Thanks for the timing - this is very interesting to know! :) BTW, I think also the Bitcoin devs discussed a change like this some time ago. Snailbrain dug out a link, but I don't have it at hand right now. |
|
Has this been tested enough? I have been running it ever since and did not notice any problems. Domob, do you feel comfortable with merging? |
|
As far as I know, it has been running quite a while for Huntercoin without problems. The same for me. So if it also works for you on Windows and shows improvements, I think it should be fine to merge. |
phelixbtc
added a commit
that referenced
this pull request
Aug 11, 2014
Improve disk fragmentation of block files
domob1812
pushed a commit
to domob1812/namecoin
that referenced
this pull request
Dec 16, 2014
bccaf86 Merge pull request namecoin#150 2a53a47 Merge pull request namecoin#151 5f5a31f Merge pull request namecoin#149 3907277 Merge pull request namecoin#142 a3e0611 Enable tests in x86 travis builds 45da235 x86 builder 8bb0e93 Merge pull request namecoin#155 971fe81 build: fix openssl detection for cross builds f22d73e Explicitly access %0..%2 as 64-bit so we use the right registers for x32 ABI e66d4d6 Avoid the stack in assembly and use explicit registers cf7b2b4 Fix ECDSA message hashes to 32 bytes 056ad31 Really compile with -O3 by default 74ad63a Merge pull request namecoin#146 9000458 Merge pull request namecoin#145 1f46b00 build: fix __builtin_expect detection for clang aaba2e0 Merge pull request namecoin#136 8a0775c Merge pull request namecoin#144 ee1eaa7 Merge pull request namecoin#141 c88e2b8 Compile with -O3 by default 6558a26 Make the benchmarks print out stats 000bdf6 Rename bench_verify to bench_recovery 7c6fed2 Add a few more additional tests. 992e03b travis: add clang to the test matrix b43b79a Merge pull request namecoin#143 e06a924 Include time.h header for time(). 8d11164 Add some additional tests. 3545627 Merge pull request namecoin#118 6a9901e Merge pull request namecoin#137 376b28b Merge pull request namecoin#128 1728806 Merge pull request namecoin#138 a5759c5 Check return value of malloc 39bd94d Variable time normalize ad86bdf Merge pull request namecoin#140 54b768c Another redundant secp256k1_fe_normalize 69dcaab Merge pull request #139 1c29f2e Remove redundant secp256k1_fe_normalize from secp256k1_gej_add_ge_var. 2b9388b Remove unused secp256k1_fe_inv_all f461b76 Allocate precomputation arrays on the heap b2c9681 Make {mul,sqr}_inner use the same argument order as {mul,sqr} 6793505 Convert YASM code into inline assembly f048615 Rewrite field assembly to match the C version 3ce74b1 Tweak precomputed table size for G git-subtree-dir: src/secp256k1 git-subtree-split: bccaf86caa9c44166e5a66600b742c516e03c3f0
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is a port of chronokings/huntercoin#66. The patch changes adding to the block files (blk????.dat) to be done in chunks of 16 MiB, which decreases file system fragmentation and is (I have heard) a big improvement especially on NTFS systems.
It has been tested already a while in Huntercoin, and I'm running a sync from scratch with this patch applied right now. It correctly allocates the file in chunks, and is already at block 20k. So it seems to work - but it would be cool to get more testing results, especially on Windows with NTFS.