Join GitHub today
GitHub is home to over 31 million developers working together to host and review code, manage projects, and build software together.Sign up
Allow to optional specify the directory for the blocks storage #12653
Since the actual block files taking up more and more space, it may be desirable to have them stored in a different location then the data directory (use case: SSD for chainstate, etc., HD for blocks).
This PR adds a
I fist had an option to keep the blockindex within the datadir, but seems to make no sense since accessing the index will (always) lead to access (r/w) the block files.
randolf left a comment
As the size of the blockchain continues to increase, this will facilitate the need to store the blocks on a different hard drive, hence this feature serves a probable future need. There are likely other uses for this feature too.
The argument for keeping the block index in the data dir would be that in the vast majority of cases it means the block index will be on the same filesystem as the utxo database, and therefore we can rely on regular filesystem semantics regarding data ordering. If the block index and utxo database are on different filesystems then it's not possible to guarantee that one is ahead of the other without fully serializing the operations with fsync.
I will try to look through the code more and think about this more closely. I think this change is fine the way the code currently works because the flushes are basically serialized anyway (and we fsync for block storage and block index updates), but this might become annoying long term.
I might have to retract my earlier statement, as @sipa says here that the blocks index and block files should be on the same filesystem, contradicting my earlier statement. I'm not sure if I understand that reasoning though, as we fsync after writing the block and before committing to the block index, which should be fine across filesystems. If it's really true that they need to be on the same filesystem that would be problematic in the future if we ever wanted to merge the block index and the chainstate index.
referenced this pull request
Mar 16, 2018
@sipa: can you elaborate your concerns of having the index outside of the blocks directory?
Mar 27, 2018
1 check passed
added a commit
this pull request
Mar 27, 2018
I ran a little test just to see the effect of putting the entire datadir an external USB 3.0 HDD versus the internal SSD (cross ref #10736):
Timings are gaps between block validations. There is a ~30x slow down when using the USB 3.0 HDD.
I also experimented with different combinations of the following settings:
I'm surprised that signature verification speed increase is negligible between a 2013 Core i5 and a 2017 Core i5 [and that par doesn't seem to effect results significantly, though that's off topic.]