fix possible block db breakage during re-index #5864
Conversation
|
Seems very reasonable to me. Even if this doesn't fix anything, it should be safe. |
|
I ran into this issue today. Power failured during IBD. Restarted with -reindex=1 Waited for IBD to finish and then called getblock RPC call for all blocks and got about a dozen ReadBlockFromDisk |
| vinfoBlockFile[nFile].AddBlock(nHeight, nTime); | ||
| if (fKnown && vinfoBlockFile[nFile].nSize < (pos.nPos + nAddSize)) | ||
| vinfoBlockFile[nFile].nSize = pos.nPos + nAddSize; |
There was a problem hiding this comment.
Looks good to me, although I'm not sure about the fKnown && vinfoBlockFile[nFile].nSize >= (pos.nPos + nAddSize) case. Would we want to increase nSize at all? Or is this better:
if (fKnown)
vinfoBlockFile[nFile].nSize = std::max(pos.nPos + nAddSize, vinfoBlockFile[nFile].nSize);
else
vinfoBlockFile[nFile].nSize += nAddSize;
There was a problem hiding this comment.
When can I fetch latest source code, compile and test?
On Mon, Mar 9, 2015 at 1:01 PM, Jeff Garzik notifications@github.com
wrote:
In src/main.cpp
#5864 (comment):vinfoBlockFile[nFile].AddBlock(nHeight, nTime);
- if (fKnown && vinfoBlockFile[nFile].nSize < (pos.nPos + nAddSize))
vinfoBlockFile[nFile].nSize = pos.nPos + nAddSize;+1
—
Reply to this email directly or view it on GitHub
https://github.com/bitcoin/bitcoin/pull/5864/files#r26029884.
Mvh
-fredrik-normann-
Sent from my Gmail Account
There was a problem hiding this comment.
@laanwj Yes, thanks for catching that.
|
I tested this in the this little RPC test, https://gist.github.com/morcos/ae506817284cd776d5b2. |
theuni
force-pushed
the
fix-reindex-corruption
branch
from
9add379
to
8eb826f
Compare
theuni
force-pushed
the
fix-reindex-corruption
branch
from
8eb826f
to
bb6acff
Compare
When re-indexing, there are a few cases where garbage data may be skipped in the block files. In these cases, the indices are correctly written to the index db, however the pointer to the next position for writing in the current block file is calculated by adding the sizes of the valid blocks found. As a result, when the re-index is finished, the index db is correct for all existing blocks, but the next block will be written to an incorrect offset, likely overwriting existing blocks. Rather than using the sum of all valid blocks to determine the next write position, use the end of the last block written to the file. Don't assume that the current block is the last one in the file, since they may be read out-of-order.
bb6acff fix possible block db breakage during re-index (Cory Fields)
When re-indexing, there are a few cases where garbage data may be skipped in the block files. In these cases, the indices are correctly written to the index db, however the pointer to the next position for writing in the current block file is calculated by adding the sizes of the valid blocks found. As a result, when the re-index is finished, the index db is correct for all existing blocks, but the next block will be written to an incorrect offset, likely overwriting existing blocks. Rather than using the sum of all valid blocks to determine the next write position, use the end of the last block written to the file. Don't assume that the current block is the last one in the file, since they may be read out-of-order. Rebased-From: bb6acff Github-Pull: #5864
|
Cherry-picked to 0.10 as 002c8a2 |
When re-indexing, there are a few cases where garbage data may be skipped in the block files. In these cases, the indices are correctly written to the index db, however the pointer to the next position for writing in the current block file is calculated by adding the sizes of the valid blocks found. As a result, when the re-index is finished, the index db is correct for all existing blocks, but the next block will be written to an incorrect offset, likely overwriting existing blocks. Rather than using the sum of all valid blocks to determine the next write position, use the end of the last block written to the file. Don't assume that the current block is the last one in the file, since they may be read out-of-order. Rebased-From: bb6acff Github-Pull: bitcoin#5864 (cherry picked from commit 002c8a2)
I could really use some more eyes on this. I discussed it briefly with @sipa on IRC a few days ago.
I noticed this while looking into #5668 . This is one possible explanation I can come up with for overlapping block data. Whether it has anything to do with that issue or not, I think this still needs to be addressed.
When re-indexing, there are a few cases where garbage data may be skipped in the block files. In these cases, the indices are correctly written to the index db, however the pointer to the next position for writing in the current block file is calculated by adding the sizes of the valid blocks found.
As a result, when the re-index is finished, the index db is correct for all existing blocks, but the next block will be written to an incorrect offset, likely overwriting existing blocks.
Rather than using the sum of all valid blocks to determine the next write position, use the end of the last block written to the file. Don't assume that the current block is the last one in the file, since they may be read out-of-order.
I was able to trigger this problem by inserting some garbage data between two valid blocks on disk in the last .dat file, then reindexing. After that, run normally for a few min in order to write a few new blocks to disk, then run with -checkblocks=0.
Before this change, I would get different errors (de-serialization, eof, etc) depending on what garbage i add and where. After the change, it appears to survive the re-index without issue regardless of the garbage.