Skip to content

Commit

Permalink
Detect EBB hash truncation in the index files of the ImmutableDB
Browse files Browse the repository at this point in the history
On very rare occasions, the quickcheck-state-machine tests for the ImmutableDB
failed, e.g., https://hydra.iohk.io/build/790254/nixlog/11

The cause of this is the following. We write the hash of the EBB at the end of
the index file, after the offsets. When there is no EBB in the epoch, we write
nothing extra after the offsets.

In the tests we simulate corruptions, e.g., by truncating some bytes from the
end of some file. In some rare cases, we were simulating a truncation of the
index file such that exactly the EBB hash was removed from the end of the
file. When reading the index file again, we saw an empty bytestring at the end
of the file, which corresponds to the case where there is no EBB, so we didn't
notice that the EBB got truncated.

The solution: instead of checking whether the bytestring left-over after
parsing the offsets is non-empty to determine whether there was an EBB or not,
we look at the second offset (which indicates the size of the EBB) in the
index, which will be non-zero if there was an EBB.

While working in this module, fix a confusing comment and a TODO.
  • Loading branch information
mrBliss committed Apr 24, 2019
1 parent 540e98c commit df535de
Showing 1 changed file with 8 additions and 3 deletions.
11 changes: 8 additions & 3 deletions ouroboros-consensus/src/Ouroboros/Storage/ImmutableDB/Index.hs
Expand Up @@ -43,6 +43,7 @@ import qualified Data.ByteString.Builder as BS
import qualified Data.ByteString.Lazy as BL
import Data.List.NonEmpty (NonEmpty)
import qualified Data.List.NonEmpty as NE
import Data.Maybe (isJust)
import qualified Data.Vector.Unboxed as V
import Data.Word (Word64)

Expand Down Expand Up @@ -114,9 +115,13 @@ loadIndex hashDecoder hasFS err epoch indexSize = do
let offsetsBS = BL.toStrict offsetsBL
offsets = V.generate expectedOffsets mkEntry
mkEntry ix = decodeIndexEntryAt (ix * indexEntrySizeBytes) offsetsBS
-- If the second offset is non-zero, there is an EBB
hasEBB = V.length offsets >= 2 && offsets V.! 1 /= 0
case deserialiseHash hashDecoder ebbHashBL of
-- TODO throw an error when leftover is not empty?
Right (_leftover, ebbHash) -> return $ MkIndex offsets ebbHash
Right (leftover, ebbHash) -> do
when (hasEBB /= isJust ebbHash || not (BL.null leftover)) $
throwUnexpectedError err $ InvalidFileError indexFile callStack
return $ MkIndex offsets ebbHash
Left df -> throwUnexpectedError err $
DeserialisationError df callStack

Expand Down Expand Up @@ -354,7 +359,7 @@ truncateToSlots slots index@(MkIndex offsets ebbHash)
{-------------------------------------------------------------------------------
Auxiliary: encoding and decoding the EBB hash
When no EBB is present we use an all-NULL bytestring.
When no EBB is present we use an empty bytestring.
-------------------------------------------------------------------------------}

deserialiseHash :: (forall s. Decoder s hash)
Expand Down

0 comments on commit df535de

Please sign in to comment.