-
Notifications
You must be signed in to change notification settings - Fork 9
Refactor blob refs #454
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Refactor blob refs #454
Changes from all commits
Commits
Show all changes
16 commits
Select commit
Hold shift + click to select a range
3e105b5
Introduce a new BlobFile abstraction
dcoutts feb1cf4
Convert WriteBufferBlobs to use BlobFile
dcoutts 3b801cc
Convert Run to use BlobFile
dcoutts 0034666
Refactor: change BlobRef m (Handle h) to BlobRef m h
dcoutts d49ccbc
Rename internal BlobRef type to RawBlobRef
dcoutts 65c3f04
Change RawBlobRef to contain a BlobFile
dcoutts fd61a17
Introduce proper distinction between {Raw,Weak,Strong}BlobRef
dcoutts c918bec
Remove BlobRef.readBlob and add BlobRef.readWeakBlobRef
dcoutts 54c2348
Move impl of retrieveBlobs to new BlobRef.readWeakBlobRefs
dcoutts 301357a
Remove now-unused definitions in BlobRef
dcoutts 8293486
Add a couple of TODOs into related code
dcoutts 1488d9e
Temporary hack until we have proper snapshots
dcoutts 85d2611
stylish-haskell export list formatting fixes
dcoutts 68caad0
Address review comments
dcoutts 0cf251d
Add BlobFile.writeBlob and use it in WriteBufferBlobs
dcoutts 1c2269e
Add SPECIALISE for the CRC32C module functions
dcoutts File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dcoutts marked this conversation as resolved.
Show resolved
Hide resolved
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,124 @@ | ||
| module Database.LSMTree.Internal.BlobFile ( | ||
| BlobFile (..) | ||
| , BlobSpan (..) | ||
| , removeReference | ||
| , RemoveFileOnClose (..) | ||
| , openBlobFile | ||
| , readBlob | ||
| , writeBlob | ||
| ) where | ||
|
|
||
| import Control.DeepSeq (NFData (..)) | ||
| import Control.Monad (unless) | ||
| import Control.Monad.Class.MonadThrow (MonadMask, MonadThrow) | ||
| import Control.Monad.Primitive (PrimMonad) | ||
| import Control.RefCount (RefCounter) | ||
| import qualified Control.RefCount as RC | ||
| import qualified Data.Primitive.ByteArray as P | ||
| import qualified Data.Vector.Primitive as VP | ||
| import Data.Word (Word32, Word64) | ||
| import qualified Database.LSMTree.Internal.RawBytes as RB | ||
| import Database.LSMTree.Internal.Serialise (SerialisedBlob (..)) | ||
| import qualified System.FS.API as FS | ||
| import System.FS.API (HasFS) | ||
| import qualified System.FS.BlockIO.API as FS | ||
|
|
||
| -- | A handle to a file containing blobs. | ||
| -- | ||
| -- This is a reference counted object. Upon finalisation, the file is closed | ||
| -- and deleted. | ||
| -- | ||
| data BlobFile m h = BlobFile { | ||
| blobFileHandle :: {-# UNPACK #-} !(FS.Handle h), | ||
| blobFileRefCounter :: {-# UNPACK #-} !(RefCounter m) | ||
| } | ||
| deriving stock (Show) | ||
|
|
||
| instance NFData h => NFData (BlobFile m h) where | ||
| rnf (BlobFile a b) = rnf a `seq` rnf b | ||
|
|
||
| -- | The location of a blob inside a blob file. | ||
| data BlobSpan = BlobSpan { | ||
| blobSpanOffset :: {-# UNPACK #-} !Word64 | ||
| , blobSpanSize :: {-# UNPACK #-} !Word32 | ||
| } | ||
| deriving stock (Show, Eq) | ||
|
|
||
| instance NFData BlobSpan where | ||
| rnf (BlobSpan a b) = rnf a `seq` rnf b | ||
|
|
||
| {-# INLINE removeReference #-} | ||
| removeReference :: | ||
| (MonadMask m, PrimMonad m) | ||
| => BlobFile m h | ||
| -> m () | ||
| removeReference BlobFile{blobFileRefCounter} = | ||
| RC.removeReference blobFileRefCounter | ||
|
|
||
| -- | TODO: this hack can be removed once snapshots are done properly and so | ||
| -- runs can delete their files on close. | ||
| data RemoveFileOnClose = RemoveFileOnClose | DoNotRemoveFileOnClose | ||
| deriving stock Eq | ||
|
|
||
| -- | Open the given file to make a 'BlobFile'. The finaliser will close and | ||
| -- delete the file. | ||
| -- | ||
| -- TODO: Temporarily we have a 'RemoveFileOnClose' flag, which can be removed | ||
| -- once 'Run' no longer needs it, when snapshots are implemented. | ||
| -- | ||
| {-# SPECIALISE openBlobFile :: HasFS IO h -> FS.FsPath -> FS.OpenMode -> RemoveFileOnClose -> IO (BlobFile IO h) #-} | ||
| openBlobFile :: | ||
| PrimMonad m | ||
| => HasFS m h | ||
| -> FS.FsPath | ||
| -> FS.OpenMode | ||
| -> RemoveFileOnClose | ||
| -> m (BlobFile m h) | ||
| openBlobFile fs path mode remove = do | ||
| blobFileHandle <- FS.hOpen fs path mode | ||
| let finaliser = do | ||
| FS.hClose fs blobFileHandle | ||
| unless (remove == DoNotRemoveFileOnClose) $ | ||
| FS.removeFile fs (FS.handlePath blobFileHandle) | ||
| blobFileRefCounter <- RC.mkRefCounter1 (Just finaliser) | ||
| return BlobFile { | ||
| blobFileHandle, | ||
| blobFileRefCounter | ||
| } | ||
|
|
||
| {-# SPECIALISE readBlob :: HasFS IO h -> BlobFile IO h -> BlobSpan -> IO SerialisedBlob #-} | ||
| readBlob :: | ||
| (MonadThrow m, PrimMonad m) | ||
| => HasFS m h | ||
| -> BlobFile m h | ||
| -> BlobSpan | ||
| -> m SerialisedBlob | ||
| readBlob fs BlobFile {blobFileHandle} | ||
| BlobSpan {blobSpanOffset, blobSpanSize} = do | ||
| let off = FS.AbsOffset blobSpanOffset | ||
| len :: Int | ||
| len = fromIntegral blobSpanSize | ||
| mba <- P.newPinnedByteArray len | ||
| _ <- FS.hGetBufExactlyAt fs blobFileHandle mba 0 | ||
| (fromIntegral len :: FS.ByteCount) off | ||
| ba <- P.unsafeFreezeByteArray mba | ||
| let !rb = RB.fromByteArray 0 len ba | ||
| return (SerialisedBlob rb) | ||
|
|
||
| {-# SPECIALISE writeBlob :: HasFS IO h -> BlobFile IO h -> SerialisedBlob -> Word64 -> IO () #-} | ||
| writeBlob :: | ||
| (MonadThrow m, PrimMonad m) | ||
| => HasFS m h | ||
| -> BlobFile m h | ||
| -> SerialisedBlob | ||
| -> Word64 | ||
| -> m () | ||
| writeBlob fs BlobFile {blobFileHandle} | ||
| (SerialisedBlob' (VP.Vector boff blen ba)) off = do | ||
| mba <- P.unsafeThawByteArray ba | ||
| _ <- FS.hPutBufExactlyAt | ||
| fs blobFileHandle mba | ||
| (FS.BufferOffset boff) | ||
| (fromIntegral blen :: FS.ByteCount) | ||
| (FS.AbsOffset off) | ||
| return () |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.