Introduce BlobFileCache and add support for blob files to Get() #7540
Conversation
ltamasi
changed the title
There was a problem hiding this comment.
@ltamasi has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.
There was a problem hiding this comment.
@ltamasi has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.
There was a problem hiding this comment.
Thanks @ltamasi for the PR. Mostly LGTM with a few minor comments and questions.
|
@ltamasi has updated the pull request. You must reimport the pull request before landing. |
…d now return Corruption
…h tier we're reading from
…ile is marked obsolete
… for the same file
|
@ltamasi has updated the pull request. You must reimport the pull request before landing. |
There was a problem hiding this comment.
@ltamasi has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.
|
@ltamasi has updated the pull request. You must reimport the pull request before landing. |
There was a problem hiding this comment.
@ltamasi has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.
…book#7540) Summary: The patch adds blob file support to the `Get` API by extending `Version` so that whenever a blob reference is read from a file, the blob is retrieved from the corresponding blob file and passed back to the caller. (This is assuming the blob reference is valid and the blob file is actually part of the given `Version`.) It also introduces a cache of `BlobFileReader`s called `BlobFileCache` that enables sharing `BlobFileReader`s between callers. `BlobFileCache` uses the same backing cache as `TableCache`, so `max_open_files` (if specified) limits the total number of open (table + blob) files. TODO: proactively open/cache blob files and pin the cache handles of the readers in the metadata objects similarly to what `VersionBuilder::LoadTableHandlers` does for table files. Pull Request resolved: facebook#7540 Test Plan: `make check` Reviewed By: riversand963 Differential Revision: D24260219 Pulled By: ltamasi fbshipit-source-id: a8a2a4f11d3d04d6082201b52184bc4d7b0857ba
Summary:
The patch adds blob file support to the
GetAPI by extendingVersionso thatwhenever a blob reference is read from a file, the blob is retrieved from the corresponding
blob file and passed back to the caller. (This is assuming the blob reference is valid
and the blob file is actually part of the given
Version.) It also introduces a cacheof
BlobFileReaders calledBlobFileCachethat enables sharingBlobFileReadersbetween callers.
BlobFileCacheuses the same backing cache asTableCache, somax_open_files(if specified) limits the total number of open (table + blob) files.TODO: proactively open/cache blob files and pin the cache handles of the readers in the
metadata objects similarly to what
VersionBuilder::LoadTableHandlersdoes fortable files.
Test Plan:
make check