Add DB properties for BlobDB #8734
Conversation
|
@Zhiyi-Zhang has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator. |
|
@Zhiyi-Zhang has updated the pull request. You must reimport the pull request before landing. |
|
@Zhiyi-Zhang has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator. |
Zhiyi-Zhang
force-pushed
the
new_feature_blobDB_properties
branch
from
261e0c2
to
474d180
Compare
|
@Zhiyi-Zhang has updated the pull request. You must reimport the pull request before landing. |
|
@Zhiyi-Zhang has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator. |
|
@Zhiyi-Zhang has updated the pull request. You must reimport the pull request before landing. |
Zhiyi-Zhang
force-pushed
the
new_feature_blobDB_properties
branch
from
8c18850
to
dc22263
Compare
|
@Zhiyi-Zhang has updated the pull request. You must reimport the pull request before landing. |
Zhiyi-Zhang
force-pushed
the
new_feature_blobDB_properties
branch
from
dc22263
to
4d28ebc
Compare
|
@Zhiyi-Zhang has updated the pull request. You must reimport the pull request before landing. |
|
@Zhiyi-Zhang has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator. |
There was a problem hiding this comment.
Thanks so much for the PR @Zhiyi-Zhang ! It looks great in general 👍👍 Please see some comments below.
|
@Zhiyi-Zhang has updated the pull request. You must reimport the pull request before landing. |
There was a problem hiding this comment.
Thanks for the updates @Zhiyi-Zhang, this is awesome! I think the only missing piece is the extending the unit tests as we discussed. (Also, see a few minor comments below.)
Please let me know if you hit any blockers in that area. |
Zhiyi-Zhang
force-pushed
the
new_feature_blobDB_properties
branch
from
30a9d81
to
bfafa96
Compare
Zhiyi-Zhang
force-pushed
the
new_feature_blobDB_properties
branch
from
bfafa96
to
e7b7a96
Compare
|
@Zhiyi-Zhang has updated the pull request. You must reimport the pull request before landing. |
Zhiyi-Zhang
force-pushed
the
new_feature_blobDB_properties
branch
from
e7b7a96
to
fa2c9d1
Compare
|
@Zhiyi-Zhang has updated the pull request. You must reimport the pull request before landing. |
|
@Zhiyi-Zhang has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator. |
Zhiyi-Zhang
force-pushed
the
new_feature_blobDB_properties
branch
from
fa2c9d1
to
87c3db4
Compare
|
@Zhiyi-Zhang has updated the pull request. You must reimport the pull request before landing. |
|
@Zhiyi-Zhang has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator. |
1 similar comment
|
@Zhiyi-Zhang has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator. |
|
@Zhiyi-Zhang merged this pull request in 0cb0fc6. |
Summary: RocksDB exposes certain internal statistics via the DB property interface. However, there are currently no properties related to BlobDB. For starters, we would like to add the following BlobDB properties: `rocksdb.num-blob-files`: number of blob files in the current Version (kind of like `num-files-at-level` but note this is not per level, since blob files are not part of the LSM tree). `rocksdb.blob-stats`: this could return the total number and size of all blob files, and potentially also the total amount of garbage (in bytes) in the blob files in the current Version. `rocksdb.total-blob-file-size`: the total size of all blob files (as a blob counterpart for `total-sst-file-size`) of all Versions. `rocksdb.live-blob-file-size`: the total size of all blob files in the current Version. `rocksdb.estimate-live-data-size`: this is actually an existing property that we can extend so it considers blob files as well. When it comes to blobs, we actually have an exact value for live bytes. Namely, live bytes can be computed simply as total bytes minus garbage bytes, summed over the entire set of blob files in the Version. Pull Request resolved: facebook/rocksdb#8734 Test Plan: ``` ➜ rocksdb git:(new_feature_blobDB_properties) ./db_blob_basic_test [==========] Running 16 tests from 2 test cases. [----------] Global test environment set-up. [----------] 10 tests from DBBlobBasicTest [ RUN ] DBBlobBasicTest.GetBlob [ OK ] DBBlobBasicTest.GetBlob (12 ms) [ RUN ] DBBlobBasicTest.MultiGetBlobs [ OK ] DBBlobBasicTest.MultiGetBlobs (11 ms) [ RUN ] DBBlobBasicTest.GetBlob_CorruptIndex [ OK ] DBBlobBasicTest.GetBlob_CorruptIndex (10 ms) [ RUN ] DBBlobBasicTest.GetBlob_InlinedTTLIndex [ OK ] DBBlobBasicTest.GetBlob_InlinedTTLIndex (12 ms) [ RUN ] DBBlobBasicTest.GetBlob_IndexWithInvalidFileNumber [ OK ] DBBlobBasicTest.GetBlob_IndexWithInvalidFileNumber (9 ms) [ RUN ] DBBlobBasicTest.GenerateIOTracing [ OK ] DBBlobBasicTest.GenerateIOTracing (11 ms) [ RUN ] DBBlobBasicTest.BestEffortsRecovery_MissingNewestBlobFile [ OK ] DBBlobBasicTest.BestEffortsRecovery_MissingNewestBlobFile (13 ms) [ RUN ] DBBlobBasicTest.GetMergeBlobWithPut [ OK ] DBBlobBasicTest.GetMergeBlobWithPut (11 ms) [ RUN ] DBBlobBasicTest.MultiGetMergeBlobWithPut [ OK ] DBBlobBasicTest.MultiGetMergeBlobWithPut (14 ms) [ RUN ] DBBlobBasicTest.BlobDBProperties [ OK ] DBBlobBasicTest.BlobDBProperties (21 ms) [----------] 10 tests from DBBlobBasicTest (124 ms total) [----------] 6 tests from DBBlobBasicTest/DBBlobBasicIOErrorTest [ RUN ] DBBlobBasicTest/DBBlobBasicIOErrorTest.GetBlob_IOError/0 [ OK ] DBBlobBasicTest/DBBlobBasicIOErrorTest.GetBlob_IOError/0 (12 ms) [ RUN ] DBBlobBasicTest/DBBlobBasicIOErrorTest.GetBlob_IOError/1 [ OK ] DBBlobBasicTest/DBBlobBasicIOErrorTest.GetBlob_IOError/1 (10 ms) [ RUN ] DBBlobBasicTest/DBBlobBasicIOErrorTest.MultiGetBlobs_IOError/0 [ OK ] DBBlobBasicTest/DBBlobBasicIOErrorTest.MultiGetBlobs_IOError/0 (10 ms) [ RUN ] DBBlobBasicTest/DBBlobBasicIOErrorTest.MultiGetBlobs_IOError/1 [ OK ] DBBlobBasicTest/DBBlobBasicIOErrorTest.MultiGetBlobs_IOError/1 (10 ms) [ RUN ] DBBlobBasicTest/DBBlobBasicIOErrorTest.CompactionFilterReadBlob_IOError/0 [ OK ] DBBlobBasicTest/DBBlobBasicIOErrorTest.CompactionFilterReadBlob_IOError/0 (1011 ms) [ RUN ] DBBlobBasicTest/DBBlobBasicIOErrorTest.CompactionFilterReadBlob_IOError/1 [ OK ] DBBlobBasicTest/DBBlobBasicIOErrorTest.CompactionFilterReadBlob_IOError/1 (1013 ms) [----------] 6 tests from DBBlobBasicTest/DBBlobBasicIOErrorTest (2066 ms total) [----------] Global test environment tear-down [==========] 16 tests from 2 test cases ran. (2190 ms total) [ PASSED ] 16 tests. ``` Reviewed By: ltamasi Differential Revision: D30690849 Pulled By: Zhiyi-Zhang fbshipit-source-id: a7567319487ad76bd1a2e24bf143afdbbd9e4346
RocksDB exposes certain internal statistics via the DB property interface.
However, there are currently no properties related to BlobDB.
For starters, we would like to add the following BlobDB properties:
rocksdb.num-blob-files: number of blob files in the current Version (kind of likenum-files-at-levelbut note this is not per level, since blob files are not part of the LSM tree).rocksdb.blob-stats: this could return the total number and size of all blob files, and potentially also the total amount of garbage (in bytes) in the blob files in the current Version.rocksdb.total-blob-file-size: the total size of all blob files (as a blob counterpart fortotal-sst-file-size) of all Versions.rocksdb.live-blob-file-size: the total size of all blob files in the current Version.rocksdb.estimate-live-data-size: this is actually an existing property that we can extend so it considers blob files as well. When it comes to blobs, we actually have an exact value for live bytes. Namely, live bytes can be computed simply as total bytes minus garbage bytes, summed over the entire set of blob files in the Version.Test Plan: