-
Notifications
You must be signed in to change notification settings - Fork 6.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Log the amount of blob garbage generated by compactions in the MANIFEST #8450
Conversation
…h KV during compaction if set
…ces blob files, chain iterators correctly
@ltamasi has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM.
@@ -1136,6 +1147,15 @@ void CompactionJob::ProcessKeyValueCompaction(SubcompactionState* sub_compact) { | |||
input = clip.get(); | |||
} | |||
|
|||
std::unique_ptr<InternalIterator> blob_counter; | |||
|
|||
if (sub_compact->compaction->DoesInputReferenceBlobFiles()) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
cool that we check it first to avoid unnecessary counting.👍
…nge" Summary: We ended up using a different approach for tracking the amount of garbage in blob files (see e.g. facebook#8450), so the ability to apply only a range of table property collectors is now unnecessary. The patch reverts this part of facebook#8298 while keeping the cleanup. Test Plan: `make check`
…ange (#8465) Summary: We ended up using a different approach for tracking the amount of garbage in blob files (see e.g. #8450), so the ability to apply only a range of table property collectors is now unnecessary. The patch reverts this part of #8298 while keeping the cleanup done in that PR. Pull Request resolved: #8465 Test Plan: `make check` Reviewed By: jay-zhuang Differential Revision: D29399921 Pulled By: ltamasi fbshipit-source-id: af64816c357d0829b9d7ba8ca1477038138f6f0a
Summary:
The patch builds on
BlobGarbageMeter
andBlobCountingIterator
(introduced in #8426 and
#8443 respectively)
and ties it all together. It measures the amount of garbage
generated by a compaction and logs the corresponding
BlobFileGarbage
records as part of the compaction job's
VersionEdit
. Note: in orderto have accurate results,
kRemoveAndSkipUntil
for compaction filtersis implemented using iteration.
Test Plan:
Ran
make check
and the crash test script.