Blob storage pr #2269
Closed
Blob storage pr #2269
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Summary: 1. Options Class. 2. Impl class BlobDBImpl 3. made BlobDB::Open Test Plan: Reviewers: Subscribers: Tasks: Blame Revision:
Summary: 1. Header for Blob DB file. version/magic #/ttl etc 2. some more options 3. move blob db under a configurable directory Test Plan: Reviewers: Subscribers: Tasks: Blame Revision:
Summary: to get better control over the header and the blob log, replicating the code from WAL log reader/writer. Will have to figure out later, if this functionality can be merged Test Plan: Reviewers: Subscribers: Tasks: Blame Revision:
Summary: Optimized file formats for blobs with proper headers and footers. BlobDB::Open with support for reading the footers at open time and building the in-memory metadata. Test Plan: Rebased the blob db test on top of the changes. Reviewers: Subscribers: Tasks: Blame Revision:
Summary: 1. If files were still open on shutdown, reuse them by opening in append mode. 2. slightly more robust file selection heuristic (Still naive). Test Plan: Reviewers: Subscribers: Tasks: Blame Revision:
Summary: 1. Options Class. 2. Impl class BlobDBImpl 3. made BlobDB::Open Test Plan: Reviewers: Subscribers: Tasks: Blame Revision:
Summary: 1. Header for Blob DB file. version/magic #/ttl etc 2. some more options 3. move blob db under a configurable directory Test Plan: Reviewers: Subscribers: Tasks: Blame Revision:
Summary: to get better control over the header and the blob log, replicating the code from WAL log reader/writer. Will have to figure out later, if this functionality can be merged Test Plan: Reviewers: Subscribers: Tasks: Blame Revision:
Summary: Optimized file formats for blobs with proper headers and footers. BlobDB::Open with support for reading the footers at open time and building the in-memory metadata. Test Plan: Rebased the blob db test on top of the changes. Reviewers: Subscribers: Tasks: Blame Revision:
Summary: 1. If files were still open on shutdown, reuse them by opening in append mode. 2. slightly more robust file selection heuristic (Still naive). Test Plan: Reviewers: Subscribers: Tasks: Blame Revision:
Summary: 1. column family support 2. proper sequence numbers 3. GC threads 4. Fsync In different schedule Test Plan: Reviewers:
Summary: Reindenting parts of file. Reorganizing code. Test Plan:
Summary: Test Plan: Reviewers: Subscribers: Tasks: Blame Revision:
Summary: Test Plan: Reviewers: Subscribers: Tasks: Blame Revision:
reads than writes Summary: Test Plan: Reviewers: Subscribers: Tasks: Blame Revision:
Summary: Test Plan: Reviewers: Subscribers: Tasks: Blame Revision:
Summary: 1. Read Write Locks. 2. Reduce critical section during writes 3. valgrind runs and fixes Test Plan: Reviewers: Subscribers: Tasks: Blame Revision:
Summary: 1. unit test with lot of random blobs 2. stackable db should not take ownership of base db, because we share with blob db. 3. fixing ttl_guess logic for now, to make sure search doesn't miss files. Test Plan: Reviewers: Subscribers: Tasks: Blame Revision:
Summary: Test Plan: Reviewers: Subscribers: Tasks: Blame Revision:
Summary: Test Plan: Reviewers: Subscribers: Tasks: Blame Revision:
Summary: 1. sequential file buffer management 2. rewind functionality 3. some logs for error cases. 4. making the time range of a blob log pick up correct option Test Plan: Reviewers: Subscribers: Tasks: Blame Revision:
Summary: Test Plan: Reviewers: Subscribers: Tasks: Blame Revision:
Summary: Test Plan: Reviewers: Subscribers: Tasks: Blame Revision:
Summary: 1. align the ttl start and end to multiples of ttl_range this keeps the ttl range of files tractable and non-overlapping 2. remove an option from options file Test Plan: Reviewers: Subscribers: Tasks: Blame Revision:
Summary: Also using make format to reformat changes Test Plan: Reviewers: Subscribers: Tasks: Blame Revision:
Summary: Test Plan: Reviewers: Subscribers: Tasks: Blame Revision:
Summary: Test Plan: Reviewers: Subscribers: Tasks: Blame Revision:
Summary: This callback ensures that before a flush happens blob log files are also fsync'd Test Plan: Reviewers: Subscribers: Tasks: Blame Revision:
Summary: Test Plan: Reviewers: Subscribers: Tasks: Blame Revision:
…DB::Open Summary: Test Plan: Reviewers: Subscribers: Tasks: Blame Revision:
Summary: Test Plan: Reviewers: Subscribers: Tasks: Blame Revision:
…ckable DB work easier Summary: Test Plan: Reviewers: Subscribers: Tasks: Blame Revision:
Summary: Test Plan: Reviewers: Subscribers: Tasks: Blame Revision:
Summary: 1. open files are now a multiset. 2. fixing minor race condition between async closeSeqFiles and Put 3. more diagnostics and asserts 4. removing lambda's for debug simplicity 5. providig a functor for TTL extraction Test Plan: db_bench Reviewers: Subscribers: Tasks: Blame Revision:
Summary: Test Plan: Reviewers: Subscribers: Tasks: Blame Revision:
Summary: Added a callback which informs the keys that are being compacted. No Filtering. The EvictAllVersions compaction filter, registers the packet into a lockless queue and marks the blogs evicted Test Plan: OverrideTest added to blob_db_test Reviewers: Subscribers: Tasks: Blame Revision:
Summary: Review comments from Siying, Murali, Yuslepukhin and Lichuang 1. Relocating files from db area to utilities area 2. Replacing DecodeFixed32/64 and offset calculation with GetFixed32/64 3. removing invalid comments from blob_log_writer.h 4. changing comment style 5. doing make_format again 6. changing the enum style 7. directory fsync on file add and delete 8. putting magic number as last field in footer 9. Making Header Construct use PutFixed32 instead of Encode... 10. Added CRC checks on Get 11. Removing extra option in options.h for flush_begin_listener 12. removing raw pointers from blob log format 13. Removing has_ttl and has_ts and replacing with single unique_ptr 14. Appveyor and Travis failures 15. Better APIs for Column Families. Simplifying interfaces. 16. Compression of Blob Support 17. Splitting into multiple files, to reduce file size 18. GC transaction 1 at a time, instead of all at once 19. Proper bookkeeping of deletes after open of db(crash safety) Test Plan: Reviewers: Subscribers: Tasks: Blame Revision:
|
@yiwu-arbug has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator. |
siying
reviewed
May 10, 2017
There was a problem hiding this comment.
Windows build is not passing and we don't know OSX because it is broken by another PR. Feel free to check-in as long as @yiwu-arbug approves it.
| @@ -0,0 +1,240 @@ | |||
| // Copyright (c) 2011-present, Facebook, Inc. All rights reserved. | |||
There was a problem hiding this comment.
If I remember correctly, cc_options.cc should have already been moved to options/.
|
Didn't notice the Windows failures.. if there's any failure remaining I'll fix in another patch. |
yiwu-arbug
pushed a commit
that referenced
this pull request
May 11, 2017
Summary: The final pull request for Blob Storage. Closes #2269 Differential Revision: D5033189 Pulled By: yiwu-arbug fbshipit-source-id: 6356b683ccd58cbf38a1dc55e2ea400feecd5d06
This was referenced May 15, 2017
Closed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The final pull request for Blob Storage.