Blob storage pr #2269
Blob storage pr #2269
Commits on May 9, 2017
-
Adding an options class for BlobDB and also Impl class.
Summary: 1. Options Class. 2. Impl class BlobDBImpl 3. made BlobDB::Open Test Plan: Reviewers: Subscribers: Tasks: Blame Revision:
-
Header for blob db file and move blobs under a directory
Summary: 1. Header for Blob DB file. version/magic #/ttl etc 2. some more options 3. move blob db under a configurable directory Test Plan: Reviewers: Subscribers: Tasks: Blame Revision:
-
Creating a Blob Log reader writer from WAL reader/writer
Summary: to get better control over the header and the blob log, replicating the code from WAL log reader/writer. Will have to figure out later, if this functionality can be merged Test Plan: Reviewers: Subscribers: Tasks: Blame Revision:
-
support for PutWithTTL and PutUntil.
Summary: Optimized file formats for blobs with proper headers and footers. BlobDB::Open with support for reading the footers at open time and building the in-memory metadata. Test Plan: Rebased the blob db test on top of the changes. Reviewers: Subscribers: Tasks: Blame Revision:
-
improved file selection for TTL and reuse files after Open
Summary: 1. If files were still open on shutdown, reuse them by opening in append mode. 2. slightly more robust file selection heuristic (Still naive). Test Plan: Reviewers: Subscribers: Tasks: Blame Revision:
-
Adding an options class for BlobDB and also Impl class.
Summary: 1. Options Class. 2. Impl class BlobDBImpl 3. made BlobDB::Open Test Plan: Reviewers: Subscribers: Tasks: Blame Revision:
-
Header for blob db file and move blobs under a directory
Summary: 1. Header for Blob DB file. version/magic #/ttl etc 2. some more options 3. move blob db under a configurable directory Test Plan: Reviewers: Subscribers: Tasks: Blame Revision:
-
Creating a Blob Log reader writer from WAL reader/writer
Summary: to get better control over the header and the blob log, replicating the code from WAL log reader/writer. Will have to figure out later, if this functionality can be merged Test Plan: Reviewers: Subscribers: Tasks: Blame Revision:
-
support for PutWithTTL and PutUntil.
Summary: Optimized file formats for blobs with proper headers and footers. BlobDB::Open with support for reading the footers at open time and building the in-memory metadata. Test Plan: Rebased the blob db test on top of the changes. Reviewers: Subscribers: Tasks: Blame Revision:
-
improved file selection for TTL and reuse files after Open
Summary: 1. If files were still open on shutdown, reuse them by opening in append mode. 2. slightly more robust file selection heuristic (Still naive). Test Plan: Reviewers: Subscribers: Tasks: Blame Revision:
-
Summary: 1. column family support 2. proper sequence numbers 3. GC threads 4. Fsync In different schedule Test Plan: Reviewers:
-
Summary: Reindenting parts of file. Reorganizing code. Test Plan:
-
Adding a mutex to BlobFile. Better critical sections.
Summary: Test Plan: Reviewers: Subscribers: Tasks: Blame Revision:
-
Summary: Test Plan: Reviewers: Subscribers: Tasks: Blame Revision:
-
Moving to Read Write Locks for top level mutex. Many more
reads than writes Summary: Test Plan: Reviewers: Subscribers: Tasks: Blame Revision:
-
adding delete handler. Minor fixes
Summary: Test Plan: Reviewers: Subscribers: Tasks: Blame Revision:
-
Incremental changes (e.g. Read Write Locks )
Summary: 1. Read Write Locks. 2. Reduce critical section during writes 3. valgrind runs and fixes Test Plan: Reviewers: Subscribers: Tasks: Blame Revision:
-
stackable db fix. more rigorous unit test
Summary: 1. unit test with lot of random blobs 2. stackable db should not take ownership of base db, because we share with blob db. 3. fixing ttl_guess logic for now, to make sure search doesn't miss files. Test Plan: Reviewers: Subscribers: Tasks: Blame Revision:
-
Summary: Test Plan: Reviewers: Subscribers: Tasks: Blame Revision:
-
BREAK from delete eviction loop, when you meet a DELETE
Summary: Test Plan: Reviewers: Subscribers: Tasks: Blame Revision:
-
debugged and fixed issues with sequential file buffers. Rewind
Summary: 1. sequential file buffer management 2. rewind functionality 3. some logs for error cases. 4. making the time range of a blob log pick up correct option Test Plan: Reviewers: Subscribers: Tasks: Blame Revision:
-
Reorganizing functions to share with regular no-TTL put
Summary: Test Plan: Reviewers: Subscribers: Tasks: Blame Revision:
-
Summary: Test Plan: Reviewers: Subscribers: Tasks: Blame Revision:
-
Clipping start and end ttls of files
Summary: 1. align the ttl start and end to multiples of ttl_range this keeps the ttl range of files tractable and non-overlapping 2. remove an option from options file Test Plan: Reviewers: Subscribers: Tasks: Blame Revision:
-
Adding a timer queue. Minor changes.
Summary: Also using make format to reformat changes Test Plan: Reviewers: Subscribers: Tasks: Blame Revision:
-
Summary: Test Plan: Reviewers: Subscribers: Tasks: Blame Revision:
-
Factor out shouldGCFile. Partial file gc
Summary: Test Plan: Reviewers: Subscribers: Tasks: Blame Revision:
-
Adding support for OnFlushBegin listener
Summary: This callback ensures that before a flush happens blob log files are also fsync'd Test Plan: Reviewers: Subscribers: Tasks: Blame Revision:
-
Assertions in sequential read and write logs for safety
Summary: Test Plan: Reviewers: Subscribers: Tasks: Blame Revision:
-
Framework for WAL-filter callback. BlobDB needs to be created before …
…DB::Open Summary: Test Plan: Reviewers: Subscribers: Tasks: Blame Revision:
-
DestroyBlobDB and another unit test for GC
Summary: Test Plan: Reviewers: Subscribers: Tasks: Blame Revision:
-
Delete simple and complex blob test. Adding Put Interface to make sta…
…ckable DB work easier Summary: Test Plan: Reviewers: Subscribers: Tasks: Blame Revision:
-
Single Delete, Multi Get and Write handling
Summary: Test Plan: Reviewers: Subscribers: Tasks: Blame Revision:
-
making set of open files a multiset.
Summary: 1. open files are now a multiset. 2. fixing minor race condition between async closeSeqFiles and Put 3. more diagnostics and asserts 4. removing lambda's for debug simplicity 5. providig a functor for TTL extraction Test Plan: db_bench Reviewers: Subscribers: Tasks: Blame Revision:
-
Fixing Lint issues and code review feedback
Summary: Test Plan: Reviewers: Subscribers: Tasks: Blame Revision:
-
Supported for override callback during compaction
Summary: Added a callback which informs the keys that are being compacted. No Filtering. The EvictAllVersions compaction filter, registers the packet into a lockless queue and marks the blogs evicted Test Plan: OverrideTest added to blob_db_test Reviewers: Subscribers: Tasks: Blame Revision:
-
Addressing review comments from Siying and Murali
Summary: Review comments from Siying, Murali, Yuslepukhin and Lichuang 1. Relocating files from db area to utilities area 2. Replacing DecodeFixed32/64 and offset calculation with GetFixed32/64 3. removing invalid comments from blob_log_writer.h 4. changing comment style 5. doing make_format again 6. changing the enum style 7. directory fsync on file add and delete 8. putting magic number as last field in footer 9. Making Header Construct use PutFixed32 instead of Encode... 10. Added CRC checks on Get 11. Removing extra option in options.h for flush_begin_listener 12. removing raw pointers from blob log format 13. Removing has_ttl and has_ts and replacing with single unique_ptr 14. Appveyor and Travis failures 15. Better APIs for Column Families. Simplifying interfaces. 16. Compression of Blob Support 17. Splitting into multiple files, to reduce file size 18. GC transaction 1 at a time, instead of all at once 19. Proper bookkeeping of deletes after open of db(crash safety) Test Plan: Reviewers: Subscribers: Tasks: Blame Revision:
