Add SstFileReader to read sst files #4717
Conversation
A user friendly sst file reader is useful when we want to access sst files outside of RocksDB. For example, we can generate an sst file with SstFileWriter and send it to other places, then use SstFileReader to read the file and process the entries in other ways. Also rename the original SstFileReader to SstFileDumper because of name conflict, and seems SstFileDumper is more appropriate for tools.
| kMaxSequenceNumber; | ||
| auto internal_iter = r->table_reader->NewIterator( | ||
| options, r->moptions.prefix_extractor.get()); | ||
| return NewDBIterator(r->options.env, options, r->ioptions, r->moptions, |
There was a problem hiding this comment.
interesting, so you don't need to expose sequence number?
| const Slice& key, PinnableSlice* value); | ||
|
|
||
| // Returns a new iterator over the table contents. | ||
| Iterator* NewIterator(const ReadOptions& options); |
There was a problem hiding this comment.
maybe document how it returns only latest keys unless ReadOptions::snapshot is set to a live snapshot
Do you plan to use this API for reading from snapshots? If so is this API easier for you compared to having an iterator that returns internal keys?
|
@ajkr IMO, there are two kinds of requirements to read from an ss file:
This PR tries to provide the API for the first use case. For whether to support snapshot read, I don't need that personally because I only need to read from sst files generated by |
There was a problem hiding this comment.
Thanks for the explanation @huachaohuang. I guess there's no harm in supporting snapshot reads, even if they won't be used right now.
There was a problem hiding this comment.
@ajkr has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.
Summary: A user friendly sst file reader is useful when we want to access sst files outside of RocksDB. For example, we can generate an sst file with SstFileWriter and send it to other places, then use SstFileReader to read the file and process the entries in other ways. Also rename the original SstFileReader to SstFileDumper because of name conflict, and seems SstFileDumper is more appropriate for tools. TODO: there is only a very simple test now, because I want to get some feedback first. If the changes look good, I will add more tests soon. Pull Request resolved: facebook#4717 Differential Revision: D13212686 Pulled By: ajkr fbshipit-source-id: 737593383264c954b79e63edaf44aaae0d947e56
A user friendly sst file reader is useful when we want to access sst
files outside of RocksDB. For example, we can generate an sst file
with SstFileWriter and send it to other places, then use SstFileReader
to read the file and process the entries in other ways.
Also rename the original SstFileReader to SstFileDumper because of
name conflict, and seems SstFileDumper is more appropriate for tools.
TODO: there is only a very simple test now, because I want to get some feedback first.
If the changes look good, I will add more tests soon.