iterator.seekToFirst() high latency #261
Comments
|
Thanks for reporting this. Can I first get your testing code to further investigate the issue? |
|
Hey @naveenatceg, this post on our discussion group might be insightful: https://www.facebook.com/groups/rocksdb.dev/permalink/604723469626171/. Since deletes are not actually deletes, but rather tombstones (which get cleaned up after a compaction), your iterator might have to skip a bunch of deletes the get to the first key in the database. Can you profile your code and see where it's spending most of the time? |
|
Hey guys, |
|
@naveenatceg any updates? :) |
|
Hey Igor, The all() call does something very simple to open the iterator. |
|
Yup, this is what's probably happening. You need to iterate over bunch of deletions on SeekToFirst(). One way to confirm this is by utilizing our perf_context functionality: https://github.com/facebook/rocksdb/blob/master/include/rocksdb/perf_context.h From C++: You should notice that If your updates are always increasing, you can fix this by caching smallest live key and then instead of doing SeekToFirst(), just do Seek(smallest_live_key). I did something similar in mongo-rocks: https://github.com/mongodb-partners/mongo-rocks/blob/master/src/rocks_record_store.cpp#L418 (check out _oplogNextToDelete) |
|
Thank you for reporting this issue and appreciate your patience. We've notified the core team for an update on this issue. We're looking for a response within the next 30 days or the issue may be closed. |
|
I recently found another issue in the test. One of our application stuck with long latency in this SeekToFirst() operation. I took a test program try to open the db and list all the keys and found that the latency listing n keys vs the latency listing n+1 keys sometimes differs by orders of magnitude: i.e. in my test, listing 141 keys from an open iterator takes 64ms and listing 142 keys from an open iterator takes 4. second. It seems likes that the problem does not only exist in SeekToFirst(). Iterator.next() may hit the same problem, if there are a large number of deleted records in-between two keys. The only solution to this seems to be the compaction. However, we noticed that there is no log indicating that compaction thread is active for about 6 days. Is it a known issue? Under which conditions the compaction thread will not actually compact the SST files? What would be the suggested solution? Thanks a lot! |
|
If there are no writes, the compaction will not happen. You can manually issue compaction by calling CompactRange() method. |
|
@igorcanadi thanks for the quick response. I have the following additional questions:
Lastly, a newbie question, for the following snapshot of a RocksDB instance, how do I detect sorted runs here? Are the smaller number .sst files older runs or newer runs? Thanks a lot in advance! |
|
Closing this via automation due to lack of activity. If discussion is still needed here, please re-open or create a new/updated issue. |
Changed cloud_file_deletion_delay to std::optional. If it's not set, instead of scheduling cloud file deletion on file deletion queue asynchronously, we delete the cloud files immediately. This is different from setting the value to 0, in which case, the file deletion is still scheduled asynchronously. Also did some minor cleanup: * removed useless TEST_ functions * only included TEST_ functions if compiling in debug build
Hey all,
I am using RocksDB iterator to seek to beginning and then perform some deletes (around 32 deletes/second). The behavior we are noticing is that the seek call takes about a second to return and takes up almost all of the CPU.
We observed the iostats and we do not see any heavy IO operation being performed. We keep doing this operation repeatedly (for testing purposes), initially it doesn't take that long (takes about 100ms), but slowly creeps up and stays at 1s for a long time. When it hits this 1s threshold, RocksDB doesn't print any logs after this (nothing for 8 hours sometimes).
The first graph is the number of key-value pairs and the second graph is the time it takes to seek to the beginning.
Can someone please explain this behavior ? Why would it take the iterator that long to seek to beginning ? Why do the logs start disappearing ?
Configuration:
The last few minutes before RocksDB stops printing log:
The text was updated successfully, but these errors were encountered: