[Backport stable/8.4] [Backport main] Fix Migration to 8.3 OOM Failure #16142
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ation state classes In the previous implementation, one instance of RocksIterator was iterating through the whole ColumnFamily. Since each instance of RocksIterator opens its transaction, that led the previous implementation to occupy the memory equal to size of key-value pairs. RocksDB manages memory using malloc/free algorithm. In that implementation if there is enough memory left in the total allocated memory, it keeps freed memory unused, because it expects it to be used again by the same object. For that reason, when RocksIterator occupies too much memory, there is a possibility that after it is done, the free memory might not be allocated for other threads. In the updated implementation, the memory usage per iterator is limited to a fixed size. This will provide OS to re-use (or at least keep less sized non-used memory) occupied memory when it is no longer used by another iterator. The fixed limit is chosen as default 50MB. It is because, 50MB appeared to be a sweet spot while testing the fix. When I use 100MB, it was almost doubling the memory usage and when I use 25MB, it only increased the migration time while not improving on the memory usage. (cherry picked from commit 4a0de14) (cherry picked from commit 3232747)
For the same reason as mentioned in the previous commit, we should keep the memory usage of each transaction low. Otherwise, the data retrieved will be kept in memory and after its usage, the freed memory might not be reused. If we do not remove this transaction, we will not benefit from memory limiting each `columnFamily.foreach` loop because it will keep changes uncommitted. Therefore, it will cause occupying the whole migration data in memory until it is committed. (cherry picked from commit d94a32b) (cherry picked from commit e6053f6)
(cherry picked from commit c4d6466)
(cherry picked from commit 229996a)
camundait
approved these changes
Jan 30, 2024
|
Backport-ception 🥳 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Backport of #16113 to
stable/8.4.relates to #16090 #14975
original author: @backport-action