KAFKA-16089: Fix memory leak in RocksDBStore #15174
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
`ColumnFamilyDescriptor` is _not_ a `RocksObject`, which in theory means it's not backed by any native memory allocated by RocksDB. However, in practice, `ColumnFamilyHandle#getDescriptor()`, which returns a `ColumnFamilyDescriptor`, allocates an internal `rocksdb::db::ColumnFamilyDescriptor`, copying the name and handle of the column family into it. Since the Java `ColumnFamilyDescriptor` is not a `RocksObject`, it's not possible to track this allocation and free it from Java. Fortunately, in our case, we can simply avoid calling `ColumnFamilyHandle#getDescriptor()`, since we're only interested in the column family name, which is already available on `ColumnFamilyHandle#getName()`, which does not leak memory. We can also optimize away the temporary `Options`, which was previously a source of memory leaks, because `userSpecifiedOptions` is an instance of `Options`.
|
@lucasbru I'm just writing up a more detailed explanation, including graphs, in the ticket. I'm also going to raise a bug with RocksDB, because I believe this to be a bug in RocksJNI. |
nicktelford
added a commit
to nicktelford/kafka
that referenced
this pull request
Jan 12, 2024
`ColumnFamilyDescriptor` is _not_ a `RocksObject`, which in theory means it's not backed by any native memory allocated by RocksDB. However, in practice, `ColumnFamilyHandle#getDescriptor()`, which returns a `ColumnFamilyDescriptor`, allocates an internal `rocksdb::db::ColumnFamilyDescriptor`, copying the name and handle of the column family into it. Since the Java `ColumnFamilyDescriptor` is not a `RocksObject`, it's not possible to track this allocation and free it from Java. Fortunately, in our case, we can simply avoid calling `ColumnFamilyHandle#getDescriptor()`, since we're only interested in the column family name, which is already available on `ColumnFamilyHandle#getName()`, which does not leak memory. We can also optimize away the temporary `Options`, which was previously a source of memory leaks, because `userSpecifiedOptions` is an instance of `Options`. Reviewers: Lucas Brutschy <lbrutschy@confluent.io>
showuon
pushed a commit
to showuon/kafka
that referenced
this pull request
Jan 22, 2024
`ColumnFamilyDescriptor` is _not_ a `RocksObject`, which in theory means it's not backed by any native memory allocated by RocksDB. However, in practice, `ColumnFamilyHandle#getDescriptor()`, which returns a `ColumnFamilyDescriptor`, allocates an internal `rocksdb::db::ColumnFamilyDescriptor`, copying the name and handle of the column family into it. Since the Java `ColumnFamilyDescriptor` is not a `RocksObject`, it's not possible to track this allocation and free it from Java. Fortunately, in our case, we can simply avoid calling `ColumnFamilyHandle#getDescriptor()`, since we're only interested in the column family name, which is already available on `ColumnFamilyHandle#getName()`, which does not leak memory. We can also optimize away the temporary `Options`, which was previously a source of memory leaks, because `userSpecifiedOptions` is an instance of `Options`. Reviewers: Lucas Brutschy <lbrutschy@confluent.io>
nicktelford
added a commit
to nicktelford/kafka
that referenced
this pull request
Jan 24, 2024
`ColumnFamilyDescriptor` is _not_ a `RocksObject`, which in theory means it's not backed by any native memory allocated by RocksDB. However, in practice, `ColumnFamilyHandle#getDescriptor()`, which returns a `ColumnFamilyDescriptor`, allocates an internal `rocksdb::db::ColumnFamilyDescriptor`, copying the name and handle of the column family into it. Since the Java `ColumnFamilyDescriptor` is not a `RocksObject`, it's not possible to track this allocation and free it from Java. Fortunately, in our case, we can simply avoid calling `ColumnFamilyHandle#getDescriptor()`, since we're only interested in the column family name, which is already available on `ColumnFamilyHandle#getName()`, which does not leak memory. We can also optimize away the temporary `Options`, which was previously a source of memory leaks, because `userSpecifiedOptions` is an instance of `Options`. Reviewers: Lucas Brutschy <lbrutschy@confluent.io>
yyu1993
pushed a commit
to yyu1993/kafka
that referenced
this pull request
Feb 15, 2024
`ColumnFamilyDescriptor` is _not_ a `RocksObject`, which in theory means it's not backed by any native memory allocated by RocksDB. However, in practice, `ColumnFamilyHandle#getDescriptor()`, which returns a `ColumnFamilyDescriptor`, allocates an internal `rocksdb::db::ColumnFamilyDescriptor`, copying the name and handle of the column family into it. Since the Java `ColumnFamilyDescriptor` is not a `RocksObject`, it's not possible to track this allocation and free it from Java. Fortunately, in our case, we can simply avoid calling `ColumnFamilyHandle#getDescriptor()`, since we're only interested in the column family name, which is already available on `ColumnFamilyHandle#getName()`, which does not leak memory. We can also optimize away the temporary `Options`, which was previously a source of memory leaks, because `userSpecifiedOptions` is an instance of `Options`. Reviewers: Lucas Brutschy <lbrutschy@confluent.io>
clolov
pushed a commit
to clolov/kafka
that referenced
this pull request
Apr 5, 2024
`ColumnFamilyDescriptor` is _not_ a `RocksObject`, which in theory means it's not backed by any native memory allocated by RocksDB. However, in practice, `ColumnFamilyHandle#getDescriptor()`, which returns a `ColumnFamilyDescriptor`, allocates an internal `rocksdb::db::ColumnFamilyDescriptor`, copying the name and handle of the column family into it. Since the Java `ColumnFamilyDescriptor` is not a `RocksObject`, it's not possible to track this allocation and free it from Java. Fortunately, in our case, we can simply avoid calling `ColumnFamilyHandle#getDescriptor()`, since we're only interested in the column family name, which is already available on `ColumnFamilyHandle#getName()`, which does not leak memory. We can also optimize away the temporary `Options`, which was previously a source of memory leaks, because `userSpecifiedOptions` is an instance of `Options`. Reviewers: Lucas Brutschy <lbrutschy@confluent.io>
Phuc-Hong-Tran
pushed a commit
to Phuc-Hong-Tran/kafka
that referenced
this pull request
Jun 6, 2024
`ColumnFamilyDescriptor` is _not_ a `RocksObject`, which in theory means it's not backed by any native memory allocated by RocksDB. However, in practice, `ColumnFamilyHandle#getDescriptor()`, which returns a `ColumnFamilyDescriptor`, allocates an internal `rocksdb::db::ColumnFamilyDescriptor`, copying the name and handle of the column family into it. Since the Java `ColumnFamilyDescriptor` is not a `RocksObject`, it's not possible to track this allocation and free it from Java. Fortunately, in our case, we can simply avoid calling `ColumnFamilyHandle#getDescriptor()`, since we're only interested in the column family name, which is already available on `ColumnFamilyHandle#getName()`, which does not leak memory. We can also optimize away the temporary `Options`, which was previously a source of memory leaks, because `userSpecifiedOptions` is an instance of `Options`. Reviewers: Lucas Brutschy <lbrutschy@confluent.io>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
ColumnFamilyDescriptoris not aRocksObject, which in theory means it's not backed by any native memory allocated by RocksDB.However, in practice,
ColumnFamilyHandle#getDescriptor(), which returns aColumnFamilyDescriptor, allocates an internalrocksdb::db::ColumnFamilyDescriptor, copying the name and options of the column family into it.Since the Java
ColumnFamilyDescriptoris not aRocksObject, it's not possible to track this allocation and free it from Java.Fortunately, in our case, we can simply avoid calling
ColumnFamilyHandle#getDescriptor(), since we're only interested in the column family name, which is already available onColumnFamilyHandle#getName(), which does not leak memory.We can also optimize away the temporary
Options, which was previously a source of memory leaks, becauseuserSpecifiedOptionsis an instance ofOptions.