KAFKA-14412: Decouple RocksDB access from CF #15105
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
@cadonna @mjsax @ableegoldman @lucasbru @wcarlson5 @bbejeck @vvcephei @guozhangwang This is part of KIP-892, and has been broken out into a separate PR to reduce the review burden on the main KIP-892 implementation, since it can be merged independently. There are no tests, because there are no behavioural changes, just a refactoring. The existing test suite should ensure no regressions. |
nicktelford
force-pushed
the
rocksdb-refactor-accessor-2
branch
2 times, most recently
from
4e94445
to
fdd9171
Compare
|
@nicktelford Seems like all build jobs timed out. Could you take a look? |
|
@lucasbru I think this is just the CI causing trouble again. The build passes locally. Is there a way to rerun the build without pushing more commits? |
|
@nicktelford I'll rerun it. That being said, the last jobs on trunk have all finished within 3-5 hours, so this must be caused by either infrastructure or the code in this PR. |
To support future use-cases that use different strategies for accessing
RocksDB, we need to de-couple the RocksDB access strategy from the
Column Family access strategy.
To do this, we now have two separate accessors:
* `DBAccessor`: dictates how we access RocksDB. Currently only one
strategy is supported: `DirectDBAccessor`, which access RocksDB
directly, via the `RocksDB` class for all operations. In the future, a
`BatchedDBAccessor` will be added, which enables transactions via
`WriteBatch`.
* `ColumnFamilyAccessor`: maps StateStore operations to operations on
one or more column families. This is a rename of the old
`RocksDBDBAccessor`.
nicktelford
force-pushed
the
rocksdb-refactor-accessor-2
branch
from
fdd9171
to
dce96bb
Compare
|
@lucasbru OK, my bad. It turns out I did a minor refactoring after I ran the test suite yesterday that was so insignificant I didn't think I needed to run the tests again... Turns out I was wrong 🙈 I've fixed the bug now and the tests pass locally for real now. |
lucasbru
added a commit
to lucasbru/kafka
that referenced
this pull request
Jan 10, 2024
This reverts commit 5bc3aa4.
nicktelford
added a commit
to nicktelford/kafka
that referenced
this pull request
Jan 12, 2024
To support future use-cases that use different strategies for accessing
RocksDB, we need to de-couple the RocksDB access strategy from the
Column Family access strategy.
To do this, we now have two separate accessors:
* `DBAccessor`: dictates how we access RocksDB. Currently only one
strategy is supported: `DirectDBAccessor`, which access RocksDB
directly, via the `RocksDB` class for all operations. In the future, a
`BatchedDBAccessor` will be added, which enables transactions via
`WriteBatch`.
* `ColumnFamilyAccessor`: maps StateStore operations to operations on
one or more column families. This is a rename of the old
`RocksDBDBAccessor`.
Reviewers: Lucas Brutschy <lbrutschy@confluent.io>
nicktelford
added a commit
to nicktelford/kafka
that referenced
this pull request
Jan 24, 2024
To support future use-cases that use different strategies for accessing
RocksDB, we need to de-couple the RocksDB access strategy from the
Column Family access strategy.
To do this, we now have two separate accessors:
* `DBAccessor`: dictates how we access RocksDB. Currently only one
strategy is supported: `DirectDBAccessor`, which access RocksDB
directly, via the `RocksDB` class for all operations. In the future, a
`BatchedDBAccessor` will be added, which enables transactions via
`WriteBatch`.
* `ColumnFamilyAccessor`: maps StateStore operations to operations on
one or more column families. This is a rename of the old
`RocksDBDBAccessor`.
Reviewers: Lucas Brutschy <lbrutschy@confluent.io>
3 tasks
yyu1993
pushed a commit
to yyu1993/kafka
that referenced
this pull request
Feb 15, 2024
To support future use-cases that use different strategies for accessing
RocksDB, we need to de-couple the RocksDB access strategy from the
Column Family access strategy.
To do this, we now have two separate accessors:
* `DBAccessor`: dictates how we access RocksDB. Currently only one
strategy is supported: `DirectDBAccessor`, which access RocksDB
directly, via the `RocksDB` class for all operations. In the future, a
`BatchedDBAccessor` will be added, which enables transactions via
`WriteBatch`.
* `ColumnFamilyAccessor`: maps StateStore operations to operations on
one or more column families. This is a rename of the old
`RocksDBDBAccessor`.
Reviewers: Lucas Brutschy <lbrutschy@confluent.io>
AnatolyPopov
pushed a commit
to aiven/kafka
that referenced
this pull request
Feb 16, 2024
To support future use-cases that use different strategies for accessing
RocksDB, we need to de-couple the RocksDB access strategy from the
Column Family access strategy.
To do this, we now have two separate accessors:
* `DBAccessor`: dictates how we access RocksDB. Currently only one
strategy is supported: `DirectDBAccessor`, which access RocksDB
directly, via the `RocksDB` class for all operations. In the future, a
`BatchedDBAccessor` will be added, which enables transactions via
`WriteBatch`.
* `ColumnFamilyAccessor`: maps StateStore operations to operations on
one or more column families. This is a rename of the old
`RocksDBDBAccessor`.
Reviewers: Lucas Brutschy <lbrutschy@confluent.io>
clolov
pushed a commit
to clolov/kafka
that referenced
this pull request
Apr 5, 2024
To support future use-cases that use different strategies for accessing
RocksDB, we need to de-couple the RocksDB access strategy from the
Column Family access strategy.
To do this, we now have two separate accessors:
* `DBAccessor`: dictates how we access RocksDB. Currently only one
strategy is supported: `DirectDBAccessor`, which access RocksDB
directly, via the `RocksDB` class for all operations. In the future, a
`BatchedDBAccessor` will be added, which enables transactions via
`WriteBatch`.
* `ColumnFamilyAccessor`: maps StateStore operations to operations on
one or more column families. This is a rename of the old
`RocksDBDBAccessor`.
Reviewers: Lucas Brutschy <lbrutschy@confluent.io>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
To support future use-cases that use different strategies for accessing RocksDB, we need to de-couple the RocksDB access strategy from the Column Family access strategy.
To do this, we now have two separate accessors:
DBAccessor: dictates how we access RocksDB. Currently only one strategy is supported:DirectDBAccessor, which access RocksDB directly, via theRocksDBclass for all operations. In the future, aBatchedDBAccessorwill be added, which enables transactions viaWriteBatch.ColumnFamilyAccessor: maps StateStore operations to operations on one or more column families. This is a rename of the oldRocksDBDBAccessor.