Owner:
Is your feature request related to a problem? Please describe.
In Kafka Streams up to 2.6, to speed up restoration of RocksDB-backed state stores, we used to apply bulk loading when restoring. We used Options::prepareForBulkLoad() for that. So all data was first written to L0 during restoration without any compactions. When restoration ended and before normal processing started, we switched the state store back to non-bulk-loading mode. Now compactions started on all the restored data on L0. During a compaction writes to the state store stall. These meant, we sometimes experienced write stalls when normal processing started. The duration of these write stalls exceeded a timeout that we have in Kafka Streams (max.poll.interval.ms for the interested reader) which hindered progress in processing. Due to this issue, we removed bulk loading during restoration.
Now we did some improvements in restoration that would maybe allow to re-add bulk loading during restoration. However, we would need to make a manual compaction with compactRange() before we switch to normal processing for a specific store. Since we have multiple state stores to restore, we do not want to be blocked by the manual compaction of each state store after restoration. We would rather want to start manual compaction for one state store that finished restoration and then continue to restore the other state stores. Once the compaction on the state store has finished we want to switch it to normal processing. For that we need some ways to verify if the compaction has ended like a future.
Describe the solution you'd like
A non-blocking compactRange() and a way to verify that the compaction started with compactRange() finished.
Owner:
Is your feature request related to a problem? Please describe.
In Kafka Streams up to 2.6, to speed up restoration of RocksDB-backed state stores, we used to apply bulk loading when restoring. We used
Options::prepareForBulkLoad()for that. So all data was first written to L0 during restoration without any compactions. When restoration ended and before normal processing started, we switched the state store back to non-bulk-loading mode. Now compactions started on all the restored data on L0. During a compaction writes to the state store stall. These meant, we sometimes experienced write stalls when normal processing started. The duration of these write stalls exceeded a timeout that we have in Kafka Streams (max.poll.interval.ms for the interested reader) which hindered progress in processing. Due to this issue, we removed bulk loading during restoration.Now we did some improvements in restoration that would maybe allow to re-add bulk loading during restoration. However, we would need to make a manual compaction with
compactRange()before we switch to normal processing for a specific store. Since we have multiple state stores to restore, we do not want to be blocked by the manual compaction of each state store after restoration. We would rather want to start manual compaction for one state store that finished restoration and then continue to restore the other state stores. Once the compaction on the state store has finished we want to switch it to normal processing. For that we need some ways to verify if the compaction has ended like a future.Describe the solution you'd like
A non-blocking
compactRange()and a way to verify that the compaction started withcompactRange()finished.