KAFKA-12467: Add controller-side snapshot generation #10366
Conversation
cmccabe
force-pushed
the
KAFKA-12467
branch
2 times, most recently
from
da4d236 to
5c4ed91
Compare
cmccabe
force-pushed
the
KAFKA-12467
branch
2 times, most recently
from
9ae7f3c to
6cc1e49
Compare
There was a problem hiding this comment.
Thanks for the patch @cmccabe, this looks nice 👍
Returning an iterator of batches works well when the control managers only have a single iterable thing that needs to be translated to records. Do you think we might have cases (eventually) where the control classes have more complex state? It might be more flexible to take a consumer rather than return an iterable:
// In the control manager
void toRecords(Consumer<List<ApiMessageAndVersion>> batchConsumer) {
...
}A few other comments/questions inline.
|
|
||
| ClientQuotaControlIterator(long epoch) { | ||
| this.epoch = epoch; | ||
| this.iterator = clientQuotaData.entrySet(epoch).iterator(); |
There was a problem hiding this comment.
Since we're getting the entry set of a particular epoch, does this mean we'll be getting consistent snapshots?
There was a problem hiding this comment.
Yes, snapshots are consistent.
metadata/src/main/java/org/apache/kafka/controller/QuorumController.java
Show resolved
Hide resolved
| if (result < 0) { | ||
| return OptionalLong.empty(); | ||
| } else if (result > 0) { | ||
| return OptionalLong.of(exponentialBackoff.backoff(numWriteTries - 1)); |
There was a problem hiding this comment.
Should we use result here instead of numWriteTries? If not, maybe we can change generateBatch() to return a boolean instead
There was a problem hiding this comment.
It's fine to use result here, I guess.
Hmm... I don't think that would be a problem. You could have a single iterator that iterates over multiple data structures X and Y. You would just need to store the state of whether you were done with X before starting Y. In general one of the goals of controller snapshots is to avoid "stopping the world" while the snapshot is generated. So a synchronous API would not be useful here. The iterator API allows us to take only a few batches of records at a time. |
Implement controller-side snapshot generation.
|
Fix spotbugs |
Implement writing snapshots in the controller, and reading them back.