feat(inkless): implement InitDisklessLog Controller API#531
Merged
Conversation
giuseppelillo
force-pushed
the
giuseppelillo/push-diskless-start-offset-to-controller
branch
2 times, most recently
from
959f497 to
5e0d79a
Compare
Persist disklessStartOffset and producer states into KRaft metadata. Only the leader is allowed to call InitDisklessLog. New fields are also persisted in the KRaft snapshots through PartitionRegistration. To avoid conflicts with upstream changes to the metadata records, new metadata fields are written directly in the unknown tagged fields of PartitionRecord and PartitionChangeRecord.
giuseppelillo
force-pushed
the
giuseppelillo/push-diskless-start-offset-to-controller
branch
from
5e0d79a to
4573f9f
Compare
jeqo
reviewed
Mar 11, 2026
metadata/src/main/java/org/apache/kafka/controller/InitDisklessLogRequestData.java
Outdated
Show resolved
Hide resolved
| int partitionId, | ||
| long disklessStartOffset, | ||
| int leaderEpoch, | ||
| List<ProducerState> producerStates |
Contributor
There was a problem hiding this comment.
Does ProducerState need to be persisted on KRaft? I wonder if it could be sent from the leader to DisklessController directly
Contributor
Author
There was a problem hiding this comment.
Yes it needs to be because it might happen that the broker that is able to successfully call InitDisklessLog on the controller (so the real leader) is not the same that calls the Diskless Controller (it might be another one, even a zombie leader in a split brain scenario). To keep this protocol 100% safe without any other step involved we also need to "commit" the producer state, so that once InitDisklessLog controller API is called the partition state is completely frozen.
jeqo
reviewed
Mar 11, 2026
metadata/src/main/java/org/apache/kafka/controller/ReplicationControlManager.java
Show resolved
Hide resolved
metadata/src/main/java/org/apache/kafka/controller/ReplicationControlManager.java
Show resolved
Hide resolved
metadata/src/main/java/org/apache/kafka/controller/ReplicationControlManager.java
Show resolved
Hide resolved
metadata/src/main/java/org/apache/kafka/controller/ReplicationControlManager.java
Show resolved
Hide resolved
jeqo
requested changes
Mar 12, 2026
| buf.putLong(s.assignedOffset()); | ||
| buf.putLong(s.batchMaxTimestamp()); | ||
| } | ||
| buf.flip(); |
Contributor
There was a problem hiding this comment.
nit: can be dropped as the backing array doesn't care about position
metadata/src/main/java/org/apache/kafka/controller/ReplicationControlManager.java
Outdated
Show resolved
Hide resolved
jeqo
approved these changes
Mar 12, 2026
AnatolyPopov
pushed a commit
that referenced
this pull request
Mar 23, 2026
* feat(inkless): implement InitDisklessLog Controller API Persist disklessStartOffset and producer states into KRaft metadata. Only the leader is allowed to call InitDisklessLog. New fields are also persisted in the KRaft snapshots through PartitionRegistration. To avoid conflicts with upstream changes to the metadata records, new metadata fields are written directly in the unknown tagged fields of PartitionRecord and PartitionChangeRecord. (cherry picked from commit 0070155) # Conflicts: # metadata/src/main/java/org/apache/kafka/metadata/PartitionRegistration.java
jeqo
pushed a commit
that referenced
this pull request
Mar 23, 2026
* feat(inkless): implement InitDisklessLog Controller API Persist disklessStartOffset and producer states into KRaft metadata. Only the leader is allowed to call InitDisklessLog. New fields are also persisted in the KRaft snapshots through PartitionRegistration. To avoid conflicts with upstream changes to the metadata records, new metadata fields are written directly in the unknown tagged fields of PartitionRecord and PartitionChangeRecord.
jeqo
pushed a commit
that referenced
this pull request
Mar 23, 2026
* feat(inkless): implement InitDisklessLog Controller API Persist disklessStartOffset and producer states into KRaft metadata. Only the leader is allowed to call InitDisklessLog. New fields are also persisted in the KRaft snapshots through PartitionRegistration. To avoid conflicts with upstream changes to the metadata records, new metadata fields are written directly in the unknown tagged fields of PartitionRecord and PartitionChangeRecord. (cherry picked from commit 0070155) # Conflicts: # metadata/src/main/java/org/apache/kafka/metadata/PartitionRegistration.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Persist disklessStartOffset and producer states into KRaft metadata.
Only the leader is allowed to call InitDisklessLog.
To avoid conflicts with upstream changes to the metadata records, new metadata fields are written directly in the unknown tagged fields.
New fields are also persisted in the KRaft snapshots through PartitionRegistration.