Summary
Disk polls run asynchronously so that file I/O does not block the shard that owns a partition. The poll captures the resources it needs, performs the read in a detached task, and completes later.
The detached poll currently also has access to shared consumer progress state. When the read finishes, it can apply an automatic commit or update a consumer group's last_polled value before the partition owner establishes that the result still belongs to the current message history.
This becomes incorrect when a topic is purged while the disk read is pending. Purge removes the old messages, clears consumer progress, and starts a new offset space at zero. The pending poll is not revoked. It can therefore finish with information from the history that existed before the purge and apply that information to the history that exists after the purge.
Example
Consider an individual consumer that polls with Next and automatic commit enabled:
| Time |
Partition state |
Delayed poll |
| 1 |
Old messages occupy offsets 0 through 2 |
A disk poll starts for offsets 0 through 2 |
| 2 |
The topic is purged; messages and consumer progress are cleared |
The disk read is still pending |
| 3 |
Five new messages are appended at offsets 0 through 4 |
The pending poll still belongs to the old history |
| 4 |
The new history is ready for consumers |
The old poll completes and records offset 2 |
| 5 |
The consumer issues its next Next poll |
Polling resumes after offset 2 |
The consumer can now skip the new messages at offsets 0 through 2. Checking only whether offset 2 is within the current range cannot detect the problem because offset values are reused after purge. The same number identifies different messages before and after the purge.
If fewer new messages exist, the stale progress can instead move the consumer beyond the current end of the topic and cause empty polls until the new history catches up.
Why the current completion checks are insufficient
PollPlan is created while the partition is borrowed by its owning shard. To let the read continue after that borrow ends, the plan captures owned file resources and shared handles for consumer offsets, automatic commit capacity, durable offset state, and group last_polled state.
For a disk poll, PollPlan::execute runs outside the partition owner. After the file read completes, it can update the captured offset map and produce an AutoCommitApplied value. The server later borrows the partition again to decide whether to submit the corresponding StoreConsumerOffset operation. If that decision rejects the operation, the earlier local update is rolled back.
This protects some changes in partition ownership and consensus state, but it does not prove that the message history is unchanged. A purge clears the existing shared maps rather than replacing every handle captured by older polls. A plan created before the purge can therefore still point at the same allocation after the purge. The allocation is current, but the read result is not.
There is a similar path for consumer groups. A poll without automatic commit can update last_polled directly from the detached completion, without returning to the partition owner for history validation.
This means the current flow mixes two responsibilities:
- Reading message data, which may wait for file I/O and may safely happen outside the owner.
- Deciding whether the result is still valid and applying consumer progress, which depends on the partition's current history and must be serialized with changes to that history.
The second responsibility cannot be made safe by validating offset numbers alone.
Observable impact
Depending on the poll type and timing, a completion from before the purge can:
- advance an individual consumer's offset in the new history;
- advance a consumer group's
last_polled value in the new history;
- enqueue or replicate an automatic commit derived from the old history;
- cause a later
Next poll to skip messages that were appended after the purge;
- authorize a reply whose message data and current offset were captured from a history that no longer exists.
The race requires a disk read to remain pending across purge, so it may be uncommon in normal tests. Slow storage, a busy runtime, or a deliberately suspended read makes the ordering reproducible.
Expected behavior
A poll result must only affect the partition history from which it was created. If that history has been replaced before completion:
- the result must be rejected with a retryable error;
- no messages from that result should be returned to the client;
- consumer offsets and group
last_polled values must remain unchanged;
- no automatic commit or capacity reservation from that result may survive.
This rule must still hold when the stale offset is numerically valid in the new history.
Suggested test
A deterministic regression test can expose the bug without relying on timing:
- Persist messages at offsets 0 through 2 so polling uses the disk path.
- Start a poll for those messages and suspend it at the file read.
- Purge the topic while the poll is suspended.
- Append new messages at offsets 0 through 4 and persist them.
- Resume the old poll and run its completion path.
- Verify that the completion is rejected and that consumer progress remains empty.
- Poll with
Next and verify that all five new messages are returned, starting at offset 0.
The regression should cover an individual consumer with automatic commit and a consumer group whose last_polled value would otherwise change. A poll without automatic commit should also be checked to ensure that an obsolete result cannot authorize a reply.
Possible implementation direction
Keep the asynchronous worker limited to reading messages. It should return the read fragments, relevant immutable request context, and an identity for the message history from which the read was planned. It should not hold handles that allow it to change consumer progress.
The partition owner can then receive the result, compare its history identity with the current one, perform admission and capacity checks, apply progress, and authorize the reply as one synchronous completion step. Any queued automatic commit should retain the same history identity so it can be discarded if the history changes before operation assignment.
The exact representation can be decided in the fix. The required invariant is that work created for an old partition history cannot change progress or authorize a successful poll in a newer history.
Summary
Disk polls run asynchronously so that file I/O does not block the shard that owns a partition. The poll captures the resources it needs, performs the read in a detached task, and completes later.
The detached poll currently also has access to shared consumer progress state. When the read finishes, it can apply an automatic commit or update a consumer group's
last_polledvalue before the partition owner establishes that the result still belongs to the current message history.This becomes incorrect when a topic is purged while the disk read is pending. Purge removes the old messages, clears consumer progress, and starts a new offset space at zero. The pending poll is not revoked. It can therefore finish with information from the history that existed before the purge and apply that information to the history that exists after the purge.
Example
Consider an individual consumer that polls with
Nextand automatic commit enabled:NextpollThe consumer can now skip the new messages at offsets 0 through 2. Checking only whether offset 2 is within the current range cannot detect the problem because offset values are reused after purge. The same number identifies different messages before and after the purge.
If fewer new messages exist, the stale progress can instead move the consumer beyond the current end of the topic and cause empty polls until the new history catches up.
Why the current completion checks are insufficient
PollPlanis created while the partition is borrowed by its owning shard. To let the read continue after that borrow ends, the plan captures owned file resources and shared handles for consumer offsets, automatic commit capacity, durable offset state, and grouplast_polledstate.For a disk poll,
PollPlan::executeruns outside the partition owner. After the file read completes, it can update the captured offset map and produce anAutoCommitAppliedvalue. The server later borrows the partition again to decide whether to submit the correspondingStoreConsumerOffsetoperation. If that decision rejects the operation, the earlier local update is rolled back.This protects some changes in partition ownership and consensus state, but it does not prove that the message history is unchanged. A purge clears the existing shared maps rather than replacing every handle captured by older polls. A plan created before the purge can therefore still point at the same allocation after the purge. The allocation is current, but the read result is not.
There is a similar path for consumer groups. A poll without automatic commit can update
last_polleddirectly from the detached completion, without returning to the partition owner for history validation.This means the current flow mixes two responsibilities:
The second responsibility cannot be made safe by validating offset numbers alone.
Observable impact
Depending on the poll type and timing, a completion from before the purge can:
last_polledvalue in the new history;Nextpoll to skip messages that were appended after the purge;The race requires a disk read to remain pending across purge, so it may be uncommon in normal tests. Slow storage, a busy runtime, or a deliberately suspended read makes the ordering reproducible.
Expected behavior
A poll result must only affect the partition history from which it was created. If that history has been replaced before completion:
last_polledvalues must remain unchanged;This rule must still hold when the stale offset is numerically valid in the new history.
Suggested test
A deterministic regression test can expose the bug without relying on timing:
Nextand verify that all five new messages are returned, starting at offset 0.The regression should cover an individual consumer with automatic commit and a consumer group whose
last_polledvalue would otherwise change. A poll without automatic commit should also be checked to ensure that an obsolete result cannot authorize a reply.Possible implementation direction
Keep the asynchronous worker limited to reading messages. It should return the read fragments, relevant immutable request context, and an identity for the message history from which the read was planned. It should not hold handles that allow it to change consumer progress.
The partition owner can then receive the result, compare its history identity with the current one, perform admission and capacity checks, apply progress, and authorize the reply as one synchronous completion step. Any queued automatic commit should retain the same history identity so it can be discarded if the history changes before operation assignment.
The exact representation can be decided in the fix. The required invariant is that work created for an old partition history cannot change progress or authorize a successful poll in a newer history.