Purge can record completion after consumer offset directory sync fails

[diegomrsantos]

Bug description

Purge can durably record its applied generation even when synchronizing a consumer offset directory fails. If a subsequent power loss loses the offset file deletions but preserves the purge generation, restart can restore the old consumer offsets without retrying the purge. Consumers can then skip messages written after the purge.

This affects both individual consumers and consumer groups. The failure path remains after #4092, including when consumer offset durability is persisted.

This report is based on code inspection. I have not injected the synchronization failure or reproduced the power loss sequence.

Affected component and versions

Iggy server. Inspected master at e0027506f2a0379ee3bb5508427ff570e3a9a435 and #4119 at d89b81f0297cbcb6577a5a69cd44857c94b0cdf0. The links below refer to the inspected master commit.

Evidence

1. After deleting consumer offset files, purge() synchronizes both offset directories. A failure only logs a warning. It does not return an error or preserve a pending cleanup obligation.
2. Execution can then persist purge.gen and advance applied_purge_generation. Recording that generation synchronizes its own parent directory, which does not establish durability for deletions inside the separate consumer offset directories.
3. On restart, recovery loads consumer and group offset files and seeds their values into partition state. It can clamp a value to the recovered offset range, but does not discard it based on the purge generation. A clamped offset is still a consumer entry; "consumed through 0" is different from having no committed offset.
4. The reconciler only requests another purge when the committed generation exceeds the applied generation. A surviving purge.gen therefore prevents the cleanup from being retried for that generation.

The durable journal marker added in #4092 does not close this gap. It is written before offset cleanup. Its synchronization cannot cover deletions that occur later. Recovery restores the journal purge floor, but that floor does not invalidate the restored consumer offset files.

Failure sequence derived from the code

1. Persist consumer and group offsets for an existing message history.
2. Purge the topic. Offset file deletions succeed, but synchronization of either offset directory fails.
3. Allow the subsequent write of purge.gen to succeed.
4. Lose power before the unsynchronized directory changes become durable. Model the old offset files surviving while the new purge generation survives too.
5. Restart. Recovery loads the old offsets, while the reconciler sees the purge generation as already applied.
6. Poll fresh messages with Next. Restored consumer progress can skip messages in the new history.

A process restart alone is not a reliable reproduction because the operating system may still retain and flush the pending directory changes.

Expected behavior and suggested regression

The partition must not record the purge as locally applied while removal of its consumer offset files remains unconfirmed on durable storage. A synchronization failure must preserve a retry or recovery obligation. Recovery or retry must also avoid deleting fresh writes that the system has already acknowledged.

A deterministic regression should fail synchronization of each offset directory separately after successful deletion. It should verify that purge cannot reach an applied state that suppresses cleanup, then simulate recovery with the old offset files restored. After cleanup completes, both consumer offset maps should be empty and Next should return the complete fresh history. Cover both durability policies, a successful synchronization control, and preservation of any acknowledged fresh writes across cleanup retries.

Related issues

• server: partition journal repair resurrects purged data after restart #3793 tracks old operations being replayed through journal repair after purge. This report concerns the durability of local file deletions and does not require repair from another replica.
• A delayed disk poll can update consumer progress after a topic purge #4117, addressed by fix(partitions): complete polls on the owning shard #4119, concerns stale poll completions changing progress after purge. This report does not require a pending poll.

[hubcio]

i was thinking if we could test it using our simulator and yes, SimStorage already supports directory-sync failures and Crash::PowerLoss, including restoring directory entries whose deletion wasn’t synced. that gives us the failure model needed here.

what needs plumbing is the production purge and recovery path. directory scans, offset-file deletion, syncing the consumer and consumer-group directories, and persisting/reloading purge.gen need to go through DurableStorage. recovery needs to reload offsets from the simulated filesystem, and Next checks must use that recovered state. the cluster simulator carries offsets across restart in memory, so its normal restart path won’t exercise this failure. the test should preserve the separate offset and marker directories, since syncing one doesn’t make changes in the others durable.

the existing storage purge test implements its own sequence and propagates directory-sync errors with ?, while production logs them and continues. i’d reuse the production cleanup and completion code in that harness so the regression exercises the behavior we’re fixing.

then we can persist consumer and group offsets, successfully delete their files, inject a sync failure for each offset directory separately, allow subsequent operations, and trigger power loss. after recovery, we should verify that the completion marker can’t suppress required cleanup, stale offsets are gone once cleanup finishes, and Next returns the full fresh history. retries also need to preserve any acknowledged fresh writes. cover both durability policies and a successful-sync control.

@krishvishal does that make sense?