Search before asking
Fluss version
main (development)
Please describe the bug 馃悶
When remote log is enabled and no new records are appended after the active log segment exceeds table.log.ttl, the periodic local retention cleanup does not roll the active segment.
Steps to reproduce
- Enable remote log with
remote.log.task-interval-duration > 0.
- Configure a short
table.log.ttl.
- Append records to create a non-empty active segment.
- Stop writing and wait until the active segment exceeds the configured TTL.
- Trigger the periodic local log retention cleanup.
Actual behavior
The active segment remains active and is not uploaded to remote storage. It is therefore not eligible for the subsequent local segment cleanup.
The current implementation of LogTablet.deletableExpiredSegments() only scans inactive segments and excludes the active segment.
Expected behavior
The cleanup should roll a non-empty expired active segment when:
- remote log is enabled;
- the active segment is expired by
table.log.ttl;
highWatermark >= logEndOffset.
The rolled segment should not be deleted in the same cleanup pass. After it is uploaded to remote storage, a later cleanup pass should remove it according to the existing remote boundary and retention rules.
This behavior is similar to KAFKA-15047.
Solution
- Extend the existing contiguous TTL scan to inspect the active segment.
- Roll the active segment only after preceding inactive segments satisfy the existing TTL and offset boundaries.
- Keep the active segment out of the current deletion list.
- Preserve the existing
remoteLogEndOffset, minRetainOffset, high-watermark, and empty-active-segment behavior.
- Add regression coverage in
TieredLocalSegmentTTLTest for:
- active segment roll after TTL expiration;
- high watermark below LEO;
- remote end offset boundaries;
- remote log disabled;
- empty active segment.
This change does not modify any public API, RPC protocol, or storage format.
Potential leader/follower segment boundary differences during timestamp roll are out of scope and will be handled separately.
Are you willing to submit a PR?
Search before asking
Fluss version
main (development)
Please describe the bug 馃悶
When remote log is enabled and no new records are appended after the active log segment exceeds
table.log.ttl, the periodic local retention cleanup does not roll the active segment.Steps to reproduce
remote.log.task-interval-duration > 0.table.log.ttl.Actual behavior
The active segment remains active and is not uploaded to remote storage. It is therefore not eligible for the subsequent local segment cleanup.
The current implementation of
LogTablet.deletableExpiredSegments()only scans inactive segments and excludes the active segment.Expected behavior
The cleanup should roll a non-empty expired active segment when:
table.log.ttl;highWatermark >= logEndOffset.The rolled segment should not be deleted in the same cleanup pass. After it is uploaded to remote storage, a later cleanup pass should remove it according to the existing remote boundary and retention rules.
This behavior is similar to KAFKA-15047.
Solution
remoteLogEndOffset,minRetainOffset, high-watermark, and empty-active-segment behavior.TieredLocalSegmentTTLTestfor:This change does not modify any public API, RPC protocol, or storage format.
Potential leader/follower segment boundary differences during timestamp roll are out of scope and will be handled separately.
Are you willing to submit a PR?