[server] Support time-based log segment rolling - #3896
Open
fxbing wants to merge 1 commit into
Open
Conversation
- roll non-empty segments from replicated commit timestamps with optional jitter - validate duration settings and preserve first-batch timestamps across recovery - cover configuration, rolling, recovery, and replica consistency
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Linked issue: close #3833
Fluss currently rolls log segments only when size, index, or relative-offset limits are reached. For low-throughput buckets, this can leave data in the active segment until another size-based condition is met.
This PR adds optional, append-triggered time-based segment rolling based on record-batch commit timestamps. An idle segment is not actively rolled; the next append triggers the check.
Brief change log
log.segment.max-timeandlog.segment.max-time-jitter, both disabled by default.LogTablet.maybeRollandLogSegment.shouldRollpath to roll non-empty segments when their commit-time span exceeds the configured threshold.commitTimestampvalues instead of the replica's local wall clock.Per-segment jitter means physical segment boundaries are not required to match across replicas. The replicated timestamp avoids local clock dependence, while overlapping remote segment boundaries rely on the Manifest V2 support introduced by #3822.
Tests
LogSegmentTestcoverage for threshold boundaries, jitter, reopen, recovery, truncation, and unavailable timestamps.LogTabletTestcoverage for append-triggered rolling and replicated commit timestamps.LogLoaderTestcoverage for time-based rolling after recovery.FlussConfigUtilsTestLogSegmentTestLogTabletTestLogLoaderTest./mvnw -q -pl fluss-common,fluss-server spotless:check./mvnw -q -pl fluss-common,fluss-server -am -DskipTests validategit diff --checkAPI and Format
Adds two static server configuration options. There are no public API, RPC, or storage-format changes.
Documentation
Updated the server configuration reference for the new options and their append-triggered semantics.
Generative AI disclosure
Yes. OpenAI Codex was used to assist with research, implementation, testing, and review.