Skip to content

[server] Support time-based log segment rolling - #3896

Open
fxbing wants to merge 1 commit into
apache:mainfrom
fxbing:feature/20260805-segment-time-roll
Open

[server] Support time-based log segment rolling#3896
fxbing wants to merge 1 commit into
apache:mainfrom
fxbing:feature/20260805-segment-time-roll

Conversation

@fxbing

@fxbing fxbing commented Aug 8, 2026

Copy link
Copy Markdown
Contributor

Purpose

Linked issue: close #3833

Fluss currently rolls log segments only when size, index, or relative-offset limits are reached. For low-throughput buckets, this can leave data in the active segment until another size-based condition is met.

This PR adds optional, append-triggered time-based segment rolling based on record-batch commit timestamps. An idle segment is not actively rolled; the next append triggers the check.

Brief change log

  • Add the static server options log.segment.max-time and log.segment.max-time-jitter, both disabled by default.
  • Reuse the existing LogTablet.maybeRoll and LogSegment.shouldRoll path to roll non-empty segments when their commit-time span exceeds the configured threshold.
  • Use replicated record-batch commitTimestamp values instead of the replica's local wall clock.
  • Cache and recover the first batch timestamp across reopen, recovery, and truncation.
  • Sample optional jitter per segment to avoid synchronized rolling across buckets.
  • Preserve existing size, index, and relative-offset rolling behavior.

Per-segment jitter means physical segment boundaries are not required to match across replicas. The replicated timestamp avoids local clock dependence, while overlapping remote segment boundaries rely on the Manifest V2 support introduced by #3822.

Tests

  • Added configuration validation coverage for defaults, sub-millisecond values, overflow, invalid jitter combinations, and minimum accepted values.
  • Added LogSegmentTest coverage for threshold boundaries, jitter, reopen, recovery, truncation, and unavailable timestamps.
  • Added LogTabletTest coverage for append-triggered rolling and replicated commit timestamps.
  • Added LogLoaderTest coverage for time-based rolling after recovery.
  • Ran:
    • FlussConfigUtilsTest
    • LogSegmentTest
    • LogTabletTest
    • LogLoaderTest
    • ./mvnw -q -pl fluss-common,fluss-server spotless:check
    • ./mvnw -q -pl fluss-common,fluss-server -am -DskipTests validate
    • git diff --check

API and Format

Adds two static server configuration options. There are no public API, RPC, or storage-format changes.

Documentation

Updated the server configuration reference for the new options and their append-triggered semantics.

Generative AI disclosure

Yes. OpenAI Codex was used to assist with research, implementation, testing, and review.

- roll non-empty segments from replicated commit timestamps with optional jitter
- validate duration settings and preserve first-batch timestamps across recovery
- cover configuration, rolling, recovery, and replica consistency
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[server] Support time-based log segment rolling (segment.ms)

1 participant