Skip to content

[lake] Re-initial tiering splits for tables if reader failover happen #1984

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
luoyuxia merged 5 commits into from
Jan 15, 2026
Merged

[lake] Re-initial tiering splits for tables if reader failover happen #1984

luoyuxia merged 5 commits into from
Jan 15, 2026

Conversation

@beryllw
Copy link
Contributor

@beryllw beryllw commented Nov 16, 2025

Purpose

Linked issue: close #1929

Brief change log

Tests

API and Format

Documentation

@beryllw beryllw marked this pull request as draft November 16, 2025 03:28
@beryllw beryllw marked this pull request as ready for review November 25, 2025 06:46
@luoyuxia luoyuxia requested a review from Copilot November 25, 2025 09:27
Copilot AI reviewed Nov 25, 2025
View reviewed changes
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR implements failover handling for the lake tiering feature by detecting reader failover through attempt number tracking. It also introduces a VALUES lake format for testing purposes.

Key Changes

  • Implements failover detection in TieringSourceEnumerator by tracking reader attempt numbers to reinitialize tiering splits when failover occurs
  • Removes the explicit TieringFailOverEvent mechanism in favor of automatic detection via attempt number monitoring
  • Adds a comprehensive VALUES lake storage implementation for testing tiering functionality with failure injection capabilities

Reviewed changes

Copilot reviewed 21 out of 21 changed files in this pull request and generated 14 comments.

Show a summary per file
File Description
fluss-flink/fluss-flink-common/src/main/java/org/apache/fluss/flink/tiering/source/enumerator/TieringSourceEnumerator.java Critical: Adds failover detection logic but calls non-existent registeredReadersOfAttempts() method on standard Flink API
fluss-flink/fluss-flink-common/src/main/java/org/apache/fluss/flink/tiering/committer/TieringCommitOperator.java Removes explicit failover event sending from setup method
fluss-flink/fluss-flink-common/src/test/java/org/apache/fluss/flink/tiering/source/enumerator/FlussMockSplitEnumeratorContext.java Adds mock context supporting attempt number tracking for tests
fluss-flink/fluss-flink-common/src/test/java/org/apache/fluss/flink/tiering/source/enumerator/TieringSourceEnumeratorTest.java Updates tests to use new mock context and validate failover scenarios
fluss-flink/fluss-flink-common/src/test/java/org/apache/fluss/flink/tiering/TieringFailoverITCase.java Adds end-to-end integration test for tiering failover with write failure injection
fluss-flink/fluss-flink-common/src/test/java/org/apache/fluss/lake/values/* Implements complete VALUES lake storage plugin for testing
fluss-common/src/main/java/org/apache/fluss/metadata/DataLakeFormat.java Adds VALUES data lake format enum value
fluss-common/src/main/java/org/apache/fluss/row/encode/KeyEncoder.java Adds VALUES format support to key encoder factory
fluss-common/src/main/java/org/apache/fluss/bucketing/BucketingFunction.java Adds VALUES format support to bucketing function factory

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

luoyuxia
luoyuxia reviewed Dec 15, 2025
View reviewed changes
Copy link
Contributor

@luoyuxia luoyuxia left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@beryllw Thanks for the pr. Left some comments. PTAL

@beryllw beryllw force-pushed the fix-tiering-failover branch from c026d94 to 71648c2 Compare January 9, 2026 07:51
@beryllw beryllw requested a review from luoyuxia January 12, 2026 02:27
@luoyuxia luoyuxia requested a review from Copilot January 12, 2026 12:43
Copilot AI reviewed Jan 12, 2026
View reviewed changes
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 16 out of 16 changed files in this pull request and generated 10 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

luoyuxia
luoyuxia reviewed Jan 12, 2026
View reviewed changes
Copy link
Contributor

@luoyuxia luoyuxia left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@beryllw Thanks for the pr. Left minor comments. PTAL

@beryllw beryllw requested a review from luoyuxia January 14, 2026 05:36
luoyuxia
luoyuxia approved these changes Jan 14, 2026
View reviewed changes
Copy link
Contributor

@luoyuxia luoyuxia left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@luoyuxia luoyuxia merged commit 2ce8378 into apache:main Jan 15, 2026
6 checks passed
rionmonster pushed a commit to rionmonster/fluss that referenced this pull request Jan 28, 2026
@beryllw @rionmonster
[lake] Re-initial tiering splits for tables if reader failover happen (…
44a1e16 
…apache#1984)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot Copilot left review comments

@luoyuxia luoyuxia luoyuxia approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

TieringEnumerator should re-initial tiering splits for tables if reader failover happen

2 participants

@beryllw @luoyuxia