Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FLINK-34516] Move CheckpointingMode to flink-core #24381

Closed
wants to merge 6 commits into from

Conversation

Zakelly
Copy link
Contributor

@Zakelly Zakelly commented Feb 26, 2024

What is the purpose of the change

In FLIP-406, we want to merge all options in ExecutionCheckpointingOptions to CheckpointingOptions, which depends on the CheckpointingMode being moved to flink-core as well. This PR introduce a new CheckpointingMode in flink-core, and deprecate the old one as well as corresponding user-facing APIs.

Brief change log

  • introduce a new CheckpointingMode in flink-core, providing new APIs get/setConsistencyMode.
  • deprecate the old CheckpointingMode in flink-streaming-java as well as the APIs.
  • Migrate all necessary usage to new APIs.

Verifying this change

This change is already covered by new introduced UTs.

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): (yes / no)
  • The public API, i.e., is any changed class annotated with @Public(Evolving): (yes / no)
  • The serializers: (yes / no / don't know)
  • The runtime per-record code paths (performance sensitive): (yes / no / don't know)
  • Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / no / don't know)
  • The S3 file system connector: (yes / no / don't know)

Documentation

  • Does this pull request introduce a new feature? (yes / no)
  • If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

@flinkbot
Copy link
Collaborator

flinkbot commented Feb 26, 2024

CI report:

Bot commands The @flinkbot bot supports the following commands:
  • @flinkbot run azure re-run the last Azure build

@Zakelly Zakelly changed the title [DRAFT][FLINK-34516] Move CheckpointingMode to flink-core [FLINK-34516] Move CheckpointingMode to flink-core Mar 5, 2024
Copy link
Contributor

@masteryhx masteryhx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR, PTAL my comments.
BTW, let's wait until there is a no rejection in the discussion mail thread.

*/
@Deprecated
public CheckpointingMode getCheckpointingMode() {
return configuration.get(ExecutionCheckpointingOptions.CHECKPOINTING_MODE);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IIUC, we should replace all internal usages of original CheckpointMode and docs with new one except for the user-facing interface, right?
I'd suggest to transform to the new CheckpointMode for all original public interfaces, and use the new one for other inner places.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IIUC, we should replace all internal usages of original CheckpointMode and docs with new one except for the user-facing interface, right?

Yeah, and this will be addressed by another PR.

I'd suggest to transform to the new CheckpointMode for all original public interfaces, and use the new one for other inner places.

Well, I suggest we make this easier since the public API and the option will be removed at the same time, no more complex conversion needed, WDYT?

@Zakelly
Copy link
Contributor Author

Zakelly commented Mar 13, 2024

@flinkbot run azure

@Zakelly Zakelly force-pushed the f34516 branch 4 times, most recently from ccc1f1c to 315e2f4 Compare March 14, 2024 03:55
@Zakelly
Copy link
Contributor Author

Zakelly commented Mar 14, 2024

@masteryhx I have transformed to the new CheckpointMode everywhere, and rebased master. Would you please take a look?

@Zakelly
Copy link
Contributor Author

Zakelly commented Mar 14, 2024

Rebased for conflicts....

@Zakelly Zakelly force-pushed the f34516 branch 2 times, most recently from ccba6d5 to b27cdbb Compare March 14, 2024 15:50
@Zakelly
Copy link
Contributor Author

Zakelly commented Mar 15, 2024

@flinkbot run azure

1 similar comment
@Zakelly
Copy link
Contributor Author

Zakelly commented Mar 15, 2024

@flinkbot run azure

Copy link
Contributor

@masteryhx masteryhx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the update. Overall LGTM.
Just left a comment, PTAL.

*
* @return The checkpointing mode.
*/
public CheckpointingMode getConsistencyMode() {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CheckpointingConsistencyMode looks good to me.
Could we also update the config name and the class name since it's also a public API same as config ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@masteryhx Personally, I'd be fine with name the enum CheckpointingConsistencyMode, but that's a breaking change and needs more discussion. My current plan is to keep the CheckpointingMode here, and provide API with 'Consistency' in its name just for avoiding conflicts. We will reuse the original API name later in 2.0, WDYT?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just have a minor concern that users may lost about CheckpointingMode and ConsistencyMode.
How about renaming this to CheckpointingConsistencyMode and adding some descriptions about the relationship between them ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd change the API name get/setConsistencyMode to get/setCheckpointingConsistencyMode and add some description showing the relationship between the name CheckpointingConsistencyMode and CheckpointingMode. But I'd suggest keep CheckpointingMode as it is since it is a first class citizen of Flink and widely accepted by users. WDYT?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good to me.

Copy link
Contributor

@masteryhx masteryhx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the update.
LGTM.

@masteryhx
Copy link
Contributor

merged 8fac804...7794591 into master

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants