Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FLINK-34456][configuration]Move all checkpoint-related options into CheckpointingOptions #24374

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

[FLINK-34456][configuration]Move all checkpoint-related options into CheckpointingOptions #24374

wants to merge 1 commit into from
+475 −154

Conversation

spoon-lz
Copy link
Contributor

@spoon-lz spoon-lz commented Feb 23, 2024 •
edited

What is the purpose of the change

According to the FLIP-406,Move all checkpoint-related options that are out of CheckpointingOptions into CheckpointingOptions. Deprecate the original ones.

Brief change log

  • Move all configurations under org.apache.flink.streaming.api.environment.ExecutionCheckpointingOptions into org.apache.flink.configuration.CheckpointingOptions
  • Mark ExecutionCheckpointingOptions as deprecated
  • Move org.apache.flink.streaming.api.CheckpointingMode to org.apache.flink.configuration.CheckpointingMode
  • Split ExternalizedCheckpointCleanup out of CheckpointConfig and move it to flink-core module

Verifying this change

This change is a minor rework and is already covered by existing tests.

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): (yes / no)
  • The public API, i.e., is any changed class annotated with @Public(Evolving): (yes / no)
  • The serializers: (yes / no / don't know)
  • The runtime per-record code paths (performance sensitive): (yes / no / don't know)
  • Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / no / don't know)
  • The S3 file system connector: (yes / no / don't know)

Documentation

  • Does this pull request introduce a new feature? (yes / no)
  • If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

@spoon-lz
Copy link
Contributor Author

@flinkbot run azure

@flinkbot
Copy link
Collaborator

flinkbot commented Feb 23, 2024 •
edited

CI report:

Bot commands The @flinkbot bot supports the following commands:
  • @flinkbot run azure re-run the last Azure build

@spoon-lz
Copy link
Contributor Author

@flinkbot run azure

@spoon-lz
Copy link
Contributor Author

@flinkbot run azure

2 similar comments
@spoon-lz
Copy link
Contributor Author

@flinkbot run azure

@spoon-lz
Copy link
Contributor Author

@flinkbot run azure

Zakelly
Zakelly reviewed Feb 26, 2024
View reviewed changes
Copy link
Contributor

@Zakelly Zakelly left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @spoon-lz , thanks for your contribution! At first glance of your PR, I would suggest keep the old org.apache.flink.streaming.api.CheckpointingMode and mark as deprecated. However it is required to start a discussion about such deprecation on dev mailing list first. I will start the discussion and work on the deprecation in Flink 1.19 (will soon be released this month).

@spoon-lz
Copy link
Contributor Author

Hi @spoon-lz , thanks for your contribution! At first glance of your PR, I would suggest keep the old org.apache.flink.streaming.api.CheckpointingMode and mark as deprecated. However it is required to start a discussion about such deprecation on dev mailing list first. I will start the discussion and work on the deprecation in Flink 1.19 (will soon be released this month).

If CheckpointingMode is not moved, do we need to copy the same class in the flink-core module? Otherwise, dependency problems will occur.
Or wait until the discussion is over before modifying this PR.

@Zakelly
Copy link
Contributor

Zakelly commented Feb 26, 2024

Hi @spoon-lz , thanks for your contribution! At first glance of your PR, I would suggest keep the old org.apache.flink.streaming.api.CheckpointingMode and mark as deprecated. However it is required to start a discussion about such deprecation on dev mailing list first. I will start the discussion and work on the deprecation in Flink 1.19 (will soon be released this month).

If CheckpointingMode is not moved, do we need to copy the same class in the flink-core module? Otherwise, dependency problems will occur. Or wait until the discussion is over before modifying this PR.

I suggest both keep the old one and the new introduced one. All the internal usage will use the new one, while only user-facing APIs leverage the old one.

We could wait the conclusion of the discussion.

@Zakelly
Copy link
Contributor

Zakelly commented Mar 5, 2024

@spoon-lz Sorry for the late reply, seems we come to an agreement in discussion. To be brief, we agree to deprecate the old class and introduce a new one. Corresponding APIs will be changed accordingly.
There is one PR ready for this #24381 . I suggest building your PR on top of this after it is merged. WDYT?

@spoon-lz
Copy link
Contributor Author

spoon-lz commented Mar 6, 2024

@Zakelly I also saw the conclusion of the discussion in the email. I will re-edit this PR after #24381 is merged. Another question is, "Split ExternalizedCheckpointCleanup out of CheckpointConfig and move it to flink-core module". Does this change also need to be discussed?

@Zakelly
Copy link
Contributor

Zakelly commented Mar 6, 2024

@Zakelly I also saw the conclusion of the discussion in the email. I will re-edit this PR after #24381 is merged. Another question is, "Split ExternalizedCheckpointCleanup out of CheckpointConfig and move it to flink-core module". Does this change also need to be discussed?

I think this is much easier since it is only annotated with @\PublicEvolving. I think we can do this in a similar way without another discussion. A seperated PR is also good.

@spoon-lz
Copy link
Contributor Author

spoon-lz commented Mar 6, 2024

@Zakelly A separate PR seems more reasonable, I will submit a separate PR to split ExternalizedCheckpointCleanup

@spoon-lz
Copy link
Contributor Author

@Zakelly There are some references to org.apache.flink.streaming.api.CheckpointingMode in ExecutionCheckpointingOptions. When we move these configurations into CheckpointingOptions, do we need to replace them with org.apache.flink .core.execution.CheckpointingMode?

@Zakelly
Copy link
Contributor

Zakelly commented May 20, 2024

@Zakelly There are some references to org.apache.flink.streaming.api.CheckpointingMode in ExecutionCheckpointingOptions. When we move these configurations into CheckpointingOptions, do we need to replace them with org.apache.flink .core.execution.CheckpointingMode?

@spoon-lz
Firstly, we should keep ExecutionCheckpointingOptions and mark it as deprecated.
Secondly, any new introduced(moved) options should use the new enum class. Right?

@spoon-lz
Copy link
Contributor Author

@flinkbot run azure

@spoon-lz
Copy link
Contributor Author

@Zakelly New code has been submitted.

Zakelly
Zakelly requested changes May 23, 2024
View reviewed changes
Copy link
Contributor

@Zakelly Zakelly left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@spoon-lz Thanks for the update! There seems to be a conflict. And I think you should re-generate the document as flink-docs/README.md says. The documents reference among each other should also be updated if there is any.

flink-core/src/main/java/org/apache/flink/configuration/CheckpointingOptions.java
ConfigOptions.key("execution.checkpointing.mode")
.enumType(CheckpointingMode.class)
.defaultValue(CheckpointingMode.EXACTLY_ONCE)
.withDescription("The checkpointing mode (exactly-once vs. at-least-once).");
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought we don't need to move those deprecated ones.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right, I will delete these deprecated options

@spoon-lz spoon-lz requested a review from Zakelly May 27, 2024 11:39
@spoon-lz
Copy link
Contributor Author

@Zakelly The code has been modified, but I don’t know why. After I force push to the current branch, I can see the modification in my warehouse, but not in this PR. Do I need to submit a new PR?

@masteryhx
Copy link
Contributor

@Zakelly The code has been modified, but I don’t know why. After I force push to the current branch, I can see the modification in my warehouse, but not in this PR. Do I need to submit a new PR?

Hi, Could you check your branch 'lz-CheckpointingOptions' in your repo ? Seems it doesn't include new commits.

@Zakelly
Copy link
Contributor

Zakelly commented May 27, 2024

@spoon-lz I didn't see any new commit on https://github.com/spoon-lz/flink/tree/lz-CheckpointingOptions

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Zakelly Zakelly Awaiting requested review from Zakelly

Assignees
No one assigned
Labels
component=Runtime/Checkpointing
Projects
None yet
Milestone
No milestone
4 participants
@spoon-lz @flinkbot @Zakelly @masteryhx