Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FLINK-34986][Runtime/State] Basic framework of async execution for state #24614

Merged
merged 3 commits into from
Apr 11, 2024
Merged

[FLINK-34986][Runtime/State] Basic framework of async execution for state #24614

merged 3 commits into from
Apr 11, 2024

Conversation

Zakelly
Copy link
Contributor

@Zakelly Zakelly commented Apr 3, 2024 •
edited
Loading

What is the purpose of the change

This PR ship the core part of FLIP-425, including the basic execution logic of AsyncExecutionController.

Brief change log

  • AsyncExecutionController and other components around it.
  • RecordContext and reference counting mechanism.
  • Basic implementation of KeyAccountingUnit.

Verifying this change

  • Added unit tests under org.apache.flink.runtime.asyncprocessing

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): no
  • The public API, i.e., is any changed class annotated with @Public(Evolving): no
  • The serializers: no
  • The runtime per-record code paths (performance sensitive): yes (introduce new)
  • Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
  • The S3 file system connector: no

Documentation

  • Does this pull request introduce a new feature? yes
  • If yes, how is the feature documented? not documented

@Zakelly Zakelly changed the title F34986 [FLINK-34986][Runtime/State] Basic framework of async execution for state Apr 3, 2024
@flinkbot
Copy link
Collaborator

flinkbot commented Apr 4, 2024 •
edited
Loading

CI report:

Bot commands The @flinkbot bot supports the following commands:
  • @flinkbot run azure re-run the last Azure build

@Zakelly
Copy link
Contributor Author

Zakelly commented Apr 7, 2024

Rebased and pushed again.

@Zakelly
Copy link
Contributor Author

Zakelly commented Apr 7, 2024

@fredia @masteryhx would you please take a look?

fredia
fredia requested changes Apr 7, 2024
View reviewed changes
Copy link
Contributor

@fredia fredia left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Zakelly Thanks for the PR, I left some comments first. And I haven't completely reviewed the test, some follow-up questions may be raised.

...k-runtime/src/main/java/org/apache/flink/runtime/asyncprocessing/ContextStateFutureImpl.java Outdated Show resolved Hide resolved
...k-runtime/src/main/java/org/apache/flink/runtime/asyncprocessing/ContextStateFutureImpl.java Show resolved Hide resolved
flink-runtime/src/main/java/org/apache/flink/runtime/asyncprocessing/StateFutureFactory.java Show resolved Hide resolved
flink-runtime/src/main/java/org/apache/flink/runtime/asyncprocessing/StateRequest.java Show resolved Hide resolved
@Zakelly
Copy link
Contributor Author

Zakelly commented Apr 7, 2024

@fredia thank for your suggestions!

fredia
fredia approved these changes Apr 7, 2024
View reviewed changes
Copy link
Contributor

@fredia fredia left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Zakelly Thanks for the update, overall LGTM.

...ime/src/test/java/org/apache/flink/runtime/asyncprocessing/AsyncExecutionControllerTest.java Show resolved Hide resolved
...ntime/src/test/java/org/apache/flink/runtime/asyncprocessing/ContextStateFutureImplTest.java Show resolved Hide resolved
@fredia fredia requested a review from curcur April 7, 2024 10:11
masteryhx
masteryhx reviewed Apr 8, 2024
View reviewed changes
Copy link
Contributor

@masteryhx masteryhx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR.
PTAL my comments.

flink-runtime/src/main/java/org/apache/flink/runtime/asyncprocessing/StateRequest.java Outdated Show resolved Hide resolved
flink-runtime/src/main/java/org/apache/flink/runtime/asyncprocessing/StateRequest.java Outdated Show resolved Hide resolved
flink-runtime/src/main/java/org/apache/flink/runtime/asyncprocessing/StateRequest.java Outdated Show resolved Hide resolved
flink-runtime/src/main/java/org/apache/flink/runtime/asyncprocessing/StateExecutor.java Show resolved Hide resolved
flink-runtime/src/main/java/org/apache/flink/runtime/asyncprocessing/KeyAccountingUnit.java Show resolved Hide resolved
...runtime/src/main/java/org/apache/flink/runtime/asyncprocessing/AsyncExecutionController.java Show resolved Hide resolved
...ime/src/test/java/org/apache/flink/runtime/asyncprocessing/AsyncExecutionControllerTest.java Outdated Show resolved Hide resolved
@Zakelly Zakelly force-pushed the f34986 branch 2 times, most recently from 88f2e33 to 595a10c Compare April 8, 2024 06:09
@Zakelly
Copy link
Contributor Author

Zakelly commented Apr 8, 2024

@masteryhx Thanks for your comments! Applied some suggestions, PTAL thanks.

@fredia fredia mentioned this pull request Apr 8, 2024
Merged
@Zakelly Zakelly force-pushed the f34986 branch 2 times, most recently from 42609e2 to db14529 Compare April 9, 2024 05:44
@Zakelly
Copy link
Contributor Author

Zakelly commented Apr 9, 2024

Rebased master. Extracting StateRequestType with more detailed enum entry definition.

@Zakelly Zakelly force-pushed the f34986 branch 3 times, most recently from 3bdab34 to ab070bd Compare April 10, 2024 07:06
yunfengzhou-hub
yunfengzhou-hub reviewed Apr 10, 2024
View reviewed changes
Copy link
Contributor

@yunfengzhou-hub yunfengzhou-hub left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR. Left some comments as below.

flink-runtime/src/main/java/org/apache/flink/runtime/asyncprocessing/KeyAccountingUnit.java Outdated Show resolved Hide resolved
...runtime/src/main/java/org/apache/flink/runtime/asyncprocessing/AsyncExecutionController.java
* @return the state future.
*/
public <IN, OUT> InternalStateFuture<OUT> handleRequest(
@Nullable State state, StateRequestType type, @Nullable IN payload) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you please remind me of the cases when a null state would be passed to AEC?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added some description in javadoc. This is for the strict order of 'processElement' for same-key records.

flink-runtime/src/main/java/org/apache/flink/runtime/asyncprocessing/StateFutureFactory.java Outdated Show resolved Hide resolved
...runtime/src/main/java/org/apache/flink/runtime/asyncprocessing/AsyncExecutionController.java Outdated Show resolved Hide resolved
@Zakelly
Copy link
Contributor Author

Zakelly commented Apr 10, 2024

@flinkbot run azure

yunfengzhou-hub
yunfengzhou-hub approved these changes Apr 11, 2024
View reviewed changes
Copy link
Contributor

@yunfengzhou-hub yunfengzhou-hub left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the update! LGTM.

@fredia fredia merged commit ec2a140 into apache:master Apr 11, 2024
@Zakelly Zakelly deleted the f34986 branch April 11, 2024 02:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jiexray jiexray jiexray left review comments

@fredia fredia fredia approved these changes

@ljz2051 ljz2051 ljz2051 left review comments

@masteryhx masteryhx masteryhx left review comments

@yunfengzhou-hub yunfengzhou-hub yunfengzhou-hub approved these changes

@curcur curcur Awaiting requested review from curcur

Assignees
No one assigned
Labels
component=Runtime/StateBackends component=Runtime/Task
Projects
None yet
Milestone
No milestone
7 participants
@Zakelly @flinkbot @jiexray @fredia @ljz2051 @masteryhx @yunfengzhou-hub