Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FLINK-35528][task] Skip execution of interruptible mails when yielding #24904

Merged
merged 1 commit into from
Jun 12, 2024

Conversation

pnowojski
Copy link
Contributor

This PR depends on #24895 . Only the last commit should be reviewed here.

When operators are yielding, for example waiting for async state access to complete before a checkpoint, it would be beneficial to not execute interruptible mails. Otherwise continuation mail for firing timers would be continuously re-enqeueed. To achieve that MailboxExecutor must be aware which mails are interruptible.

The easiest way to achieve this is to set MIN_PRIORITY for interruptible mails.

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): (yes / no)
  • The public API, i.e., is any changed class annotated with @Public(Evolving): (yes / no)
  • The serializers: (yes / no / don't know)
  • The runtime per-record code paths (performance sensitive): (yes / no / don't know)
  • Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / no / don't know)
  • The S3 file system connector: (yes / no / don't know)

Documentation

  • Does this pull request introduce a new feature? (yes / no)
  • If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

@flinkbot
Copy link
Collaborator

flinkbot commented Jun 6, 2024

CI report:

Bot commands The @flinkbot bot supports the following commands:
  • @flinkbot run azure re-run the last Azure build

Copy link
Contributor

@rkhachatryan rkhachatryan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM in general, I have some minor comments about the implementation, PTAL.

Copy link
Contributor

@Zakelly Zakelly left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR. Overall looks good. I left some comments below and under your previous conversations :D

@pnowojski pnowojski force-pushed the f35528 branch 2 times, most recently from 7a0735d to dd831d2 Compare June 10, 2024 09:07
@pnowojski pnowojski requested a review from Zakelly June 10, 2024 09:09
@pnowojski pnowojski force-pushed the f35528 branch 2 times, most recently from e9b1560 to a547a9a Compare June 10, 2024 12:42
Copy link
Contributor

@rkhachatryan rkhachatryan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for updating the PR, LGTM
(except for docs/layouts/shortcodes/generated/execution_checkpointing_configuration.html)

@pnowojski pnowojski force-pushed the f35528 branch 2 times, most recently from 13db505 to 13448db Compare June 11, 2024 07:08
When operators are yielding, for example waiting for async state access to complete before a checkpoint, it would be beneficial to not execute interruptible mails. Otherwise continuation mail for firing timers would be continuously re-enqeueed. To achieve that MailboxExecutor must be aware which mails are interruptible.

The easiest way to achieve this is to set MIN_PRIORITY for interruptible mails.
@pnowojski pnowojski merged commit 77cd1cd into apache:master Jun 12, 2024
@pnowojski pnowojski deleted the f35528 branch June 12, 2024 12:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants