New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[FLINK-18811] Pick another tmpDir if an IOException occurs when creating spill file #13718
Conversation
Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community Automated ChecksLast check on commit 926247a (Wed Oct 21 08:24:06 UTC 2020) Warnings:
Mention the bot in a comment to re-run the automated checks. Review Progress
Please see the Pull Request Review Guide for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commandsThe @flinkbot bot supports the following commands:
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the contribution @yuchuanchen :) I've left a couple of smaller comments.
...ime/src/main/java/org/apache/flink/runtime/io/network/api/serialization/SpanningWrapper.java
Show resolved
Hide resolved
...ime/src/main/java/org/apache/flink/runtime/io/network/api/serialization/SpanningWrapper.java
Outdated
Show resolved
Hide resolved
…y pick a starting directory index and then per each attempt increase it by one
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the change @yuchuanchen, LGTM.
Could you squash the commits together so I can merge them? (If I squashed the commits from the github ui, it would override the authorship of the commit :( ). Maybe with a commit message:
[FLINK-18811][network] Pick another tmpDir if an IOException occurs when creating spill file
Hi @pnowojski, I squashed the commits and opened a new pull request. |
Thanks @yuchuanchen . Note for the future, you didn't have to open new PR :) It would be fine to update this one. |
Thanks @pnowojski |
What is the purpose of the change
Pick another disk(tmpDir) for Spilling Channel, rather than throw an IOException, which causes flink job restart over and over again.
Brief change log
Verifying this change
This change is already covered by existing tests, such as SpanningWrapperTest.
Does this pull request potentially affect one of the following parts:
@Public(Evolving)
: (no)Documentation