Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ensure single task reads single partition in write stage #12721

Merged
merged 3 commits into from Jun 8, 2022

Conversation

losipiuk
Copy link
Member

@losipiuk losipiuk commented Jun 7, 2022

Documentation

(x) No documentation is needed.
( ) Sufficient documentation is included in this PR.
( ) Documentation PR is available with #prnumber.
( ) Documentation issue #issuenumber is filed, and can be handled later.

The thing which could be documented is config/sessino property, yet I think it is not needed.
I added it merely as a kill switch, and changing default should not be needed.

Release notes

( ) No release notes entries required.
(x) Release notes entries required with the following suggested text:

# General
* Increase high granularity of tasks for stages which are writing table data, when fault-tolerant execution is enabled (`retry-policy` is set to `TASK`).
  That is to reduce chance of hitting `Exceeded limit of N open writers for partitions` errors. ({issue}`12721`)

@@ -415,11 +424,36 @@ public static HashDistributionTaskSource create(
bucketNodeMap,
fragment.getPartitioning().getConnectorId(),
targetPartitionSplitWeight,
targetPartitionSourceSize,
(isWriteFragment && preserveInputPartitionsInWriteStage) ? DataSize.of(0, BYTE) : targetPartitionSourceSize,
Copy link
Member

@linzebing linzebing Jun 7, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shall we do:

(preserveInputPartitionsInWriteStage && isWriteFragment(fragment))

So we can shortcut the logic and avoid traversing the plan tree is flag is not set?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah - makes sense

@arhimondr arhimondr merged commit e196b37 into trinodb:master Jun 8, 2022
@github-actions github-actions bot added this to the 385 milestone Jun 8, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Development

Successfully merging this pull request may close these issues.

None yet

3 participants