HIVE-24145 Fix preemption issues in reducers and file sink operators #1485
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
rbalamohan
reviewed
Sep 11, 2020
| try { | ||
| fs.delete(updaters[i].getUpdatedFilePath(), true); | ||
| } catch (IOException e) { | ||
| e.printStackTrace(); |
rbalamohan
reviewed
Sep 11, 2020
| @@ -284,6 +285,11 @@ public Object process(Node nd, Stack<Node> stack, NodeProcessorCtx procCtx, | |||
| // Create ReduceSink operator | |||
| ReduceSinkOperator rsOp = getReduceSinkOp(partitionPositions, sortPositions, sortOrder, sortNullOrder, | |||
| allRSCols, bucketColumns, numBuckets, fsParent, fsOp.getConf().getWriteType()); | |||
| // we have to make sure not to reorder the child operators as it might cause weird behavior in the tasks at | |||
| // the same level. when there is auto stats gather at the same level as another operation then it might | |||
| // cause unnecessary preemption. Maintaining the order here to avoid such preemption and possible errors | |||
There was a problem hiding this comment.
Plz add TEZ-3296 as ref if possible.
ramesh0201
force-pushed
the
HIVE-24145
branch
from
8b391e4
to
5c188e9
Compare
rbalamohan
approved these changes
Sep 11, 2020
kgyrtkirk
added
tests unstable
tests pending
and removed
tests pending
tests unstable
labels
ramesh0201
force-pushed
the
HIVE-24145
branch
from
5c188e9
to
17d4660
Compare
ramesh0201
force-pushed
the
HIVE-24145
branch
from
17d4660
to
7bb23e7
Compare
kgyrtkirk
added
tests failed
tests pending
tests unstable
tests passed
and removed
tests pending
tests failed
tests unstable
labels
ramesh0201
force-pushed
the
HIVE-24145
branch
from
7bb23e7
to
8a7e0c3
Compare
ramesh0201
force-pushed
the
HIVE-24145
branch
from
8a7e0c3
to
9b1169c
Compare
ramesh0201
force-pushed
the
HIVE-24145
branch
from
9b1169c
to
41e1b0b
Compare
kgyrtkirk
added
tests pending
tests failed
tests unstable
tests passed
and removed
tests failed
tests pending
tests unstable
labels
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Why are the changes needed?
Does this PR introduce any user-facing change?
How was this patch tested?