Port semi-join to WorkProcessorOperator #1119
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
sopel39
reviewed
Jul 15, 2019
| public WorkProcessor.TransformationState<Page> process(Page inputPage) | ||
| { | ||
| if (!this.channelSetFuture.isDone()) { | ||
| return WorkProcessor.TransformationState.blocked(this.channelSetFuture); |
There was a problem hiding this comment.
static import TransformationState
| public WorkProcessor.TransformationState<Page> process(Page inputPage) | ||
| { | ||
| if (!this.channelSetFuture.isDone()) { | ||
| return WorkProcessor.TransformationState.blocked(this.channelSetFuture); |
There was a problem hiding this comment.
technically, single input page will be buffered until future is done. We could account it's memory here
| } | ||
|
|
||
| if (channelSet == null) { | ||
| channelSet = tryGetFutureValue(channelSetFuture).orElse(null); |
There was a problem hiding this comment.
future must be done at this point, so tryGetFutureValue is not needed.
You can use io.airlift.concurrent.MoreFutures#checkSuccess instead
|
|
||
| @Override | ||
| public WorkProcessor.TransformationState<Page> process(Page inputPage) | ||
| { |
There was a problem hiding this comment.
you could short-circut here by checking inputPage==null early. This way we won't wait for future if probe side doesn't contain any data.
| // add the new boolean column to the page | ||
| outputPage = page.appendColumn(blockBuilder.build()); | ||
| } | ||
| if (inputPage != null && channelSet != null) { |
There was a problem hiding this comment.
channelSet cannot be null at this point
sopel39
reviewed
Jul 15, 2019
| @Override | ||
| public WorkProcessor.TransformationState<Page> process(Page inputPage) | ||
| { | ||
| if (!this.channelSetFuture.isDone()) { |
There was a problem hiding this comment.
I would change this if too:
if (channelSet == null) {
if (channelSetFuture.isDone()) {
...
} else {
return WorkProcessor.TransformationState.blocked(this.channelSetFuture);
}
}
sopel39
reviewed
Jul 15, 2019
| @Override | ||
| public WorkProcessor.TransformationState<Page> process(Page inputPage) | ||
| { | ||
| if (!this.channelSetFuture.isDone()) { |
Praveen2112
force-pushed
the
semi_join_processor
branch
from
21cac9f
to
f0b4b35
Compare
sopel39
reviewed
Jul 16, 2019
| } | ||
|
|
||
| @Override | ||
| public WorkProcessor.TransformationState<Page> process(Page inputPage) |
There was a problem hiding this comment.
static import TransformationState
Praveen2112
force-pushed
the
semi_join_processor
branch
3 times, most recently
from
654277f
to
d9ecd4f
Compare
|
@sopel39 Thanks for your comments. Have applied the comments. |
|
part of: #49 |
sopel39
approved these changes
Jul 17, 2019
|
|
||
| if (channelSet == null) { | ||
| if (!channelSetFuture.isDone()) { | ||
| localMemoryContext.setBytes(inputPage.getSizeInBytes()); |
There was a problem hiding this comment.
nit: you could add comment that this will materialize page, but it shouldn't matter for first page
| } | ||
| if (!contains && channelSet.containsNull()) { | ||
| blockBuilder.appendNull(); | ||
| checkSuccess(channelSetFuture, "Set building failed"); |
There was a problem hiding this comment.
Set building failed -> ChannelSet building failed
| private final Optional<Integer> probeHashChannel; | ||
| private final LocalMemoryContext localMemoryContext; | ||
|
|
||
| private ChannelSet channelSet; |
| } | ||
| } | ||
| // add the new boolean column to the page | ||
| return TransformationState.ofResult(inputPage.appendColumn(blockBuilder.build()), true); |
| this.channelSetFuture = requireNonNull(channelSetFuture, "hashProvider is null").getChannelSet(); | ||
| this.probeJoinChannel = probeJoinChannel; | ||
| this.probeHashChannel = requireNonNull(probeHashChannel, "hashChannel is null"); | ||
| this.localMemoryContext = requireNonNull(aggregatedMemoryContext, "AggregatedMemoryContext is null").newLocalMemoryContext(SemiJoinPages.class.getSimpleName()); |
There was a problem hiding this comment.
AggregatedMemoryContext -> aggregatedMemoryContext
|
minor comments |
Praveen2112
force-pushed
the
semi_join_processor
branch
from
d9ecd4f
to
a12f6cb
Compare
|
@sopel39 Have applied the comments. |
sopel39
reviewed
Jul 17, 2019
| if (channelSet == null) { | ||
| if (!channelSetFuture.isDone()) { | ||
| // This will materialize page but it shouldn't matter for the first page | ||
| localMemoryContext.setBytes(inputPage.getSizeInBytes()); |
There was a problem hiding this comment.
@dain this makes HashSemiJoinOperator keep first input page (to progress probe computations before build future is done). I don't think it's relevant memory usage. What do you think?
There was a problem hiding this comment.
For more context we want to progress upstream operators computations (e.g aggregation) by pulling initial page from probe side when build side future is not done. In current operator model this is also the case. However, in current model computations are performed up to input of next upstream operator. In WorkProcessorOperator model computations are performed up to input of HashSemiJoinOperator.
sopel39
reviewed
Jul 17, 2019
| @@ -11,6 +11,6 @@ node.environment=test | |||
| coordinator=false | |||
There was a problem hiding this comment.
commit message too long. This should go before port commit
| @@ -11,6 +11,6 @@ node.environment=test | |||
| coordinator=false | |||
| http-server.http.port=8081 | |||
| query.max-memory=1GB | |||
| query.max-memory-per-node=512MB | |||
| query.max-memory-per-node=768MB | |||
There was a problem hiding this comment.
sql_tests.testcases.hive_tpch.q18 failed because exceeding node query memory limit. However it was already on the edge with top consuming operators:
Query exceeded per-node user memory limit of 512MB [Allocated: 511.83MB, Delta: 1.69MB,
Top Consumers: {HashBuilderOperator=363.73MB, HashAggregationOperator=144.22MB, PartitionedOutputOperator=42.01MB}]
HashSemiJoinOperator is not even on top consumers list
There was a problem hiding this comment.
Was this change related to the PR?
There was a problem hiding this comment.
Memory accounting was improved as part of this PR (and other work processor PRs). Also first page is hold until build side is ready. It tipped already borderline mem limit (this operator is not even one of the top consumers)
| if (channelSet == null) { | ||
| if (!channelSetFuture.isDone()) { | ||
| // This will materialize page but it shouldn't matter for the first page | ||
| localMemoryContext.setBytes(inputPage.getSizeInBytes()); |
There was a problem hiding this comment.
For more context we want to progress upstream operators computations (e.g aggregation) by pulling initial page from probe side when build side future is not done. In current operator model this is also the case. However, in current model computations are performed up to input of next upstream operator. In WorkProcessorOperator model computations are performed up to input of HashSemiJoinOperator.
Praveen2112
force-pushed
the
semi_join_processor
branch
from
d65303e
to
437938c
Compare
|
Some synthetic experiment: |
Praveen2112
force-pushed
the
semi_join_processor
branch
from
437938c
to
d6b8666
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.