[BEAM-5817] Add Nexmark SqlBoundedSideInputJoin #7114
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
R: @akedin @apilloud @xumingmin Minor tweaks to BeamJoinRel and then it "just works" which is cool! |
kennknowles
commented
Nov 21, 2018
| @@ -198,6 +195,9 @@ private boolean isSideInputJoin() { | |||
| if ((leftRows.isBounded() == PCollection.IsBounded.BOUNDED | |||
| && rightRows.isBounded() == PCollection.IsBounded.BOUNDED) | |||
| || (leftRows.isBounded() == UNBOUNDED && rightRows.isBounded() == UNBOUNDED)) { | |||
| verifySupportedTrigger(leftRows); | |||
There was a problem hiding this comment.
@akedin I think you added this - can you double check my thinking that it is safe to move into here?
There was a problem hiding this comment.
I think so, this is the CoGBK path where we care about the windowing+triggering. The other path is side-input where it should work anyhow, I think
| @@ -76,6 +76,6 @@ public BoundedSideInputJoinModel(NexmarkConfiguration configuration) { | |||
|
|
|||
| @Override | |||
| protected Collection<String> toCollection(Iterator<TimestampedValue<Bid>> itr) { | |||
| return toValueTimestamp(itr); | |||
| return toValue(itr); | |||
There was a problem hiding this comment.
@apilloud I think you did the most on these models. In SQL the timestamp itself is not really observable. The next GROUP BY will have some windowing expression (or not) and the timestamp is specified according to the arguments to the windowing function.
kennknowles
force-pushed
the
join-sideinput
branch
from
9e8f59b
to
49a3695
Compare
|
Run Direct Runner Nexmark Tests |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This adds a SQL variant of the bounded side input join query to the Nexmark suite. It is in the global window. It is up to the SQL optimizer to always find a side input join plan, since otherwise this pipeline needs windowing added in order to work.
Follow this checklist to help us incorporate your contribution quickly and easily:
[BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replaceBEAM-XXXwith the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.It will help us expedite review of your Pull Request if you tag someone (e.g.
@username) to look at it.Post-Commit Tests Status (on master branch)