[BEAM-4076] Use beam join api in sql #11041
Conversation
reuvenlax
force-pushed
the
use_beam_join_api_in_sql
branch
from
1e950d3
to
80de858
Compare
|
@kennknowles This PR still needs some comments and unit tests, but what do you think? It pulls most of the complexity of the join transforms into the generic schema Join transform. A lot of code is deleted. Could've deleted even more code, except it's being used by "lookup" joins. |
|
A quick summary: As part of this, several bugs and gaps were discovered in the schema APIs. One was fixed in a previous PR, and the rest are in this PR. This also adds "broadcast" join capability to the schema Join API. This replaces the side-input join functionality that previously was implemented in SQL. Lookup join remains in SQL. I'm not sure it's worth (i.e. that it's a general-enough use case) trying to pull this into the core Beam transforms. |
|
Run Java PreCommit |
|
Run SQL Postcommit |
reuvenlax
changed the title
|
@apilloud Added a richer set of unit tests and a JIRA. Will merge once green. |
|
Run SQL Postcommit |
| if (joinType == JoinRelType.LEFT) { | ||
| context.output(combineTwoRowsIntoOne(leftRow, rightNullRow, swap, schema)); | ||
| } | ||
| private static FieldAccessDescriptor getJoinColumn( |
There was a problem hiding this comment.
This actually hardcodes a bad assumption a bit further than it was before: that the join is only on columns. We want to move in the other direction, and allow join conditions to be more general RexNodes, many of which still work for CoGBK and side input lookup joins. This is BEAM-6112.
There was a problem hiding this comment.
I noticed this, because in the current codebase on master we could inline and delete SerializableRexNode entirely. So I went looking for how we encoding a full expression to be joined on. I didn't see rules that precomputed all of them (in which case input refs would suffice).
Please add a meaningful description for your change here
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
R: @username).[BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replaceBEAM-XXXwith the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.CHANGES.mdwith noteworthy changes.See the Contributor Guide for more tips on how to make review process smoother.
Post-Commit Tests Status (on master branch)
Pre-Commit Tests Status (on master branch)
See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.