Adds support for using primitive types in joins. #4132
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Adds support for using primitive types in joins. BREAKING CHANGE: Some existing joins may now fail and the type of `ROWKEY` in the result schema of joins may have changed. When `ROWKEY` was always a `STRING` it was possible to join an `INTEGER` column with a `BIGINT` column. This is no longer the case. A `JOIN` requires the join columns to be of the same type. (See confluentinc#4130 which tracks adding support for being able to `CAST` join criteria). Where joining on two `INT` columns would previously have resulted in a schema containing `ROWKEY STRING KEY`, it would not result in `ROWKEY INT KEY`.
big-andy-coates
commented
Dec 13, 2019
Comment on lines
-255
to
-285
| final Optional<Column> keyColumn = schemaKStream | ||
| .getKeyField() | ||
| .resolve(schemaKStream.getSchema()); | ||
|
|
||
| final ColumnRef rowKey = ColumnRef.of( | ||
| tableName, | ||
| SchemaUtil.ROWKEY_NAME | ||
| ); | ||
|
|
||
| final boolean namesMatch = keyColumn | ||
| .map(field -> field.ref().equals(joinFieldName)) | ||
| .orElse(false); | ||
|
|
||
| if (namesMatch || joinFieldName.equals(rowKey)) { | ||
| return (SchemaKTable) schemaKStream; | ||
| } | ||
|
|
||
| if (!keyColumn.isPresent()) { | ||
| throw new KsqlException( | ||
| "Source table (" + tableName.name() + ") has no key column defined. " | ||
| + "Only 'ROWKEY' is supported in the join criteria." | ||
| ); | ||
| } | ||
|
|
||
| throw new KsqlException( | ||
| "Source table (" + tableName.toString(FormatOptions.noEscape()) + ") key column (" | ||
| + keyColumn.get().ref().toString(FormatOptions.noEscape()) + ") " | ||
| + "is not the column used in the join criteria (" | ||
| + joinFieldName.toString(FormatOptions.noEscape()) + "). " | ||
| + "Only the table's key column or 'ROWKEY' is supported in the join criteria." | ||
| ); |
There was a problem hiding this comment.
These checks are now done from the constructor.
2 tasks
vpapavas
approved these changes
Dec 16, 2019
There was a problem hiding this comment.
Hey @big-andy-coates ! I think it LGTM.
A couple of questions:
- Do we want to test with other data types as well besides
BIGINT? - What is the decision regarding casting? Currently, it fails if the columns are not of the same type.
I will leave it up to you if we need another +1 from someone more experienced.
vpapavas
reviewed
Dec 16, 2019
| @@ -421,33 +413,6 @@ public void testSelectWithExpression() { | |||
|
|
|||
| } | |||
|
|
|||
| @Test | |||
There was a problem hiding this comment.
Also, why did you remove these tests?
There was a problem hiding this comment.
Because there are QTT tests that cover the same thing.
agavra
approved these changes
Dec 16, 2019
ksql-engine/src/main/java/io/confluent/ksql/planner/plan/JoinNode.java
Outdated
Show resolved
Hide resolved
@vpapavas - I think this is OK for now, I think it makes sense to require explicit casting for non-equivalent types of fields. |
I'll ensure that's done.
There's an issue tracking that work: #4130 Thanks for the review! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Fixes: #4094
BREAKING CHANGE: Some existing joins may now fail and the type of
ROWKEYin the result schema of joins may have changed.When
ROWKEYwas always aSTRINGit was possible to join anINTEGERcolumn with aBIGINTcolumn. This is no longer the case. AJOINrequires the join columns to be of the same type. (See #4130 which tracks adding support for being able toCASTjoin criteria).Where joining on two
INTcolumns would previously have resulted in a schema containingROWKEY STRING KEY, it would not result inROWKEY INT KEY.NOTE: a fair few tests needed to be fixed as they were now doing invalid joins or expecting a STRING key. I also removed some tests that were failing as they duplicated existing QTT tests.
Testing done
usual
Reviewer checklist