Move statement re-write logic into engine #3400
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
KSQL uses a statement rewritter to convert STRUCT field dereferences into `FETCH_FIELD_FROM_STRUCT` function calls. The rewrite is done for `Query`, `C*AS` and `INSERT INTO` statements, but was missing `EXPLAIN`. It is valid to execute `EXPLAIN` on a query string, e.g. `EXPLAIN SELECT address->street FROM Y;`. It is therefore important that the query within the `EXPLAIN` statement has the same rewrites applied to it as would be applied should the statement be executed directly.
We have to statement re-writers. These currently reside in the parser, but more correctly belong in the engine. First, we have `StatementRewriteForStruct`, which searches out and replaces any `DereferencedExpression` with a special UDF for accessing fields in structs. Previously, this re-write happened in the parser. This PR sees this logic move into the engine. After all, it's an implementation of the engine that we handle struct field access via a UDF. Second, we have `StatementRewriteForRowtime` which looks to allow comparisons between rowtime and strings, given we don't have a datatime type yet. This is already only used in the engine. So moving the type itself to the engine seems sensible. While moving, the API to these types was cleaned up. The `if(statement needs rewrite)` check has been moved inside the re-write class and the class has been refactored to make it more suitable for dependency injection.
# Conflicts: # ksql-engine/src/main/java/io/confluent/ksql/engine/rewrite/StatementRewriteForStruct.java # ksql-engine/src/main/java/io/confluent/ksql/engine/rewrite/StatementRewriter.java # ksql-parser/src/test/java/io/confluent/ksql/parser/rewrite/StatementRewriteForStructTest.java
rodesai
approved these changes
Sep 23, 2019
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
We have to statement re-writers. These currently reside in the parser, but more correctly belong in the engine.
First, we have
StatementRewriteForStruct, which searches out and replaces anyDereferencedExpressionwith a special UDF for accessing fields in structs.Previously, this re-write happened in the parser. This PR sees this logic move into the engine. After all, it's an implementation of the engine that we handle struct field access via a UDF.
Second, we have
StatementRewriteForRowtimewhich looks to allow comparisons between rowtime and strings, given we don't have a datatime type yet.This is already only used in the engine. So moving the type itself to the engine seems sensible.
While moving, the API to these types was cleaned up. The
if(statement needs rewrite)check has been moved inside the re-write class and the class has been refactored to make it more suitable for dependency injection.Testing done
usual.
Reviewer checklist