Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP][SPARK-38432][SQL] Reactor framework so as JDBC dialect could compile filter by self way #35746

Closed
wants to merge 1 commit into from

Conversation

beliefer
Copy link
Contributor

@beliefer beliefer commented Mar 7, 2022

What changes were proposed in this pull request?

Currently, Spark DS V2 could push down filters into JDBC source. However, only the most basic form of filter is supported.
On the other hand, some JDBC source could not compile the filters by themselves way.

This PR reactor the framework so as JDBC dialect could compile expression by self way.
First, The framework translate catalyst expressions to DS V2 filters.
Second, The JDBC dialect could compile DS V2 filters to different SQL syntax.

Why are the changes needed?

Make the framework be more common use.

Does this PR introduce any user-facing change?

'No'.
The feature is not released.

How was this patch tested?

Exists tests.

@github-actions github-actions bot added the SQL label Mar 7, 2022
@cloud-fan
Copy link
Contributor

It seems like we can completely replace v2 Filter with GeneralScalarExpression, as they are both tree-like API to represent catalyst expressions in DS v2.

For better type safety, I think we can make Filter extends GeneralScalarExpression, to indicate that it's a predicate that returns boolean.

That said, the V2ExpressionBuilder should create Filter if the catalyst expression is a predicate (e.g. binary comparison).

also cc @huaxingao for more opinions.

@cloud-fan
Copy link
Contributor

BTW GeneralScalarExpression should overwrite equals and hashCode

@beliefer
Copy link
Contributor Author

#35768 replace this one.

@beliefer beliefer closed this Mar 13, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
2 participants