-
Notifications
You must be signed in to change notification settings - Fork 44
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Bugfix] Support "starrocks.filter.query" for Spark SQL read #92
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the PR. I think it's ok to support user-defined filters in the configuration. Just curious about why not add filter in the join clause like
select * from spark_starrocks_4 a join spark_starrocks_4 b on a.name = b.name where a.name = 'jhon' or a.name = 'Lucy'
The filter will be pushed down to the source automatically, and it seems more flexible. Defining the filter in the configuration is generally used for DataFrame program.
@@ -83,7 +83,11 @@ private[sql] class StarrocksRelation( | |||
} | |||
|
|||
if (filters != null && filters.length > 0) { | |||
paramWithScan += (ConfigurationOptions.STARROCKS_FILTER_QUERY -> filterWhereClause) | |||
val userFilters = paramWithScan.get(ConfigurationOptions.STARROCKS_FILTER_QUERY) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
- what if the user configures the filter, but the framework's
filters
is null? - add a test to verify it. please refer to
ReadWriteITTest#testSql
for an example.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
- use old config, ConfigurationOptions.STARROCKS_FILTER_QUERY
- ok
Please deal with the DCO and CLA check failure |
Thanks for you answer. |
68f0d30
to
2234006
Compare
Signed-off-by: feihengye <qiangsheng.yet@gmail.com>
2234006
to
73a81e6
Compare
LGTM. Thanks. |
What type of PR is this:
BugFix
Which issues of this PR fixes :
Fixes #91
Problem Summary(Required) :