Conversation
|
@szehon-ho Do we need to back-port to 3.1, 3.0 and 2.4 too? |
|
@huaxingao yea I think its always beneficial if the issue exists in that Spark version, I was more eager about fixing 3.2 personally but I think we can do it to the others as well. |
|
Thanks @huaxingao for backport and @hililiwei for additional review |
|
Thanks a lot @szehon-ho @hililiwei |
| try { | ||
| expr = SparkFilters.convert(filter); | ||
| } catch (IllegalArgumentException e) { | ||
| // converting to Iceberg Expression failed, so this expression cannot be pushed down |
There was a problem hiding this comment.
We should probably log something here
There was a problem hiding this comment.
Yea there was a similar comment on : #5204 (comment), but I think the point was the pushed filters are logged.
I'm ok to log unsuccessful filters too if it helps (could also add the exprs unsuccessfully bound as well in the next statement)
There was a problem hiding this comment.
It could be useful to know that a filter was intentionally not pushed down for a particular reason
There was a problem hiding this comment.
I am OK to have a follow-up to log the unsuccessful filters and also the unsuccessfully bound exprs in the next statement
4 participants
This PR backports the changes in #5204 to Spark 3.2