[SPARK-47672][SQL] Avoid double eval from filter pushDown w/ projection pushdown

[holdenk]

What changes were proposed in this pull request?

Changes the filter pushDown optimizer to not push down past projections of the same element if we reasonable expect that computing that element is likely to be expensive.

This is a slightly complex alternative to #45802 which also moves parts of projections down so that the filters can move further down.

An expression can indicate if it is too expensive to be worth the potential savings of being double evaluated as a result of pushdown (by default we do this for all UDFs).

Future Work / What else remains to do?

Right now if a cond is expensive and it references something in the projection we don't push-down. We could probably do better and gate this on if the thing we are reference is expensive rather than the condition it's self. We could do this as a follow up item or as part of this PR.

Why are the changes needed?

Currently Spark may double compute expensive operations (like json parsing, UDF eval, etc.) as a result of filter pushdown past projections.

Does this PR introduce any user-facing change?

SQL optimizer change may impact some user queries, results should be the same and hopefully a little faster.

How was this patch tested?

New tests were added to the FilterPushDownSuite, and the initial problem of double evaluation was confirmed with a github gist

Was this patch authored or co-authored using generative AI tooling?

Used claude to generate more test coverage.

[holdenk]

CC @cloud-fan do you have thoughts / cycles?

[mridulm]

+CC @shardulm94

[github-actions]

We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!

[holdenk]

Hi @cloud-fan looks like the "with" suggestion ended up being more complicated than originally suggested( see #46499 (comment) ). In the interest of progress and avoiding double evaluation of a lot of really expensive things we don't need I intend to update this PR and merge it. We can still circle back to the with approach eventually.

[cloud-fan]

Sorry for the late response to this project. I think the With approach is not that complicated and I'm fixing the nested With limitation here: #49093 . After this is merged, I can followup with the actual pushdown implementation if @zml1206 can't continue his work.

[github-actions]

We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!

[holdenk]

Just wondering if we have a consesus on the best way to go about this @zml1206 / @cloud-fan ? I'm thinking based on the > 1 year since with change it might be more complicated than we originally thought. I can re-explore as well is @zml1206 is busy but we could also go for the simpler solution in the meantime since double UDF evaluation is bad.

[cloud-fan]

Hi @holdenk , we tried very hard to solve this issue efficiently but failed. The idea was to let filter carry a project list and push them down together, but when we push through Project/Aggregate which also contains a project list, we may still hit expression duplication and need to make a decision based on cost.

Sorry I should have moved back to this PR earlier. I think we can simplify it a bit as we will likely never have a practical cost model for Spark expressions. Let's just avoid UDF expression (extends marker expression UserDefinedExpression) duplication during a filter pushdown and add a config to enable it.

[holdenk]

Sounds like a plan, I'll work on simplifying this code.

[holdenk]

Assuming this still passes CI (fingers crossed), that I've updated the docs and simplified the flow of the optimization rule a little bit, I intend to merge this next week.

[cloud-fan]

A different view to look at this problem and design the algorithm: A Project with N named expressions can at most be split into N Projects. e.g. Project(expr1 AS c1, expr2 AS c2, expr3 AS c3) can be split into

Project(c1, c2, expr3 AS c2)
  Project(leaf.output, c1, expr2 AS c2)
    Project(leaf.output, expr1 AS c1)
      leaf

Each condition references a subset or all of the N named expressions, let's group the conditions by the number of referenced named expressions, group1 means only reference one named expression, and so on.

For conditions in group1, we rank the named expressions by "how many conditions can be evaluated with it" and pick the best, let's say it's e1, we create the first Project to evaluate e1, and move conditions that can't be evaluated to group2.

For conditions in group2, again we rank the rest of the named expressions (exclude e1) by "how many conditions can be evaluated with it" and pick the best. For named expressions with the same rank, we sort by ref count and pick the best, let's say it's e2, and we repeat the same process and repeat it for group3, group4, ..., groupN.

This algorithm tries to run as many filters earlier as possible.

[holdenk]

Yes that's true, but given your previous statement around how adding projections is not free I don't think that's the right way to structure this.

[peter-toth]

let's group the conditions by the number of referenced named expressions, group1 means only reference one named expression, and so on.
...
For conditions in group1, we rank the named expressions by "how many conditions can be evaluated with it" and pick the best

This seems like a good heuristics to me to split the projections around.

[peter-toth]

Seems like you don't use replaced anywere or am I missing something?

[holdenk]

It's a good point, we currentl re-calculate replaced again further downstream when we need it.

[cloud-fan]

To discuss #46143 (comment) further:

Yes that's true, but given your previous statement around how adding projections is not free I don't think that's the right way to structure this.

That's why my initial suggestion was to not do this optimization at all. We just keep the Filter above the Project. By doing so we avoid the expensive expression duplication caused by filter pushdown, but all expressions in Project now need to be evaluated against the full input. I'm not sure how serious this issue is, and I was just trying to help simplify the algorithm given you are doing this optimization. I'm more than happier if you agree to drop this optimization and simplify the code.

[holdenk]

That's why my initial suggestion was to not do this optimization at all. We just keep the Filter above the Project. By doing so we avoid the expensive expression duplication caused by filter pushdown, but all expressions in Project now need to be evaluated against the full input. I'm not sure how serious this issue is, and I was just trying to help simplify the algorithm given you are doing this optimization. I'm more than happier if you agree to drop this optimization and simplify the code.

So just always leave up complex filters and don't don't attempt to split them if needed? I think that's sub-optimal for fairly self evident reasons but if you still find the current implementation too complex I could move it into a follow-on PR so there's less to review here and we just fix the perf regression introduced in 3.0

[cloud-fan]

A followup SGTM, at least we can fix the perf regression first.

[holdenk]

Awesome, I'll rework this then :)