[SPARK-27824][SQL] Make rule EliminateResolvedHint idempotent#24692
[SPARK-27824][SQL] Make rule EliminateResolvedHint idempotent#24692maryannxue wants to merge 1 commit intoapache:masterfrom
Conversation
| def apply(plan: LogicalPlan): LogicalPlan = { | ||
| val pulledUp = plan transformUp { | ||
| case j: Join => | ||
| case j: Join if j.hint == JoinHint.NONE => |
There was a problem hiding this comment.
@maryannxue If we don't do this transformation, we still require the traversal to remove ResolvedHint ?
There was a problem hiding this comment.
I think the "best" way to fix it is to merge the new hints into the existing JoinHint, but the question is which one should take precedence, the new one or the existing one?
So far, if things all work right, we should never come to a point where there is a non-empty new hint and a non-empty existing hint, that is, the hint has either been pulled into the join node or it hasn't been processed it. In either case, this fix is sufficient.
The removal of the ResolvedHint is taken care of by the following lines and should not happen at all if the rule has been applied once.
|
Test build #105739 has finished for PR 24692 at commit
|
## What changes were proposed in this pull request? This fix prevents the rule EliminateResolvedHint from being applied again if it's already applied. ## How was this patch tested? Added new UT. Closes apache#24692 from maryannxue/eliminatehint-bug. Authored-by: maryannxue <maryannxue@apache.org> Signed-off-by: gatorsmile <gatorsmile@gmail.com>
4 participants
What changes were proposed in this pull request?
This fix prevents the rule EliminateResolvedHint from being applied again if it's already applied.
How was this patch tested?
Added new UT.