-
Notifications
You must be signed in to change notification settings - Fork 374
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[GLUTEN-1985] avoid forceShuffledHashJoin when the join condition does not supported by the backend #1986
Conversation
Run Gluten Clickhouse CI |
e9db42a
to
65f41e3
Compare
Run Gluten Clickhouse CI |
65f41e3
to
161d454
Compare
Run Gluten Clickhouse CI |
|""".stripMargin | ||
|
||
val df = spark.sql(testSql) | ||
val sortMergeJoinExecTransFormer = df.queryExecution.executedPlan.collect { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
SortMergeJoinExecTransformer is not supported, maybe skip transform SMJ in transform rules.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done
161d454
to
92749e3
Compare
Run Gluten Clickhouse CI |
92749e3
to
8d0c547
Compare
Run Gluten Clickhouse CI |
8d0c547
to
7fe4e4c
Compare
Run Gluten Clickhouse CI |
7fe4e4c
to
c2be9d5
Compare
Run Gluten Clickhouse CI |
c2be9d5
to
57e1086
Compare
Run Gluten Clickhouse CI |
57e1086
to
f08ce5f
Compare
Run Gluten Clickhouse CI |
…s not supported by the backend
f08ce5f
to
63e3070
Compare
Run Gluten Clickhouse CI |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
In our custom JoinSelectionOverrides rule, if forceShuffledHashJoin is set to true, SortmergeJoin will be converted to ShuffledHashJoin regardless of whether the backend supports it or whether it is fallback. If it is executed in spark in the case of falllback, SortmergeJoin will be converted For ShuffledHashJoin, it will take up more memory, resulting in OOM,
Describe the solution you'd like
avoid forceShuffledHashJoin when the join condition does not supported by the backend