[SPARK-23760][SQL] CodegenContext.withSubExprEliminationExprs should save/restore CSE state correctly #20870
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
… it saves/restores CSE state correctly.
rednaxelafx
force-pushed
the
codegen-subexpr-fix
branch
from
8635969
to
df45286
Compare
rednaxelafx
changed the title
rednaxelafx
commented
Mar 21, 2018
| @@ -942,7 +940,7 @@ class CodegenContext { | |||
| def subexpressionEliminationForWholeStageCodegen(expressions: Seq[Expression]): SubExprCodes = { | |||
| // Create a clear EquivalentExpressions and SubExprEliminationState mapping | |||
| val equivalentExpressions: EquivalentExpressions = new EquivalentExpressions | |||
| val subExprEliminationExprs = mutable.HashMap.empty[Expression, SubExprEliminationState] | |||
| val localSubExprEliminationExprs = mutable.HashMap.empty[Expression, SubExprEliminationState] | |||
There was a problem hiding this comment.
This renaming isn't necessary for the fix per-se, but I'd like to piggyback it on this change so that it's clearer that we're not interfering with the current CSE state of this CodegenContext here.
|
Test build #88448 has finished for PR 20870 at commit
|
|
Test build #88449 has finished for PR 20870 at commit
|
|
jenkins retest this please |
|
Test build #88459 has finished for PR 20870 at commit
|
|
good catch! merging to master/2.3! also cc @viirya |
asfgit
pushed a commit
that referenced
this pull request
Mar 22, 2018
…save/restore CSE state correctly ## What changes were proposed in this pull request? Fixed `CodegenContext.withSubExprEliminationExprs()` so that it saves/restores CSE state correctly. ## How was this patch tested? Added new unit test to verify that the old CSE state is indeed saved and restored around the `withSubExprEliminationExprs()` call. Manually verified that this test fails without this patch. Author: Kris Mok <kris.mok@databricks.com> Closes #20870 from rednaxelafx/codegen-subexpr-fix. (cherry picked from commit 95e51ff) Signed-off-by: Wenchen Fan <wenchen@databricks.com>
viirya
reviewed
Mar 22, 2018
peter-toth
pushed a commit
to peter-toth/spark
that referenced
this pull request
Oct 6, 2018
…save/restore CSE state correctly ## What changes were proposed in this pull request? Fixed `CodegenContext.withSubExprEliminationExprs()` so that it saves/restores CSE state correctly. ## How was this patch tested? Added new unit test to verify that the old CSE state is indeed saved and restored around the `withSubExprEliminationExprs()` call. Manually verified that this test fails without this patch. Author: Kris Mok <kris.mok@databricks.com> Closes apache#20870 from rednaxelafx/codegen-subexpr-fix. (cherry picked from commit 95e51ff) Signed-off-by: Wenchen Fan <wenchen@databricks.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Fixed
CodegenContext.withSubExprEliminationExprs()so that it saves/restores CSE state correctly.How was this patch tested?
Added new unit test to verify that the old CSE state is indeed saved and restored around the
withSubExprEliminationExprs()call. Manually verified that this test fails without this patch.