-
Notifications
You must be signed in to change notification settings - Fork 28.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-22603][SQL] Fix 64KB JVM bytecode limit problem with FormatString #19817
Conversation
val argList = ctx.freshName("argLists") | ||
val numArgLists = argListGen.length | ||
val argListCode = argListGen.zipWithIndex.map { case(v, index) => | ||
v._2.code + s"\n$argList[$index] = " + |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
would it be possible to use the syntax:
s"""
...
"""
as it is done in many other places? I think it would be more readable. What do you think?
Test build #84173 has finished for PR 19817 at commit
|
Test build #84180 has finished for PR 19817 at commit
|
LGTM |
arguments = ("InternalRow", ctx.INPUT_ROW) :: ("Object[]", argList) :: Nil) | ||
} else { | ||
argListCode.mkString("\n") | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could you create a splitExpressions
in CodegenContext
for avoiding the duplicate codes, like
if (ctx.INPUT_ROW != null && ctx.currentVars == null) {
ctx.splitExpressions(...)
} else {
inputs.mkString("\n")
}
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sure, is it better to do this in this PR? Or, should I create another PR?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I wanted to submit a PR for that, but I was waiting for all the PRs related (this one should be the last one) to be merged. Let me know if I can do that or you are doing it. My suggestion would be: can't we just change the existing method to include this check?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM, merging to master/2.2! |
## What changes were proposed in this pull request? This PR changes `FormatString` code generation to place generated code for expressions for arguments into separated methods if these size could be large. This PR passes variable arguments by using an `Object` array. ## How was this patch tested? Added new test cases into `StringExpressionSuite` Author: Kazuaki Ishizaki <ishizaki@jp.ibm.com> Closes #19817 from kiszk/SPARK-22603. (cherry picked from commit 2dbe275) Signed-off-by: Wenchen Fan <wenchen@databricks.com>
## What changes were proposed in this pull request? This PR changes `FormatString` code generation to place generated code for expressions for arguments into separated methods if these size could be large. This PR passes variable arguments by using an `Object` array. ## How was this patch tested? Added new test cases into `StringExpressionSuite` Author: Kazuaki Ishizaki <ishizaki@jp.ibm.com> Closes apache#19817 from kiszk/SPARK-22603. (cherry picked from commit 2dbe275) Signed-off-by: Wenchen Fan <wenchen@databricks.com>
What changes were proposed in this pull request?
This PR changes
FormatString
code generation to place generated code for expressions for arguments into separated methods if these size could be large.This PR passes variable arguments by using an
Object
array.How was this patch tested?
Added new test cases into
StringExpressionSuite