[SPARK-13242] [SQL] Generate one method per when clause
#11221
Conversation
This is presumably slower than packing them all into a single method, but does allow many more clauses. A further improvement might be to combine the two approaches using some heuristic to determine how many clauses can be combined in a single method.
|
Can one of the admins verify this patch? |
|
cc @davies I wonder if we can do something smarter (e.g. batch some?). This is going to regress performance for very simple cases, especially when we use whole-stage codegen. |
|
@joehalliwell This will not work in whole stage codegen, because there is not a variable for the input row. I think an easy fix could be fallback to interpret mode if the number of branches go over 10? you could take Literal as example. |
|
Thanks for taking the time to review and comment. @rxin Yes, I wondered about batching to (say) 64KB of source code before splitting. That's the approach used in #7076. Without measuring, I wasn't sure what the performance impact of not batching would be. For example I believe JIT can inline small methods such as these. But if you think it's worth it, I'll push in that general direction in future work. @davies Ahh... I'm only really familiar with few bits of codegen I needed to write this patch. Are there tests for or examples of whole stage codgen you could point me at? In the meantime, I'll implement your suggestion of falling back to interpretation. Shall I do that on this PR, or open a new one? |
|
@joehalliwell Since the implementation will be totally different, it make sense to open a new one, thanks! |
|
@joehalliwell In order to test whole stage codegen, you need to have a SQL query to run. |
## What changes were proposed in this pull request? If there are many branches in a CaseWhen expression, the generated code could go above the 64K limit for single java method, will fail to compile. This PR change it to fallback to interpret mode if there are more than 20 branches. This PR is based on apache#11243 and apache#11221, thanks to joehalliwell Closes apache#11243 Closes apache#11221 ## How was this patch tested? Add a test with 50 branches. Author: Davies Liu <davies@databricks.com> Closes apache#11592 from davies/fix_when.
This patch modifies code generation to split
whenclauses into separate methods.As the included test demonstrates, with the previous behaviour it was quite easy to breach the 64KB method size limit with even moderately complex
whenexpressions.This contribution is my original work and I license the work to the project under the project's open source license.