[SPARK-23912][SQL][Followup] Refactor ArrayDistinct #22044
Conversation
|
Test build #94450 has finished for PR 22044 at commit
|
|
retest this please |
|
Test build #94461 has finished for PR 22044 at commit
|
|
retest this please |
|
Test build #94471 has finished for PR 22044 at commit
|
|
retest this please |
| } | ||
|
|
||
| @transient protected lazy val (hsPostFix, hsTypeName) = { | ||
| val ptName = CodeGenerator.primitiveTypeName (elementType) |
There was a problem hiding this comment.
nit: extra space after primitiveTypeName.
| } | ||
|
|
||
| private def setNotNullValue(isPrimitive: Boolean, | ||
| private def setNotNullValue( |
| case _ => false | ||
| } | ||
|
|
||
| @transient protected lazy val canUseSpecializedHashSet = elementType match { |
There was a problem hiding this comment.
Can we extract those common methods?
There was a problem hiding this comment.
We can do so. To minimize the changes due to remaining time for cutting, I would like to do this in another PR #21912.
| s"$distinctArray.set$primitiveValueTypeName($pos, $getValue1)" | ||
| } | ||
|
|
||
| private def setValueForFastEval( |
|
Test build #94487 has finished for PR 22044 at commit
|
| val openHashSet = classOf[OpenHashSet[_]].getName | ||
| val hs = ctx.freshName("hs") | ||
| val classTag = s"scala.reflect.ClassTag$$.MODULE$$.$hsTypeName()" | ||
| val getValue1 = CodeGenerator.getValue(array, elementType, i) |
|
Test build #94475 has finished for PR 22044 at commit
|
|
LGTM pending Jenkins. |
|
Test build #94488 has finished for PR 22044 at commit
|
|
Thanks! merging to master. |
3 participants
What changes were proposed in this pull request?
This PR simplified code generation for
ArrayDistinct. #21966 enabled code generation only if the type can be specialized by the hash set. This PR follows this strategy.Optimization of null handling will be implemented in #21912.
How was this patch tested?
Existing UTs