-
Notifications
You must be signed in to change notification settings - Fork 28.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-23589][SQL][FOLLOW-UP] Reuse InternalRow in ExternalMapToCatalyst eval #21137
Conversation
Test build #89760 has finished for PR 21137 at commit
|
retest this please |
@@ -1255,6 +1255,13 @@ case class ExternalMapToCatalyst private( | |||
override def dataType: MapType = MapType( | |||
keyConverter.dataType, valueConverter.dataType, valueContainsNull = valueConverter.nullable) | |||
|
|||
private lazy val rowBuffer = InternalRow.fromSeq(Array[Any](1)) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We don't need to make this expression wide state. Just move this inside the mapCatalystConverter
def.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ok
} else { | ||
throw new RuntimeException("Cannot use null as map key!") | ||
} | ||
values(i) = if (value != null) { | ||
valueConverter.eval(InternalRow.fromSeq(value :: Nil)) | ||
valueConverter.eval(rowWrapper(value)) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you also change lines 1299 & 1304?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ok
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM - pending jenkins
Test build #89770 has finished for PR 21137 at commit
|
Test build #89779 has finished for PR 21137 at commit
|
Merging to master. Thanks! |
What changes were proposed in this pull request?
This pr is a follow-up of #20980 and fixes code to reuse
InternalRow
for converting input keys/values inExternalMapToCatalyst
eval.How was this patch tested?
Existing tests.