[SPARK-56248][PYTHON][SS] Optimize python stateful processor serialization to skip unnecessary list/dict/row construction.#55039
Closed
jiateoh wants to merge 2 commits into
Closed
Conversation
jiateoh
commented
Mar 26, 2026
| return serializer | ||
|
|
||
| field_names = [f.name for f in schema.fields] | ||
| row_value = Row(**dict(zip(field_names, converted))) |
Contributor
Author
There was a problem hiding this comment.
PR Description contains explanation/breakdown of why this is safe: TLDR is that the inputs to Schema.toInternal before/after this change are a tuple/list that is positionally ordered based on the input data.
jiateoh
changed the title
jiateoh
changed the title
jiateoh
force-pushed
the
tws_python_serialization_improvements
branch
from
9d1665d to
243a49b
Compare
jiateoh
commented
Mar 27, 2026
jiateoh
force-pushed
the
tws_python_serialization_improvements
branch
from
6a3582f to
615af5d
Compare
…list/dict/row construction. - Eliminate per-call Row(**dict(zip(...))) construction in _serialize_to_bytes; pass normalized tuples directly to schema.toInternal which handles them by index AI-assisted + human reviewed/edited Generated-by: Claude Code (Claude Opus 4.6)
jiateoh
force-pushed
the
tws_python_serialization_improvements
branch
from
615af5d to
58b97b1
Compare
jiateoh
changed the title
jiateoh
changed the title
HeartSaVioR
approved these changes
Apr 2, 2026
Contributor
HeartSaVioR
left a comment
HeartSaVioR
left a comment
There was a problem hiding this comment.
+1 but let's double confirm with one of PySpark folks to confirm the PR is based on the invariants (so it won't be changed in future).
Maybe @zhengruifeng, would you please help taking a look? Thanks in advance!
zhengruifeng
approved these changes
Apr 2, 2026
Contributor
zhengruifeng
left a comment
zhengruifeng
left a comment
There was a problem hiding this comment.
LGTM, it seems the intermediate Row is not necessary.
And Row(**dict(zip(field_names, converted))) doesn't support duplicated field names, while after this change, it should be able to support.
Contributor
|
Thanks! Merging to master. (Sorry for missing this.) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Eliminate per-call Row(**dict(zip(...))) construction in _serialize_to_bytes, pass normalized tuples directly to schema.toInternal which handles them by index
To better explain the removal of
Rowusage:dict(zip(...))preserves insertion order in python 3.7+ (dicts preserve insertion order)zipare field_names, assumed to be same-ordered as theconvertedinput data which is derived from the original input tuple in the same order.Rowis a tuple subclass, so it always hit Schema.toInternal's tuple/list positional branch. Replacing it with the input tuple or a converted list will execute the same Schema.toInternal branch as before.The result is that the extra list, zip, dict, and Row construction is no longer necessary: the end result remains equivalent to the input to the entire
Row(**dict(zip(...)))sequence: a positionally-ordered tuple/listAI-assisted + human reviewed/updated
Why are the changes needed?
This is a code cleanup/performance optimization. Original code has unnecessary operations that are executed for every row, including: rebuilding closures, extracting field names, building intermediate lists + dicts, and constructing Row objects (which sort by field unnecessarily). These can all add minor overhead while having no effect on the underlying usage.
Does this PR introduce any user-facing change?
No
How was this patch tested?
Unit tests
Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 4.6)