[SPARK-58013][PS] Fix loc setitem dropping writes for reordered columns on pandas 3 - #57504
[SPARK-58013][PS] Fix loc setitem dropping writes for reordered columns on pandas 3#57504Vivek1106-04 wants to merge 2 commits into
Conversation
…ns on pandas 3 The pandas 3 CoW branch in `LocIndexerLike.__setitem__` returned early (no-op) when a scalar was assigned to a list of columns not in the frame's internal order. pandas 3 actually applies such writes, so the guard silently dropped valid assignments. Remove it; both column orders now use the normal path. The change is behind `pandas >= 3.0.0`, so pandas 2 is unaffected.
f46e9a2 to
0753c3a
Compare
There was a problem hiding this comment.
After the deletion, selected_column_labels is unpacked but never used anywhere else in the method (its only other references were inside the removed block; missing_keys handling uses missing_keys/data_spark_columns, not this name). Please revert the unpack to _, data_spark_columns, _, _, _ so the now-misleading name does not imply it is live.
uros-b
left a comment
uros-b
left a comment
There was a problem hiding this comment.
Thank you @Vivek1106-04 for the cleanup, I left one suggestion, otherwise looks good
|
@uros-b! Any update?, Thanks. |
|
also cc @ueshin |
|
LGTM. The old behavior is for pandas 3.0.0 to somewhere below 3.0.5, so I guess we can just drop this behavior as pandas thought this was a bug to fix during maintenance releases; otherwise need to put the upperbound to the
% ./python/run-tests --python-executables=python --testname 'pyspark.pandas.tests.indexes.test_indexing_loc IndexingLocTests.test_frame_loc_setitem'
...
pyspark.errors.exceptions.base.PySparkAssertionError: [DIFFERENT_PANDAS_DATAFRAME] DataFrames are not almost equal:
Left:
max_speed shield
cobra 1 2
viper 10 10
sidewinder 10 10
max_speed int64
shield int64
dtype: object
Right:
max_speed shield
cobra 1 2
viper 4 5
sidewinder 7 8
max_speed int64
shield int64
dtype: object |
5 participants
What changes were proposed in this pull request?
Removes an early-return in
LocIndexerLike.__setitem__(python/pyspark/pandas/indexing.py) that, under pandas 3, turned a scalar.locassignment into a no-op when the target columns were passed in a different order than the frame's internal column order.Why are the changes needed?
The branch (added in SPARK-55296) assumed pandas 3 does not apply such writes. It does, e.g.
df.loc[["viper", "sidewinder"], ["shield", "max_speed"]] = 10sets both columns. The guard silently dropped valid writes, failingtest_frame_loc_setitem(op1) on pandas 3. For a scalar assignment the column order is irrelevant, and the CoW view-decoupling the branch was meant to protect is already handled by the normal assignment path.Does this PR introduce any user-facing change?
Yes. With pandas 3,
DataFrame.loc[rows, cols] = scalarnow correctly applies whencolsis a list in non-internal order; previously it was a silent no-op. No change on pandas 2 (the removed code was behind apandas >= 3.0.0check).How was this patch tested?
Ran
pyspark.pandas.tests.indexes.test_indexing_loc(classic and Connect parity) on pandas 3.0.5. The previously failingtest_frame_loc_setitempasses, and the fullindexessuite is green with no regressions.Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code