[pull] master from apache:master - #656
Merged
Merged
Conversation
…nRunner The registrar only registers PAR_DO and SPLITTABLE_PROCESS_SIZED_ELEMENTS_AND_RESTRICTIONS, so the SPLITTABLE_SPLIT_AND_SIZE_RESTRICTIONS cases are unreachable now that that stage has its own runner. Also collapse the two branches of the SDF-process consumer, which constructed identical objects regardless of whether windows were observed; the TODO for a non-window-observing variant is retained. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ingle-item matches (#39851) * fix(dataframe): handle pandas dimensionality reduction in .xs() for single-item matches Beam's .xs() implementation assumed static output shape (DataFrame/Series) across partitions, but pandas reduces dimensionality (DataFrame->Series, Series->scalar) when a key matches exactly one row and all index levels are selected. This caused TypeError/shape-mismatch failures during cross-partition concat. Fixes the key_size >= nlevels path to route matching partitions through a singleton unwrap stage that mirrors pandas' actual runtime behavior, while documenting the inherent proxy-time ambiguity for duplicate-match cases (proxy assumes single-match dimensionality; runtime produces whichever type pandas actually returns). Fixes #28559 Signed-off-by: ManvithPanyam <250704031+ManvithPanyam@users.noreply.github.com> * fix(dataframe): use reindex() for xs() proxy construction to support extension dtypes Addresses review feedback: dtype.type() crashes on Categorical and timezone-aware datetime dtypes since they aren't callable as zero-arg scalar constructors. Switched the DataFrame branch to build the dummy proxy via reindex() + xs(), letting pandas handle type construction internally. Kept dtype.type() as the primary path for the Series (single-match scalar) branch since reindex().iloc[0] alone silently upcasts plain numeric types (e.g. int64 -> float64) by introducing NaN; falls back to reindex().iloc[0] only on TypeError for extension types. Also handles proxy indexes with duplicate labels, which reindex() otherwise rejects. Added regression tests for Categorical, tz-aware datetime, and nullable Int64 columns. Signed-off-by: ManvithPanyam <250704031+ManvithPanyam@users.noreply.github.com> * fix(dataframe): simplify xs() proxy construction and fix non-empty duplicate-proxy crash Further simplifies the reindex()-based proxy generation from the previous commit: unifies dummy_index construction (was duplicated per-branch), and hoists the is_unique/drop_duplicates() dedup check to run once instead of twice. The dedup check is retained, not removed — traced that pandas can cache an IndexEngine on an index once inspected (e.g. via .is_unique, .loc, .get_loc), and that cache can survive slicing to iloc[:0], leaving is_unique stale as False on an otherwise-empty result. This path is reachable in practice (e.g. the existing test harness calls .xs() on the full arg before slicing to an empty proxy), so removing the check entirely would reintroduce a reindex() failure in that case. Also fixes a real bug found while testing: a non-empty user-supplied proxy with duplicate index labels (via to_dataframe(pcoll, proxy=df_with_dup_index)) crashed the old code with 'ValueError: Length mismatch', since drop_duplicates() shrinks the index but not the frame before reassignment. Pre-slicing to iloc[:0] before the dedup check resolves this. Added regression test test_dataframe_xs_non_empty_duplicate_proxy covering both DataFrame and Series non-empty duplicate-proxy cases. Signed-off-by: ManvithPanyam <250704031+ManvithPanyam@users.noreply.github.com> --------- Signed-off-by: ManvithPanyam <250704031+ManvithPanyam@users.noreply.github.com>
…ions handling from FnApiDoFnRunner
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
See Commits and Changes for more details.
Created by
pull[bot] (v2.0.0-alpha.4)
Can you help keep this open source service alive? 💖 Please sponsor : )