Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-39054][PYTHON][PS] Ensure infer schema accuracy in GroupBy.apply #36581

Closed
wants to merge 1 commit into from

Conversation

Yikun
Copy link
Member

@Yikun Yikun commented May 17, 2022

What changes were proposed in this pull request?

Ensure sampling rows >= 2 to make sure apply's infer schema is accurate.

Why are the changes needed?

GroupBy.apply infers schema when the type hints is not specified for the func of grouby.apply. We cannot guarantee that the infer schema of the sampled values ​​is completely accurate, but we should make it as accurate as possible.

Since v1.4, Pandas introduce an interface change [1], especially it has some impact the behavior of Group.apply when df has single row (head(1)):

>> psdf
Out[9]: 
  __groupkey_0__  timestamp car_id
0              A        0.0      A
1              A        0.0      A

>> psdf.head(1)._to_internal_pandas().groupby(groupkeys).apply(pandas_apply, *args, **kwargs)

Out[10]: 
   column
0     0.0

>> psdf.head(2)._to_internal_pandas().groupby(groupkeys).apply(pandas_apply, *args, **kwargs)

Out[11]: 
          column
car_id          
A      0     0.0

Finally, it causes the mismatch error of sample apply infer schema and final apply results.

[1] https://pandas.pydata.org/pandas-docs/stable/whatsnew/v1.4.0.html#groupby-apply-consistent-transform-detection

Does this PR introduce any user-facing change?

No

How was this patch tested?

  • GA CI passed
  • test_apply_infer_schema_without_shortcut and test_apply_with_new_dataframe_without_shortcut passed with v1.4.

@HyukjinKwon
Copy link
Member

Merged to master.

HyukjinKwon pushed a commit that referenced this pull request Jun 10, 2022
… changes

### What changes were proposed in this pull request?
Add migration guide for pandas 1.4 behavior changes:
* SPARK-39054 #36581: In Spark 3.4, if Pandas on Spark API `Groupby.apply`'s `func` parameter return type is not specified and `compute.shortcut_limit` is set to `0`, the sampling rows will be set to 2 (ensure sampling rows always >= 2) to make sure infer schema is accurate.

* SPARK-38822 #36168 In Spark 3.4, if Pandas on Spark API `Index.insert` is out of bounds, will rasied IndexError with `index {} is out of bounds for axis 0 with size {}` to follow pandas 1.4 behavior.

* SPARK-38857 #36159 In Spark 3.4, the series name will be preserved in Pandas on Spark API `Series.mode` to follow pandas 1.4 behavior.

* SPARK-38859 #36142 In Spark 3.4, the Pandas on Spark API `Index.__setitem__` will first to check `value` type is `Column` to avoid raise unexpected error in `is_list_like` like `Cannot convert column into bool: please use '&' for 'and', '|' for 'or', '~' for 'not' when building DataFrame boolean expressions.`.

* SPARK-38820 #36357 In Spark 3.4, the Pandas on Spark API `astype('category')` will also refresh `categories.dtype` according to original data `dtype` to follow pandas 1.4 behavior.

* SPARK-38947 #36464 In Spark 3.4, the Pandas on Spark API supports groupby positional indexing in `GroupBy.head` and `GroupBy.tail` to follow pandas 1.4. Negative arguments now work correctly and result in ranges relative to the end and start of each group, Previously, negative arguments returned empty frames.

* SPARK-39317 #36699 In Spark 3.4, the infer schema process of `groupby.apply` in Pandas on Spark, will first infer the pandas type to ensure the accuracy of the pandas `dtype` as much as possible.

* SPARK-39314 #36711 In Spark 3.4, the `Series.concat` sort parameter will be respected to follow pandas 1.4 behaviors.

For other test only fixes, I don't add migration doc: SPARK-38821 SPARK-39053 SPARK-38982

### Why are the changes needed?

### Does this PR introduce _any_ user-facing change?

### How was this patch tested?

Closes #36816 from Yikun/SPARK-39425.

Authored-by: Yikun Jiang <yikunkero@gmail.com>
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants