Skip to content

Commit

Permalink
[SPARK-39611][PYTHON][PS] Fix wrong aliases in __array_ufunc__
Browse files Browse the repository at this point in the history
### What changes were proposed in this pull request?
This PR fix the wrong aliases in `__array_ufunc__`

### Why are the changes needed?
When running test with numpy 1.23.0 (current latest), hit a bug: `NotImplementedError: pandas-on-Spark objects currently do not support <ufunc 'divide'>.`

In `__array_ufunc__` we first call `maybe_dispatch_ufunc_to_dunder_op` to try dunder methods first, and then we try pyspark API. `maybe_dispatch_ufunc_to_dunder_op` is from pandas code.

pandas fix a bug pandas-dev/pandas#44822 (comment) pandas-dev/pandas@206b249 when upgrade to numpy 1.23.0, we need to also sync this.

### Does this PR introduce _any_ user-facing change?
No

### How was this patch tested?
- Current CI passed
- The exsiting UT `test_series_datetime` already cover this, I also test it in my local env with 1.23.0
```shell
pip install "numpy==1.23.0"
python/run-tests --testnames 'pyspark.pandas.tests.test_series_datetime SeriesDateTimeTest.test_arithmetic_op_exceptions'
```

Closes #37078 from Yikun/SPARK-39611.

Authored-by: Yikun Jiang <yikunkero@gmail.com>
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
(cherry picked from commit fb48a14)
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
  • Loading branch information
Yikun authored and HyukjinKwon committed Jul 5, 2022
1 parent 9adfc3a commit 6ae97e2
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion python/pyspark/pandas/numpy_compat.py
Expand Up @@ -157,7 +157,7 @@ def maybe_dispatch_ufunc_to_dunder_op(
"true_divide": "truediv",
"power": "pow",
"remainder": "mod",
"divide": "div",
"divide": "truediv",
"equal": "eq",
"not_equal": "ne",
"less": "lt",
Expand Down

0 comments on commit 6ae97e2

Please sign in to comment.