[SPARK-35601][PYTHON] Complete arithmetic operators involving bool literals, Series, and Index #32785

xinrong-meng · 2021-06-04T18:28:54Z

What changes were proposed in this pull request?

Completing arithmetic operators involving bool literals, Series, and Index consists of two main tasks:

Support arithmetic operations against bool literals
Support operators (+, *) between bool Series/Indexes.

Why are the changes needed?

Arithmetic operators involving bool literals, Series, and Index are incomplete now.
We ought to match pandas' behaviors.

Does this PR introduce any user-facing change?

Yes.

Newly supported operations example:

>>> ps.Series([1, 2, 3]) + True
0    2
1    3
2    4
dtype: int64
>>> ps.Series([1, 2, 3]) + False
0    1
1    2
2    3
dtype: int64
>>> ps.Series([True, False, True]) + True
0    True
1    True
2    True
dtype: bool
>>> ps.Series([True, False, True]) + False
0     True
1    False
2     True
dtype: bool
>>> ps.Series([True, False, True]) * True
0     True
1    False
2     True
dtype: bool
>>> ps.Series([True, False, True]) * False
0    False
1    False
2    False
dtype: bool
>>> ps.set_option('compute.ops_on_diff_frames', True)
>>> ps.Series([True, True, False]) + ps.Series([True, False, True])
0    True
1    True
2    True
dtype: bool
>>> ps.Series([True, True, False]) * ps.Series([True, False, True])
0     True
1    False
2    False
dtype: bool

Before the change, operations above are not supported, raising a TypeError such as

>>> ps.Series([True, False, True]) + True
Traceback (most recent call last):
...
TypeError: Addition can not be applied to booleans and the given type.
>>> ps.Series([True, False, True]) + False
Traceback (most recent call last):
...
TypeError: Addition can not be applied to booleans and the given type.

How was this patch tested?

Unit tests.

Keyword: SPARK-35337

SparkQA · 2021-06-04T19:35:37Z

Test build #139348 has finished for PR 32785 at commit 3756dc3.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-06-04T19:57:35Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43870/

SparkQA · 2021-06-04T20:30:50Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43870/

xinrong-meng · 2021-06-04T21:31:00Z

CC @ueshin @HyukjinKwon @itholic

SparkQA · 2021-06-04T21:44:30Z

Test build #139353 has finished for PR 32785 at commit f90000b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-06-04T21:55:16Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43875/

SparkQA · 2021-06-04T22:25:52Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43875/

SparkQA · 2021-06-07T17:43:57Z

Test build #139428 has finished for PR 32785 at commit ee39236.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-06-07T18:10:17Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43950/

SparkQA · 2021-06-07T18:44:17Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43950/

SparkQA · 2021-06-08T00:40:34Z

Test build #139438 has finished for PR 32785 at commit 2ac8593.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-06-08T01:04:06Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43961/

SparkQA · 2021-06-08T01:38:00Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43961/

xinrong-meng · 2021-06-08T02:29:09Z

ERROR [2.315s]: test_termination_sigterm (pyspark.tests.test_daemon.DaemonTests)
Ensure that daemon and workers terminate on SIGTERM.
----------------------------------------------------------------------
Traceback (most recent call last):
...
AssertionError: Expected EnvironmentError to be raised

Might not be related to the PR. Tests are retriggered.

SparkQA · 2021-06-08T02:59:26Z

Test build #139441 has finished for PR 32785 at commit f2ff9c6.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-06-08T03:11:17Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43963/

HyukjinKwon · 2021-06-08T04:25:59Z

python/pyspark/pandas/data_type_ops/base.py

Out of curiosity, why do we need allow_bool_index_ops? Is there a case where only bool index should be allowed alone?

Good question. There is NOT a case where only bool index should be allowed alone, but there is a case allow_bool_index_ops is False, whereas allow_bool is True, for example

>>> ps.Series([True, False, True]) + ps.Series([True, False, True]) Traceback (most recent call last): ... TypeError: Addition can not be applied to booleans and the given type. >>> ps.Series([True, False, True]) + True 0 True 1 True 2 True dtype: bool

In case you are interested, there is a case allow_bool_index_ops and allow_bool are both True, for example

>>> ps.Series([1, 2, 3]) + ps.Series([True, False, True]) 0 2 1 2 2 4 dtype: int64 >>> ps.Series([1, 2, 3]) + True 0 2 1 3 2 4 dtype: int64

And there is a case allow_bool_index_ops and allow_bool are both False, for example

>> ps.Series([True, False, True]) - ps.Series([True, False, True]) Traceback (most recent call last): ... TypeError: Subtraction can not be applied to booleans and the given type. >>> ps.Series([True, False, True]) - True Traceback (most recent call last): ... TypeError: Subtraction can not be applied to booleans and the given type.

That's why allow_bool_index_ops might be needed. Does that make sense?

For the first example, pandas works for both cases:

>>> pd.__version__ '1.2.4' >>> pd.Series([True, False, True]) + pd.Series([True, False, True]) 0 True 1 False 2 True dtype: bool >>> pd.Series([True, False, True]) + True 0 True 1 True 2 True dtype: bool

so allow_bool_index_ops and allow_bool should be True for this case as well. Then we can consolidate the two?

Makes sense! https://issues.apache.org/jira/browse/SPARK-35681 is created for supporting arithmetic operators (+, *) among bool Series/Index, the parameters will be consolidated at that time.

BTW, I investigated Nan None True evaluation before at https://github.com/apache/spark/pull/24234/files#diff-2086df5f0cf7291ee71bbde83662c0e34600c53c04ee0e377ceadf61e6c496deR365-R382. might be helpful

The arithmetic operators (+, *) among bool Series/Index has been implemented in this PR; tickets have been adjusted.
allow_bool_index_ops parameter discussed above has been removed.

Thanks @HyukjinKwon, that's helpful!

SparkQA · 2021-06-08T04:41:09Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43963/

ueshin · 2021-06-08T20:20:46Z

python/pyspark/pandas/data_type_ops/boolean_ops.py

This should allow bools and behave like &?

>>> pd.Series([True, False, True]) * pd.Series([True, False, False]) 0 True 1 False 2 False dtype: bool >>> pd.Series([True, False, True]) * True 0 True 1 False 2 True dtype: bool >>> pd.Series([True, False, True]) * False 0 False 1 False 2 False dtype: bool

Good catch! pd.Series([True, False, True]) * True and pd.Series([True, False, True]) * False are supported.

+/* between bool Series/Index are supported now.

SparkQA · 2021-06-08T21:49:59Z

Test build #139507 has finished for PR 32785 at commit afe4044.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-06-08T22:04:56Z

Kubernetes integration test unable to build dist.

exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44032/

SparkQA · 2021-06-09T03:11:49Z

Test build #139521 has finished for PR 32785 at commit 5c0f559.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-06-09T03:36:50Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44046/

SparkQA · 2021-06-09T04:13:46Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44046/

Bool index_ops and bool spark_type is optional for transform_boolean_operand_to_numeric Black lint

SparkQA · 2021-06-09T17:57:49Z

Test build #139586 has finished for PR 32785 at commit 1f9f0e4.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-06-09T18:24:33Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44113/

ueshin

LGTM, pending tests.

SparkQA · 2021-06-09T19:56:09Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44113/

SparkQA · 2021-06-09T20:58:12Z

Test build #139593 has finished for PR 32785 at commit 0489747.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-06-09T21:17:39Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44120/

SparkQA · 2021-06-09T21:51:39Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44120/

ueshin · 2021-06-09T22:12:30Z

Thanks! merging to master.

github-actions bot added CORE PYTHON labels Jun 4, 2021

xinrong-meng changed the title ~~[WIP][SPARK-35601][PYTHON] Support arithmetic operations against bool literals~~ [SPARK-35601][PYTHON] Support arithmetic operations against bool literals Jun 4, 2021

xinrong-meng marked this pull request as ready for review June 4, 2021 21:30

xinrong-meng force-pushed the datatypeops_arith_bool branch from f90000b to ee39236 Compare June 7, 2021 17:01

xinrong-meng force-pushed the datatypeops_arith_bool branch from d7da448 to 2ac8593 Compare June 8, 2021 00:03

HyukjinKwon reviewed Jun 8, 2021

View reviewed changes

ueshin reviewed Jun 8, 2021

View reviewed changes

xinrong-meng force-pushed the datatypeops_arith_bool branch from afe4044 to c578614 Compare June 9, 2021 01:49

xinrong-meng changed the title ~~[SPARK-35601][PYTHON] Support arithmetic operations against bool literals~~ [SPARK-35601][PYTHON] Support arithmetic operators against bool literals, Series, and Index Jun 9, 2021

xinrong-meng changed the title ~~[SPARK-35601][PYTHON] Support arithmetic operators against bool literals, Series, and Index~~ [SPARK-35601][PYTHON] Complete arithmetic operators involving bool literals, Series, and Index Jun 9, 2021

xinrong-meng added 7 commits June 9, 2021 09:45

Numeric index_ops and bool

d75c89e

Bool index_ops and bool spark_type is optional for transform_boolean_operand_to_numeric Black lint

Introduce allow_bool

07148ca

BooleanExtensionOps.test_add

ac57234

bool indexops * bool

6c98dbe

bool +/* bool indexops

750bba7

+/* between bool series

b93bbf0

Remove allow_bool_index_ops

1f9f0e4

xinrong-meng force-pushed the datatypeops_arith_bool branch from 5c0f559 to 1f9f0e4 Compare June 9, 2021 16:46

ueshin approved these changes Jun 9, 2021

View reviewed changes

Typo

0489747

ueshin closed this in 3c66c11 Jun 9, 2021

[SPARK-35601][PYTHON] Complete arithmetic operators involving bool literals, Series, and Index #32785

[SPARK-35601][PYTHON] Complete arithmetic operators involving bool literals, Series, and Index #32785

Uh oh!

Conversation

xinrong-meng commented Jun 4, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

SparkQA commented Jun 4, 2021

Uh oh!

SparkQA commented Jun 4, 2021

Uh oh!

SparkQA commented Jun 4, 2021

Uh oh!

xinrong-meng commented Jun 4, 2021

Uh oh!

SparkQA commented Jun 4, 2021

Uh oh!

SparkQA commented Jun 4, 2021

Uh oh!

SparkQA commented Jun 4, 2021

Uh oh!

SparkQA commented Jun 7, 2021

Uh oh!

SparkQA commented Jun 7, 2021

Uh oh!

SparkQA commented Jun 7, 2021

Uh oh!

SparkQA commented Jun 8, 2021

Uh oh!

SparkQA commented Jun 8, 2021

Uh oh!

SparkQA commented Jun 8, 2021

Uh oh!

xinrong-meng commented Jun 8, 2021

Uh oh!

SparkQA commented Jun 8, 2021

Uh oh!

SparkQA commented Jun 8, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

xinrong-meng Jun 9, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Jun 8, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Jun 8, 2021

Uh oh!

SparkQA commented Jun 8, 2021

Uh oh!

SparkQA commented Jun 9, 2021

Uh oh!

SparkQA commented Jun 9, 2021

Uh oh!

SparkQA commented Jun 9, 2021

Uh oh!

SparkQA commented Jun 9, 2021

Uh oh!

SparkQA commented Jun 9, 2021

xinrong-meng commented Jun 4, 2021 •

edited

Loading

xinrong-meng Jun 9, 2021 •

edited

Loading