Skip to content

[SPARK-52356][PS] Enable divide-by-zero for boolean mod/rmod with ANSI enabled #51058

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 4 commits into from

Conversation

xinrong-meng
Copy link
Member

@xinrong-meng xinrong-meng commented May 30, 2025

What changes were proposed in this pull request?

Enable divide-by-zero for boolean mod/rmod with ANSI enabled

Why are the changes needed?

Ensure pandas on Spark works well with ANSI mode on.
Part of https://issues.apache.org/jira/browse/SPARK-52169.

Does this PR introduce any user-facing change?

Yes, divide-by-zero is enabled when ANSI is on, as shown below:

>>> ps.set_option("compute.fail_on_ansi_mode", False)
>>> pser = pd.Series([True, False])
>>> psser = ps.from_pandas(pser)

>>> ps.set_option("compute.ansi_mode_support", True)
>>> spark.conf.set("spark.sql.ansi.enabled", True)
>>> 1 % psser
0    0.0
1    NaN
dtype: float64

# Same as ANSI off
>>> spark.conf.set("spark.sql.ansi.enabled", False)
>>> 1 % psser
0    0.0
1    NaN
dtype: float64

How was this patch tested?

Unit tests, and

(dev3.10) spark (divide_0_tests) % SPARK_ANSI_SQL_MODE=true  ./python/run-tests --python-executables=python3.10 --testnames "pyspark.pandas.tests.data_type_ops.test_boolean_ops BooleanOpsTests.test_mod"

Running PySpark tests. Output is in /Users/xinrong.meng/spark/python/unit-tests.log
...
Finished test(python3.10): pyspark.pandas.tests.data_type_ops.test_boolean_ops BooleanOpsTests.test_mod (5s)
Tests passed in 5 seconds


(dev3.10) spark (divide_0_tests) % SPARK_ANSI_SQL_MODE=true  ./python/run-tests --python-executables=python3.10 --testnames "pyspark.pandas.tests.data_type_ops.test_boolean_ops BooleanOpsTests.test_rmod"
Running PySpark tests. Output is in /Users/xinrong.meng/spark/python/unit-tests.log
...
Finished test(python3.10): pyspark.pandas.tests.data_type_ops.test_boolean_ops BooleanOpsTests.test_rmod (4s)
Tests passed in 4 seconds

Was this patch authored or co-authored using generative AI tooling?

No

@xinrong-meng xinrong-meng changed the title [WIP][SPARK-52356][PS] Enable divide-by-zero for boolean mod/rmod with ANSI enabled [SPARK-52356][PS] Enable divide-by-zero for boolean mod/rmod with ANSI enabled Jun 2, 2025
@xinrong-meng xinrong-meng marked this pull request as ready for review June 2, 2025 18:23
@xinrong-meng
Copy link
Member Author

@ueshin would you please review, thanks!

@xinrong-meng
Copy link
Member Author

@ueshin @HyukjinKwon @zhengruifeng thanks!

@ueshin
Copy link
Member

ueshin commented Jun 6, 2025

Thanks! merging to master.

@ueshin ueshin closed this in 83639f5 Jun 6, 2025
yhuang-db pushed a commit to yhuang-db/spark that referenced this pull request Jun 9, 2025
…I enabled

### What changes were proposed in this pull request?
Enable divide-by-zero for boolean mod/rmod with ANSI enabled

### Why are the changes needed?
Ensure pandas on Spark works well with ANSI mode on.
Part of https://issues.apache.org/jira/browse/SPARK-52169.

### Does this PR introduce _any_ user-facing change?
Yes, divide-by-zero is enabled when ANSI is on, as shown below:

```
>>> ps.set_option("compute.fail_on_ansi_mode", False)
>>> pser = pd.Series([True, False])
>>> psser = ps.from_pandas(pser)

>>> ps.set_option("compute.ansi_mode_support", True)
>>> spark.conf.set("spark.sql.ansi.enabled", True)
>>> 1 % psser
0    0.0
1    NaN
dtype: float64

# Same as ANSI off
>>> spark.conf.set("spark.sql.ansi.enabled", False)
>>> 1 % psser
0    0.0
1    NaN
dtype: float64

```

### How was this patch tested?
Unit tests, and

```py
(dev3.10) spark (divide_0_tests) % SPARK_ANSI_SQL_MODE=true  ./python/run-tests --python-executables=python3.10 --testnames "pyspark.pandas.tests.data_type_ops.test_boolean_ops BooleanOpsTests.test_mod"

Running PySpark tests. Output is in /Users/xinrong.meng/spark/python/unit-tests.log
...
Finished test(python3.10): pyspark.pandas.tests.data_type_ops.test_boolean_ops BooleanOpsTests.test_mod (5s)
Tests passed in 5 seconds

(dev3.10) spark (divide_0_tests) % SPARK_ANSI_SQL_MODE=true  ./python/run-tests --python-executables=python3.10 --testnames "pyspark.pandas.tests.data_type_ops.test_boolean_ops BooleanOpsTests.test_rmod"
Running PySpark tests. Output is in /Users/xinrong.meng/spark/python/unit-tests.log
...
Finished test(python3.10): pyspark.pandas.tests.data_type_ops.test_boolean_ops BooleanOpsTests.test_rmod (4s)
Tests passed in 4 seconds
```

### Was this patch authored or co-authored using generative AI tooling?
No

Closes apache#51058 from xinrong-meng/bool_mod.

Authored-by: Xinrong Meng <xinrong@apache.org>
Signed-off-by: Takuya Ueshin <ueshin@databricks.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants