Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-47852][PYTHON] Support DataFrameQueryContext for reverse operations #46053

Closed
wants to merge 1 commit into from

Conversation

itholic
Copy link
Contributor

@itholic itholic commented Apr 15, 2024

What changes were proposed in this pull request?

This PR proposes to support DataFrameQueryContext for reverse operations, such as rdiv, rsub and rmod.

Why are the changes needed?

To improve debuggability for reverse binary operations.

Does this PR introduce any user-facing change?

No API changes, but the user-facing error messages for reverse binary operations will be improved:

Example

The problematic rdiv is called from 4th line which raises DIVIDE_BY_ZERO:

  1 spark.conf.set("spark.sql.ansi.enabled", True)
  2 df = spark.createDataFrame([(0,), (1,), (2,)], ["id"])
  3
  4 a = 10 / df.id
  5 b = 0 / df.id
  6
  7 df.select(
  8     a,
  9     df.id + 4,
 10     b,
 11     df.id * 5
 12 ).collect()

Before

pyspark.errors.exceptions.captured.ArithmeticException: [DIVIDE_BY_ZERO] Division by zero. Use `try_divide` to tolerate divisor being 0 and return NULL instead. If necessary set "spark.sql.ansi.enabled" to "false" to bypass this error. SQLSTATE: 22012
== DataFrame ==
"divide" was called from
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

After

pyspark.errors.exceptions.captured.ArithmeticException: [DIVIDE_BY_ZERO] Division by zero. Use `try_divide` to tolerate divisor being 0 and return NULL instead. If necessary set "spark.sql.ansi.enabled" to "false" to bypass this error. SQLSTATE: 22012
== DataFrame ==
"divide" was called from
test_error_context.py:4

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

@itholic itholic marked this pull request as draft April 16, 2024 00:00
@itholic itholic closed this Apr 16, 2024
@itholic
Copy link
Contributor Author

itholic commented Apr 16, 2024

This can be covered by #46063

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
1 participant