Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-40210][PYTHON][CORE] Fix math atan2, hypot, pow and pmod float argument call #37650

Conversation

khalidmammadov
Copy link
Contributor

What changes were proposed in this pull request?

PySpark atan2, hypot, pow and pmod functions marked as accepting float type as argument but produce error when called together. For example:

>>> from pyspark.sql.functions import *
>>> df = spark.range(1)
>>> df.select(atan2(1.1, 2.1)).first()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/khalid/dev/p/SparkTest/spark_venv3.10/lib/python3.10/site-packages/pyspark/sql/functions.py", line 962, in atan2
    return _invoke_binary_math_function("atan2", col1, col2)
  File "/home/khalid/dev/p/SparkTest/spark_venv3.10/lib/python3.10/site-packages/pyspark/sql/functions.py", line 111, in _invoke_binary_math_function
    return _invoke_function(
  File "/home/khalid/dev/p/SparkTest/spark_venv3.10/lib/python3.10/site-packages/pyspark/sql/functions.py", line 85, in _invoke_function
    return Column(jf(*args))
  File "/home/khalid/dev/p/SparkTest/spark_venv3.10/lib/python3.10/site-packages/pyspark/python/lib/py4j-0.10.9.5-src.zip/py4j/java_gateway.py", line 1321, in __call__
  File "/home/khalid/dev/p/SparkTest/spark_venv3.10/lib/python3.10/site-packages/pyspark/sql/utils.py", line 190, in deco
    return f(*a, **kw)
  File "/home/khalid/dev/p/SparkTest/spark_venv3.10/lib/python3.10/site-packages/pyspark/python/lib/py4j-0.10.9.5-src.zip/py4j/protocol.py", line 330, in get_return_value
py4j.protocol.Py4JError: An error occurred while calling z:org.apache.spark.sql.functions.atan2. Trace:
py4j.Py4JException: Method atan2([class java.lang.Double, class java.lang.Double]) does not exist
        at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:318)
        at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:339)
        at py4j.Gateway.invoke(Gateway.java:276)
        at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
        at py4j.commands.CallCommand.execute(CallCommand.java:79)
        at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
        at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
        at java.lang.Thread.run(Thread.java:748)

Although, this call could be written outside the Spark and supplied as a literal it's still good to be consistent with SQL API and help user to avoid seeing error.

after the fix:

>>> from pyspark.sql.functions import *
>>> df = spark.range(1)
>>> df.select(atan2(1.1, 2.1)).first()
Row(ATAN2(1.1, 2.1)=0.4825132950224769)

Why are the changes needed?

Improve user experience. Bug fix

Does this PR introduce any user-facing change?

Yes, no errors

How was this patch tested?

./build/sbt
test

@github-actions github-actions bot added the SQL label Aug 24, 2022
* @group math_funcs
* @since 3.4.0
*/
def atan2(yValue: Double, xValue: Double): Column = atan2(lit(yValue), lit(xValue))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should probably fix it in PySpark side instead of fixing it in Scala side (see the comments on the top at both functions.py and functions.scala)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, will do

@khalidmammadov
Copy link
Contributor Author

Will be fixed on PySpark side

@srowen
Copy link
Member

srowen commented Aug 31, 2022

Was there a follow-up PR?

@khalidmammadov
Copy link
Contributor Author

Sorry, there wasn't yet. Got distracted to other changes. Will do one today/tomorrow to see any good

@khalidmammadov
Copy link
Contributor Author

follow up PR: #37748

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
3 participants