Skip to content

Conversation

@gatorsmile
Copy link
Member

What changes were proposed in this pull request?

To register a JAVA UDF in PySpark, users are unable to specify the registered UDF is not deterministic. The proposal is to add the extra parameter deterministic at the end of the function registerJavaFunction

Below is an example.

>>> from pyspark.sql.types import DoubleType
>>> sqlContext.registerJavaFunction("javaRand",
...   "test.org.apache.spark.sql.JavaRandUDF", DoubleType(), deterministic=False)
>>> sqlContext.sql("SELECT javaRand(3)").collect()
[Row(UDF:javaRand(3)=3.12345)]

How was this patch tested?

Added test cases

* @param returnDataType return type of udf. If it is null, spark would try to infer
* via reflection.
*/
private[sql] def registerJava(name: String, className: String, returnDataType: DataType): Unit = {
Copy link
Member Author

@gatorsmile gatorsmile Jan 5, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Although this is private and only used for PySpark, the JAVA users still could call it. For not breaking the existing behavior, I keep the original one unchanged.

@SparkQA
Copy link

SparkQA commented Jan 5, 2018

Test build #85706 has finished for PR 20162 at commit cf0076b.

  • This patch fails Python style tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Jan 5, 2018

Test build #85707 has finished for PR 20162 at commit d63e816.

  • This patch fails Python style tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Jan 5, 2018

Test build #85708 has finished for PR 20162 at commit 7e4f3c0.

  • This patch fails PySpark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@gatorsmile gatorsmile closed this Jan 5, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants