[SPARK-3855][SQL] Preserve the result attribute of python UDFs though transformations #2717

marmbrus · 2014-10-08T21:30:59Z

In the current implementation it was possible for the reference to change after analysis.

…rmations

SparkQA · 2014-10-08T21:34:38Z

QA tests have started for PR 2717 at commit 9533286.

This patch merges cleanly.

davies · 2014-10-08T22:13:05Z

It's reproducable by this query:

SELECT strlen(a) FROM test WHERE strlen(a) > 1

SparkQA · 2014-10-08T22:20:46Z

QA tests have finished for PR 2717 at commit 9533286.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class EvaluatePython(

AmplabJenkins · 2014-10-08T22:20:50Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21485/Test PASSed.

davies · 2014-10-08T22:34:22Z

@marmbrus Could you add a test in pyhon/pyspark/tests.py (SQLTests) ?

SparkQA · 2014-10-09T20:44:47Z

QA tests have started for PR 2717 at commit 6343bcb.

This patch merges cleanly.

SparkQA · 2014-10-09T21:56:55Z

QA tests have finished for PR 2717 at commit 6343bcb.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class EvaluatePython(

AmplabJenkins · 2014-10-09T21:56:58Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21549/Test FAILed.

SparkQA · 2014-10-10T01:01:38Z

QA tests have started for PR 2717 at commit 6343bcb.

This patch merges cleanly.

SparkQA · 2014-10-10T02:15:30Z

QA tests have finished for PR 2717 at commit 6343bcb.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class EvaluatePython(

SparkQA · 2014-10-10T02:19:25Z

QA tests have started for PR 2717 at commit 6343bcb.

This patch merges cleanly.

SparkQA · 2014-10-10T03:36:25Z

QA tests have finished for PR 2717 at commit 6343bcb.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class EvaluatePython(

davies · 2014-10-10T04:03:27Z

python/pyspark/tests.py

@@ -679,6 +679,12 @@ def test_udf(self):
        [row] = self.sqlCtx.sql("SELECT twoArgs('test', 1)").collect()
        self.assertEqual(row[0], 5)

+    def test_udf2(self):
+        self.sqlCtx.registerFunction("strlen", lambda string: len(string))


data type should be IntegerType

SparkQA · 2014-10-10T18:54:42Z

QA tests have started for PR 2717 at commit da14879.

This patch merges cleanly.

SparkQA · 2014-10-10T20:07:12Z

QA tests have finished for PR 2717 at commit da14879.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class EvaluatePython(

AmplabJenkins · 2014-10-10T20:07:15Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21599/Test FAILed.

davies · 2014-10-10T20:17:10Z

@marmbrus LGTM, just wonder that why you do not use IntegerType as returnType in the tests? I looks wired that we compare stringType with > 1

marmbrus · 2014-10-10T20:31:24Z

@davies, I was lazy and didn't want to look up the syntax for that :P

You are right its a little weird, but Hive supports it.

pwendell · 2014-10-10T23:50:07Z

Jenkins, retest this please.

SparkQA · 2014-10-10T23:54:41Z

QA tests have started for PR 2717 at commit da14879.

This patch merges cleanly.

SparkQA · 2014-10-11T01:11:25Z

QA tests have finished for PR 2717 at commit da14879.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class EvaluatePython(

AmplabJenkins · 2014-10-11T01:11:29Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21609/Test FAILed.

pwendell · 2014-10-13T20:48:43Z

Jenkins - retest this please.

AmplabJenkins · 2014-10-13T21:02:21Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21701/
Test FAILed.

SparkQA · 2014-10-17T18:00:13Z

QA tests have started for PR 2717 at commit da14879.

This patch merges cleanly.

SparkQA · 2014-10-17T19:26:10Z

QA tests have finished for PR 2717 at commit da14879.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class EvaluatePython(

pwendell · 2014-10-17T21:10:07Z

I'd like to pull this in - is that alright @marmbrus?

marmbrus · 2014-10-17T21:10:40Z

Yes, please do.
On Oct 17, 2014 5:10 PM, "Patrick Wendell" notifications@github.com wrote:

I'd like to pull this in - is that alright @marmbrus
https://github.com/marmbrus?

—
Reply to this email directly or view it on GitHub
#2717 (comment).

Correctly preserve the result attribute of python UDFs though transfo…

9533286

…rmations

add test

6343bcb

davies reviewed Oct 10, 2014
View reviewed changes

Fix test

da14879

asfgit closed this in adcb7d3 Oct 17, 2014

marmbrus deleted the pythonUdfResults branch November 19, 2014 02:47

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-3855][SQL] Preserve the result attribute of python UDFs though transformations #2717

[SPARK-3855][SQL] Preserve the result attribute of python UDFs though transformations #2717

marmbrus commented Oct 8, 2014

SparkQA commented Oct 8, 2014

davies commented Oct 8, 2014

SparkQA commented Oct 8, 2014

AmplabJenkins commented Oct 8, 2014

davies commented Oct 8, 2014

SparkQA commented Oct 9, 2014

SparkQA commented Oct 9, 2014

AmplabJenkins commented Oct 9, 2014

SparkQA commented Oct 10, 2014

SparkQA commented Oct 10, 2014

SparkQA commented Oct 10, 2014

SparkQA commented Oct 10, 2014

davies Oct 10, 2014

SparkQA commented Oct 10, 2014

SparkQA commented Oct 10, 2014

AmplabJenkins commented Oct 10, 2014

davies commented Oct 10, 2014

marmbrus commented Oct 10, 2014

pwendell commented Oct 10, 2014

SparkQA commented Oct 10, 2014

SparkQA commented Oct 11, 2014

AmplabJenkins commented Oct 11, 2014

pwendell commented Oct 13, 2014

AmplabJenkins commented Oct 13, 2014

SparkQA commented Oct 17, 2014

SparkQA commented Oct 17, 2014

pwendell commented Oct 17, 2014

marmbrus commented Oct 17, 2014

[SPARK-3855][SQL] Preserve the result attribute of python UDFs though transformations #2717

[SPARK-3855][SQL] Preserve the result attribute of python UDFs though transformations #2717

Conversation

marmbrus commented Oct 8, 2014

SparkQA commented Oct 8, 2014

davies commented Oct 8, 2014

SparkQA commented Oct 8, 2014

AmplabJenkins commented Oct 8, 2014

davies commented Oct 8, 2014

SparkQA commented Oct 9, 2014

SparkQA commented Oct 9, 2014

AmplabJenkins commented Oct 9, 2014

SparkQA commented Oct 10, 2014

SparkQA commented Oct 10, 2014

SparkQA commented Oct 10, 2014

SparkQA commented Oct 10, 2014

davies Oct 10, 2014

Choose a reason for hiding this comment

SparkQA commented Oct 10, 2014

SparkQA commented Oct 10, 2014

AmplabJenkins commented Oct 10, 2014

davies commented Oct 10, 2014

marmbrus commented Oct 10, 2014

pwendell commented Oct 10, 2014

SparkQA commented Oct 10, 2014

SparkQA commented Oct 11, 2014

AmplabJenkins commented Oct 11, 2014

pwendell commented Oct 13, 2014

AmplabJenkins commented Oct 13, 2014

SparkQA commented Oct 17, 2014

SparkQA commented Oct 17, 2014

pwendell commented Oct 17, 2014

marmbrus commented Oct 17, 2014