[SPARK-7240][SQL] Single pass covariance calculation for dataframes #5825

brkyvz · 2015-05-01T04:14:31Z

Added the calculation of covariance between two columns to DataFrames.

cc @mengxr @rxin

make progress make progress2 finished cov implementation. Waiting for freqItems to be merged trying to debug added cov

SparkQA · 2015-05-01T04:19:33Z

Test build #31514 has finished for PR 5825 at commit 7dc6dbc.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class DataFrameStatFunctions(object):
This patch does not change any dependencies.

SparkQA · 2015-05-01T04:40:56Z

Test build #31515 has finished for PR 5825 at commit a7115f1.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class DataFrameStatFunctions(object):
This patch does not change any dependencies.

brkyvz · 2015-05-01T04:47:11Z

retest this please

rxin · 2015-05-01T05:09:30Z

python/pyspark/sql/dataframe.py

+
+        :param col1: The name of the first column
+        :param col2: The name of the second column
+        :return: the covariance of the columns


does :return: work in python doc? Can you try compiling it?

Also say something about the return type, maybe "covariance of the two columns as a double value"

I guess not. IntelliJ generated them automatically... Thought I would give it a shot.

rxin · 2015-05-01T05:47:02Z

Can you also update __init__.py in sql to add them to the python doc?

SparkQA · 2015-05-01T06:05:28Z

Test build #31521 has finished for PR 5825 at commit 4e97a50.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds no public classes.

mengxr · 2015-05-01T07:00:35Z

sql/core/src/main/scala/org/apache/spark/sql/execution/stat/StatFunctions.scala

+    var xAvg = 0.0
+    var yAvg = 0.0
+    var Ck = 0.0
+    var count = 0


Should be Long.

mengxr · 2015-05-01T07:03:11Z

The math part looks good to me except two inline comments.

SparkQA · 2015-05-01T07:04:26Z

Test build #31516 has finished for PR 5825 at commit a7115f1.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class DataFrameStatFunctions(object):
This patch does not change any dependencies.

rxin · 2015-05-01T07:24:25Z

python/pyspark/sql/dataframe.py

+            raise ValueError("col2 should be a string.")
+        return self.df._jdf.stat().cov(col1, col2)
+
+    cov.__doc__ = DataFrame.cov.__doc__


if you do the doc assignment here, then you should move the implementation to dataframe also ... or just put the doc assignment there. don't put the doc in one place and implementation in another

SparkQA · 2015-05-01T09:04:21Z

Test build #31533 has finished for PR 5825 at commit 0c6a759.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-05-01T09:29:51Z

Test build #31536 has finished for PR 5825 at commit 51e39b8.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class DataFrameStatFunctions(object):

mengxr · 2015-05-01T15:52:04Z

sql/core/src/main/scala/org/apache/spark/sql/execution/stat/StatFunctions.scala

+      this
+    }
+    // return the covariance for the observed examples
+    def cov: Double = Ck / count


Just remember that this should be sample covariance (dividing by (n-1) and 0 if n is 1). Try R and make sure we output the same, and then update the doc. See:

https://github.com/apache/spark/blob/master/mllib/src/test/scala/org/apache/spark/ml/regression/LinearRegressionSuite.scala#L144

SparkQA · 2015-05-01T16:20:57Z

Test build #31559 has finished for PR 5825 at commit f2e862b.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class DataFrameStatFunctions(object):

SparkQA · 2015-05-01T18:25:44Z

Test build #31569 has finished for PR 5825 at commit cb18046.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class DataFrameStatFunctions(object):

rxin · 2015-05-01T20:27:19Z

sql/core/src/test/scala/org/apache/spark/sql/DataFrameStatSuite.scala

@@ -25,10 +25,11 @@ import org.apache.spark.sql.test.TestSQLContext.implicits._

 class DataFrameStatSuite extends FunSuite  {

+  import TestData._


i think we prefer having data defined inline now since it is easier to read. those were there historically because it was hard to create schemardds.

rxin · 2015-05-01T20:28:56Z

LGTM. You can fix the test case later. Thanks.

Added the calculation of covariance between two columns to DataFrames. cc mengxr rxin Author: Burak Yavuz <brkyvz@gmail.com> Closes apache#5825 from brkyvz/df-cov and squashes the following commits: cb18046 [Burak Yavuz] changed to sample covariance f2e862b [Burak Yavuz] fixed failed test 51e39b8 [Burak Yavuz] moved implementation 0c6a759 [Burak Yavuz] addressed math comments 8456eca [Burak Yavuz] fix pyStyle3 aa2ad29 [Burak Yavuz] fix pyStyle2 4e97a50 [Burak Yavuz] Merge branch 'master' of github.com:apache/spark into df-cov e3b0b85 [Burak Yavuz] addressed comments v0.1 a7115f1 [Burak Yavuz] fix python style 7dc6dbc [Burak Yavuz] reorder imports 408cb77 [Burak Yavuz] initial commit

brkyvz added 2 commits April 30, 2015 21:10

initial commit

408cb77

make progress make progress2 finished cov implementation. Waiting for freqItems to be merged trying to debug added cov

reorder imports

7dc6dbc

fix python style

a7115f1

rxin reviewed May 1, 2015
View reviewed changes

brkyvz added 2 commits April 30, 2015 22:58

addressed comments v0.1

e3b0b85

Merge branch 'master' of github.com:apache/spark into df-cov

4e97a50

brkyvz added 2 commits April 30, 2015 23:10

fix pyStyle2

aa2ad29

fix pyStyle3

8456eca

mengxr reviewed May 1, 2015
View reviewed changes

addressed math comments

0c6a759

rxin reviewed May 1, 2015
View reviewed changes

moved implementation

51e39b8

fixed failed test

f2e862b

mengxr reviewed May 1, 2015
View reviewed changes

changed to sample covariance

cb18046

brkyvz mentioned this pull request May 1, 2015

[SPARK-7243][SQL] Contingency Tables for DataFrames #5842

Closed

rxin reviewed May 1, 2015
View reviewed changes

asfgit closed this in 4dc8d74 May 1, 2015

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-7240][SQL] Single pass covariance calculation for dataframes #5825

[SPARK-7240][SQL] Single pass covariance calculation for dataframes #5825

brkyvz commented May 1, 2015

SparkQA commented May 1, 2015

SparkQA commented May 1, 2015

brkyvz commented May 1, 2015

rxin May 1, 2015

brkyvz May 1, 2015

rxin commented May 1, 2015

SparkQA commented May 1, 2015

mengxr May 1, 2015

mengxr commented May 1, 2015

SparkQA commented May 1, 2015

rxin May 1, 2015

SparkQA commented May 1, 2015

SparkQA commented May 1, 2015

mengxr May 1, 2015

SparkQA commented May 1, 2015

SparkQA commented May 1, 2015

rxin May 1, 2015

rxin commented May 1, 2015

		@@ -25,10 +25,11 @@ import org.apache.spark.sql.test.TestSQLContext.implicits._

		class DataFrameStatSuite extends FunSuite {

		import TestData._

[SPARK-7240][SQL] Single pass covariance calculation for dataframes #5825

[SPARK-7240][SQL] Single pass covariance calculation for dataframes #5825

Conversation

brkyvz commented May 1, 2015

SparkQA commented May 1, 2015

SparkQA commented May 1, 2015

brkyvz commented May 1, 2015

rxin May 1, 2015

Choose a reason for hiding this comment

brkyvz May 1, 2015

Choose a reason for hiding this comment

rxin commented May 1, 2015

SparkQA commented May 1, 2015

mengxr May 1, 2015

Choose a reason for hiding this comment

mengxr commented May 1, 2015

SparkQA commented May 1, 2015

rxin May 1, 2015

Choose a reason for hiding this comment

SparkQA commented May 1, 2015

SparkQA commented May 1, 2015

mengxr May 1, 2015

Choose a reason for hiding this comment

SparkQA commented May 1, 2015

SparkQA commented May 1, 2015

rxin May 1, 2015

Choose a reason for hiding this comment

rxin commented May 1, 2015