[SPARK-12962] [SQL] [PySpark] PySpark support covar_samp and covar_pop #10876
Conversation
|
Test build #49909 has finished for PR 10876 at commit
|
| >>> df = sqlContext.createDataFrame(zip(a, b), ["a", "b"]) | ||
| >>> covDf = df.agg(covar_pop("a", "b").alias('c')) | ||
| >>> covDf.select("c").collect() | ||
| [Row(c=565.25)] |
There was a problem hiding this comment.
Should we maybe compare with a tolerance as done in the other doctests since floating point?
There was a problem hiding this comment.
|
Test build #50164 has finished for PR 10876 at commit
|
|
This looks good to me, maybe @davies or @jkbradley could take a look? |
| >>> b = range(20) | ||
| >>> df = sqlContext.createDataFrame(zip(a, b), ["a", "b"]) | ||
| >>> covDf = df.agg(covar_pop("a", "b").alias('c')) | ||
| >>> covDf.selectExpr('abs(c - 565.25) < 1e-16 as t').collect() |
There was a problem hiding this comment.
Since these are going to be part of the API docs, I'd like to be as simple as possible.
Also the Python API is only a wrapper of Scala one, so we don't need to test the correctness here.
There was a problem hiding this comment.
So just as a side note - it seems the other functions (including corr) do have these tests for correctness in their doctests. We should probably be consistent with them one way or the other yes?
There was a problem hiding this comment.
Yeah, we should cleanup them up.
There was a problem hiding this comment.
@davies OK, I will cleanup them.
|
@yanboliang Could you also add Python API for corr? |
|
@felixcheung I see, thanks! |
|
Test build #50344 has finished for PR 10876 at commit
|
|
ping @davies |
|
@yanboliang I'm sorry that I didn't make it clear: It's good to have a simple test, that could also be used as an example to demo how to use these functions (it should be as simple as possible). |
yanboliang
force-pushed
the
spark-12962
branch
from
ad15ff4
to
8029911
Compare
|
Test build #51104 has finished for PR 10876 at commit
|
|
Test build #51106 has finished for PR 10876 at commit
|
|
|
||
| >>> a = [x * x - 2 * x + 3.5 for x in range(20)] | ||
| >>> b = range(20) | ||
| >>> df = sqlContext.createDataFrame(zip(a, b), ["a", "b"]) |
There was a problem hiding this comment.
How about a == b, then the result will be zero?
|
Test build #51181 has finished for PR 10876 at commit
|
|
Test build #51183 has finished for PR 10876 at commit
|
|
LGTM, merging into master, thanks! |
PySpark support
covar_sampandcovar_pop.cc @rxin @davies @marmbrus